How Can We Best Incorporate Academic Literature into IPS.SE?

Question

I'm on record as being concerned about the difficulties of basing questions and answers exclusively about published academic literature here on IPS.SE. The summary version is that I don't think that more focus on published research will improve IPS.SE question or answer quality very much, will cause a proliferation of questions we would otherwise close, and will lead to IPS becoming a less valuable resource overall. I still feel that way, and will try to describe my concerns further down in this post.

However, it also seems clear to me from discussions on this meta that there is a strong appetite for citations from these sorts of papers, and even some for answers which are nothing but indices of such papers. This site is what we make of it, so let's discuss!

I also have concerns about how well academic studies will fit here. They're not all equally important and they can (in some cases) be addressed, but they are obstacles that bear discussion. This is not an exhaustive list (I haven't even touched on observational vs. experimental studies, empirical vs. simulated, verisimilitude of the study and the real world, and lots of others), but it's long enough already. At least meta allows for a bit more back-and-forth discussion than the main site.

My major concerns:

1. Academic literature is a specific type of thing, intended for a specific environment, and we don't have that environment here.

The SE format is a poor one for discussion, but discussion is what scientific publications are about. A fish may swim in a freshwater lake and thrive there. Drop it on a dry sidewalk and it probably won't. Put it in a bowl with distilled water, and it probably won't. Put it into a saltwater tank, and it probably won't. I have a difficult time imagining anything we could do here that would replace this integral element of scientific publication.

2. Academic papers can be very difficult to read

There's plenty of variety, but academic papers tend to be dry and complex. The more an answer requires the asker to simply read the paper, the less valuable it is as an "answer" in the SE format. And the more difficult a paper is to read, the less likely people will be to actually read through it, making up/downvoting, suggestions for improvement, and selecting an answer to accept more difficult.

3. The context in which research is published is really, really important

There are lots of elements that can wildly impact how a study should be interpreted. We will have a hard time including that here, I think-- meta discussion of a study doesn't really fit well when that study is being used to support a specific answer to a specific question that is not about the study itself. I am concerned that this will lead to systematic overstatement of study conclusions, assuming that they are accurately described in the first place. People who have faith in science but do not participate with the literature much tend to over-emphasize the value of publication in terms of describing the true state of reality.

4. Research is an ongoing endeavor. SE questions and answers, in large part, are not

Papers are sometimes updated, improved, discredited, and more, and entire conceptualizations rise and fall. I do not believe that there will be much maintenance of answers here. Think of what you've heard about what types of cholesterols and fats are "best" to eat, and why, and which will kill you, and why, over the last 30 years. It's all over the place! Pulling individual studies (or even groups of studies) at a single moment in time to post here does not keep up with continuing research.

5. Posting papers here will suggest that papers are of similar quality

Some papers are simply better at discovering information than others. It might be due to budget limiting how big a study can be, or a lack of pre-registration meaning that a researcher p-hacked until they found a "publishable" result, and any number of others. The lack of discussion capability here means that these sorts of differences will be hard to display, leading to answers of varying quality that will all look roughly the same. It will also become impossible to assess answers (including for voting purposes) without reading each and every paper cited to determine their quality and the accuracy of the write-up.

6. A lot of current research is not available free of charge

Relying only on what's available at PubMed or arXiv isn't going to be enough to ensure that information is current for fields in which active research is taking place. Citing papers that are paywalled means that someone without a relevant subscription cannot view, let alone assess, the background for the answers in which those papers are cited. Relying on only freely available studies means that only a portion of the research conversation is going to be presented (which will vary in importance across fields of study).

7. These, and other, difficulties can make "back it up" harder

For a "back it up" statement based on an answerer's personal experience or analysis, they can describe it in the answer. For a "back it up" that is entirely based on published research, most of the relevant information is going to be in those papers and will be hard to put into the answer.

Answer 1

My suggestions for what would make for "good" inclusions of academic research:

1. Papers must be publicly available

Things are changing, somewhat (if slowly), but it is still the case that a lot of current (and sometimes not-so-current) literature is expensive. Papers that are currently paywalled can't be examined by anyone without journal access, nor can they be legally reproduced in their essential elements. So, to actually be valuable in an answer to a question, the papers must be available without charge.

2. The relevant essentials of a paper should be excerpted in the answer

Someone with a question wants an answer, not necessarily 80 pages of dry, technical literature in which their answer may, perhaps, be found. We discourage link-only answers on SE for a variety of reasons, and that the contexts of a published paper may be unwieldy doesn't make those reasons go away. This doesn't apply to "further reading" suggestions, but in general the substance of an answer should be in the answer itself.

3. Relevant context from the paper should be included explicitly in the answer

Academic studies make a lot of assumptions, judgment calls, and so on, as a necessary element of controlling for variables, isolating effects, defining contexts, and tons of other stuff. This sets a very specific, precise standard for what the study is, what its results can mean, and the situations in which those results might apply. Interpreting the results of a study absolutely requires this information, and if an answer is to be meaningful on its own this information should be there. I acknowledge that this is likely to be a significant burden on answerers.

4. The results of a study should be stated plainly, precisely, and with any necessary context

In practice I imagine this will mostly be excerpts or paraphrasing of the results/discussion offered by the study authors.

5. Those posting published research as the core of an answer should have already read and understood the entire paper

This might seem like a no-brainer, but I have seen a lot of people cite papers that they've only skimmed or, much worse, only read the abstract of. This is not enough to understand a paper fully, nor to present its conclusions (as outlined above). Further, an enormous amount of research uses very specific methods which can be difficult to understand, and extremely easy to overinterpret and misrepresent (p-values being one of the most common, and famous).

Secondary summaries are similarly undesirable (so no "according to a write-up about this paper I saw at Vox...). The answerer cannot vouch for the capability of the person who wrote that summary, and without reading the actual paper cannot assess how well or poorly the summary-writer did. Meta-reviews of published literature should generally be OK, but those reviews should be read and understood as would be the case for an individual paper.

6. Those who answer questions seeking academic research in answers should be conversant with the broader published literature on the topic

I don't see how this will be enforceable, but it's important, perhaps the most important of these points. Reading a single paper, or even a handful of papers, is not enough to suggest anything about what "science" says. Scientific papers are part of a discussion among academics and they are written for, and intended to be consumed in, that context.

As an example, consider if someone posted a question like "Do vaccines cause autism?" (and further assume that that question is on-topic for the stack and requests academic research). And a well-meaning answerer googles around a bit and comes up with Andrew Wakefield's paper, which kicked off the topic, reads it, and posts it as an answer without further research.

Now, whatever your views on this topic, that particular paper has serious methodological flaws, conflicts of interest on the part of the author, and was ultimately retracted by the journal that published it (which is a huge deal in the world of research papers). Anyone following the published literature would be aware of all of this and could place the paper in appropriate context. My hypothetical question-answerer does not know this, and someone asking the question (or stumbling across it later) likely does not know it either. The SE format makes bringing that information up and discussing it really, really hard. Sure, we can downvote the question, and maybe mods will delete it. And most research is not anywhere near as flawed as that particular work. But relying on these radically increases the review and moderation burden.