Can we start enforcing the "back it up" policy?

Question

We've had the "back it up" policy for almost a year now. It was one of the earlier things we seemed to think important for the quality of answers on the site.

We've had more than a few discussions about supporting answers with experience or references, as well as a few discussion about the problems that unsupported answers seem to be causing:

What should we do with answers that are not backed-up?

Related Answers: Why your Pakistani answer won't always work for India

Do we want references in our answers?

Should we encourage writing from experience?

How can we encourage/enforce the backing up of answers?

Do we have a problem with people answering questions that they don't know the answer to?

(I'm sure I've missed some, feel free to add to the list)

Admittedly, I once thought that votes would handle this issue. I'm generally not a big fan of hard rules and I perhaps had a little too much faith in people's common sense (pun intended) when it came to voting. Unfortunately it looks like we've had a lot of cases where votes have made the problem worse...

It seems like the majority of users seem to agree that unsupported answers are a problem, and most seem to agree that we can, and probably should, handle it the way that other, more subjective, SE sites have handled it.

https://interpersonal.meta.stackexchange.com/a/2985/59

https://interpersonal.meta.stackexchange.com/a/2988/59

https://interpersonal.meta.stackexchange.com/a/1265/59

(Again I'm sure there are others, feel free to add them to the list)

To paraphrase the above...

User side:

1. Someone posts an unsupported answer.
2. Add a nicely worded comment asking for the answer to be supported.
3. Flag the answer, using a custom mod flag, to let them know that the answer isn't supported.

Moderator side:

4. A moderator responds to the flag by commenting and/or adding a post notice.
5. If the answer isn't edited to add support, or the answer is significantly problematic, the moderator may then delete the answer.

Please realize that that's a paraphrased tl;dr of the three answers above. Please read the above answers for the fine details.

The question is... Are we ready to bite the bullet and start enforcing our back it up policy?

I would like to think that we are.

(Note that this is tagged feature-request, we've discussed this plenty, but this isn't a duplicate of the previous discussions because I'm asking that we finally take action.)

Answer 1

Finally!

Here's what I'd propose as a strategy for moderators:

1. Moderators see an unsubstantiated answer, they either delete it outright with a comment, or slap a post notice on it and revisit in a few days and delete if not substantiated.

...that's it. Pretty simple. But... We can't exactly expect moderators to review every answer posted here. Nor can we expect moderators to recognize every unsubstantiated answer that they come across. This system doesn't work that way - many hands make light work, and so if we're gonna do this we need many hands...

So here's what I'd propose as a strategy for everyone using the site:

1. See an answer that's unsubstantiated

2. Post a comment to the effect of... No, y'know what, I had a generic example here but that was a bad idea. Post a comment specific to the answer:

   • point out any claims that aren't supported by evidence of some sort - something the author has experienced, something they've read, anything tangible.
   • explain why you're having trouble accepting these claims without backup
   • politely ask the author to provide supporting evidence for their claims.

   This step is important. I cannot emphasize that enough. This site isn't an informal support forum, but sometimes it sure looks like one - and so you can forgive authors for writing in the style they would on such a forum, and gently guide them towards something a bit more rigorous. If you get into a disagreement, stay polite and ask them to raise the matter for discussion here on meta.

3. Downvote the answer

4. Flag the answer as "Not an answer"
5. Go to review and review posts flagged by other members of the site, voting or recommending deletion in cases where they too fail to provide backup for their assertions.

how this can work

If the post collects a significant number of delete / "recommend deletion" reviews, and scores <= 0, it'll be deleted - but in such a way that the author can undelete it if they wish to.

If the deleted answer is undeleted or scores more than 0, a moderator flag will automatically be raised.

Moderators, now informed of the need, can then follow the proposed "strategy for moderators" outlined at the top of this post.

Answer 2

I agree that we should be "backing answer up", but I'm quite disappointed that everyone seems takes that to mean "present some similar experience you've personally gone through" (and literally nothing else).

Justification, explanation or elaboration is the way we should be backing answers up.

What we should be looking for in answers is:

• Reasoning about the situation (e.g. "I'm recommending A because people like B and don't like C, and this can be seen by the fact that they did D")
• Addressing possible criticism (e.g. "This may cause A, because B. If you want an approach that is less likely to do so, you can try C instead of D")
• Explaining how to handle possible responses (e.g. "They are likely to say A because B, to which you can respond with C, which will do D")

Because we want others to:

• Understand why you think the advice you're giving would work (for reasons other than "well, it worked before")
• Judge the advice by the quality of the advice (instead of by the user posting it or by just taking their word that it works)
• Generalise the advice to other situations and handle unexpected ways the situation can play out

What we should not be looking for is: (for reasons explained below)

• I'm right because I went through something similar
• I'm right because I'm also X
• I'm right because I'm a psychologist

Experience (by itself) is not a good way to back an answer up.

Adding some experience to an answer might have some uses, but overall I'm not convinced about the benefit of an anecdote (or otherwise justifying experience).

• People aren't all the same. Something working once, or in a specific environment, doesn't mean it will generally work or it's good advice. Providing an anecdote of how one person responded may, in fact, mislead the reader into believing this is how something is likely to be received.

• Some of these scenarios are extremely specific, meaning there may not be anyone able to speak (much less give the best advice) from experience.

  Others may be able to speak from experience, but have privacy concerns about sharing said experience (since plenty of the questions here are about what can be considered very sensitive and private matters).

  I'll bet we'll start haemorrhaging our best answerers, have our unanswered rate skyrocket and eventually kill the site if we start strictly trying to enforce this, because there simply aren't enough questions to limit the number of questions any given person can answer by such a large amount, while still keeping them interested in staying here, and there probably simply aren't enough answerers to prevent such a large portion of people from answering (at least at this stage - it might work if we get to Stack Overflow numbers).

• Interpersonal skill is about being able to respond well to a new situation or respond well to many situations - this means you can give good advice without having lived through a similar experience.

• All the experience in the world doesn't mean you actually managed to figure out how to deal well with this situation. Would you assume someone who's gone on 10000 first dates is a good person to ask for first date advice? I wouldn't.

• Every interaction you've ever had with another person is experience. One can learn things about people based on a set of related or unrelated experiences and translate that to new experience (that's learning in a nutshell).

  Some of these experiences may be too mundane to remember or be noteworthy, other times you may be basing advice on a collection of experiences, which you can't really briefly, or at all, summarise.

• This disregards a large part of what voting is for. Other experts vote to show how useful an answer is, and may be voting based on their experience. If someone posts an answer, it should be voted for based on how useful the voter believes the advice is, not because of an anecdote that shows that it might work.

  How well does voting actually succeed in this goal? That's a good question, but, even if it fails miserably, that still wouldn't really make this a good alternative.

• Lying is a thing. If we're forced to back up answers with experiences, what's to stop people from just making up an anecdote so their answer is acceptable (at least in as far as possible)?

  Why would they do this? Well, because they want to contribute here, but, to answer any given question, there's this arbitrary hurdle of needing to have lived through that specific experience or something close enough to it to get over.

  What's to stop lying on other circumstances? Motivation, firstly - there's much less reason to lie in other scenarios (at least in as far as answering on this site goes). Also, other experts - no-one can really say "there's absolutely no way that anecdote is true" (because unbelievably improbable things do occasionally happen). If, on the other hand, you're presenting flawed reasoning, a fabricated reference or things that are simply objectively not true, other experts would easily (or at least reasonably) be able poke holes in that.

• This is not the way things work on other sites.

  On technical sites, i.e. Stack Overflow, you can be asked for proof (which an anecdote is not) or you can be asked for an explanation, but no-one will ask you to provide an anecdote or convince them why you're qualified to give this answer (or at least such requests are extremely rare... and usually not constructive).

  It also doesn't work this way on Workplace, for that matter (which has been mentioned at some point in this discussion) - no-one expects answers to be backed up with anecdotes there (but explanations - sure).

• Anecdotes are bad at teaching. If you say "I'm recommending A", adding "because I once tried A and it worked" will not teach the reader anything. That will only allow the reader to handle that specific scenario, not generalise it to other similar scenarios, and possibly totally mess up because they are unable to modify the approach in case something unexpected happens. One needs to understand the underlying reasoning for interpersonal advice to be useful.