4

This is kind of a meta question and I'm sort of "thinking aloud" so bear with me, give me constructive criticism and I'll edit the question to reflect that.

How do we know that high-ranking or designated answers to IPS questions are correct and not just the results of confirmation bias?

In other SE forums, there are definite ways to check the correctness of answers. In Stack Overflow either your answer works or it doesn't. In forums such as Music Theory and Practice, there is more leeway, but still the answers can be compared to existing theory or tried in practice in a repeatable manner.

But in interpersonal relations, we often don't have clear-cut answers or the ability to repeat experiments. When we vote on answers, how do we know that the answer is correct or that it's incorrect but agrees with our personal biases?

What I'm concerned about is that we're a group that may have issues with interpersonal issues who are ranking answers on interpersonal issues based often on nothing more than personal experience and anecdata.

Are there better criteria that we can use to rank answers? More requirements for references to current scientific research?

EDIT: In the comments, it has been suggested that there are no correct answers, only good answers and that good answers work. However, anyone can vote on answer, regardless of whether they have tried the answer or not. Confirmation bias comes into play again.

I don't have any good answers to this question so I welcome yours.

6
  • Related: interpersonal.meta.stackexchange.com/q/221/1599
    – Tinkeringbell Mod
    Commented Mar 23, 2019 at 10:14
  • 3
    Also, you might be interested in browsing existing discussions about answers, like this, this, this, this, this and this. Might be surprising, but as for me personally, for a very subjective site like IPS, there are no correct/wrong answers, only good/bad answers, and that's what the vote indicates
    – Andrew T.
    Commented Mar 23, 2019 at 11:41
  • 1
    @AndrewT. "there are no correct/wrong answers, only good/bad answers" = yup! and "How do we know if an answer is correct?" = maybe when OP updates and tells us it worked for them? ;)
    – OldPadawan
    Commented Mar 23, 2019 at 12:12
  • @AndrewT. how do you define "good" and "bad" answers in this context?
    – empty
    Commented Apr 3, 2019 at 18:20
  • 1
    I just want to make clear that I think that this is a really good and valuable conversation for the stack to have. I'm concerned that the tone of my answer might read as overly critical-- any such impression is due to the difficulties in addressing this topic, and not in any way hostility to the topic itself.
    – Upper_Case
    Commented Apr 3, 2019 at 18:29
  • @Upper_Case I am always in favor of respectful but frank exchanges of views. That's how we stumble towards the truth. Too little of that goes on in the outside world and I believe in modeling the behavior we would like to see in others. So no, I don't view your answer as too critical. I greatly value the quality of your argument and the way that you have presented it.
    – empty
    Commented Apr 3, 2019 at 18:39

1 Answer 1

6

No need to hedge-- meta is the best place for meta questions!

We can't.

1. What does it mean for an answer to be "correct" here?

As pointed out in comments on the question, "correct" is kind of fuzzy for what IPS deals with. Even trying to assess based on answers that "work" is hard, because there is no real way of knowing how many alternative approaches would "work", and (very importantly) there are nuances to situations that we simply can't observe here.

Two questions might be on the same topic, have the same basic setup, and ask for the same outcome, but the real, fully-defined states of those situations might not be identical. Those differences could easily be enough that an answer would "work" in one but not the other.

As a hypothetical example of this, consider questions about saying something difficult to someone without that person getting angry-- that's going to depend enormously on the person being told something, and we're simply not going to reliably get a good picture of that from questions.

2. What do upvotes and downvotes indicate?

I think that on the whole the voting system is good for identifying answers that the community feels have value. There is a lot of ancillary information that gets conveyed this way (though with poor resolution).

However, even on sites with falsifiable topics I think that it is pretty unlikely that every voter is actually testing the solutions offered and voting accordingly. Obviously, some do, but I'd bet that an answer with 100+ upvotes has a lot of "this looks like it would probably work" votes. I've seen questions with highly upvoted answers which are simply, demonstrably incorrect-- falsifiability isn't an absolute defense.

And downvotes can come from information totally separate from whether or not the posted answer is correct (however we want to define that). A correct answer might be downvoted because a user felt it had the wrong tone, for example.

Votes are simply not a high-integrity representation of specific information. They're more like Reddit upvotes or Facebook likes than formal sign-offs on code reviews. Whether or not that issue is worse on IPS than elsewhere (and I suspect that it is), I think that you may be over-valuing the information content of vote totals here.

3. What might be better?

I want to say upfront that I definitely believe that improvements are possible (including in ways I've not yet imagined). But I feel that the improvements are far more likely to be marginal than revolutionary.

We're always trying to strike a balance between site usability and reliability of answers. The "back it up" policy is a good example of this: we prefer answers to contain rationales for why the answerer thinks they will work, whether that's a sequence of logical reasoning or drawing from personal experience. We generally can't verify that someone who claims to be answering from experience is being honest, but I'm not sure that that's a solve-able problem here.

As for referencing research, I am on record as aggressively opposing that idea. It would be nightmarish to moderate, difficult for site users to do (are you going to read a dozen 50-page research reports, at $30 apiece, before voting on or accepting an answer?), very frequently isn't more conclusive than what we've already got (studies are almost always pretty narrowly constructed in anything that would touch on IPS), and the peer-reviewed publication system is designed for ongoing discussion among certified experts (and not black-and-white Q&A for specific instances, which is what IPS.SE is).

In a lot of ways, I think that relying on citing publications for answers inculcates a lot of these same problems and then hides them under a veneer of scientific validity.

As examples, consider the Milgram experiment and the Stanford prison experiment. People held these up for a long time as indicating a definitive frame for how humans behave. But over the last decade evidence has come out that there were some pretty serious methodological problems with those studies.

Is a standard IPS user equipped to evaluate how meaningful those problems are while focusing on the content of a question for which those studies are only background information? Do they still count as evidence in an answer? Would an answer based on a theory of human behavior derived from those studies be better or worse for citing those studies explicitly? Is an answer that misrepresents or overinterprets the results of a study better than another answer which cites nothing at all?

As much as I would like a better and more objective way to evaluate the answers here, I'm not sure that it's realistic. It's a very subjective domain, and questions are already heavily influenced by factors which are invisible to all involved, and then interpreted by potential answerers subject to another largely unobservable set of factors.

4
  • Thank you for your thorough and thoughtful answer. I'm still on the side of scientific research citations because it's falsifiable. Science isn't a "veneer". The Milgram and Stanford experiments were challenged on a scientific basis, not on the basis of downvotes. I'm have serious concerns that IPS upvotes and downvotes are simply a reflection of our collective biases as a very limited demographic.
    – empty
    Commented Apr 3, 2019 at 18:19
  • @empty I am a professional scientist, and am familiar with falsifiability. Science is not a veneer. Claiming that an answer is supported by "science", simply because a study was published somewhere, in an environment (SE) in which discussing that paper is difficult (if not impossible), absolutely is a veneer. Milgram and Stanford were meant to be examples of this-- they were published, and so would be "scientific research citations", but it's not clear that they are valid citations due to the critiques of their scientific quality. So I don't think that an answer including such a citation
    – Upper_Case
    Commented Apr 3, 2019 at 18:24
  • (continued) is automatically better than a citation-free answer. My thoughts are more fully outlined in another meta post: interpersonal.meta.stackexchange.com/questions/3504/…. Your concerns about the inherent bias due to the arbitrary composition of the IPS.SE community are well-taken, and I think that they are valid and have a real impact on the site. I'm just not sure how much can be done to fix or counteract that.
    – Upper_Case
    Commented Apr 3, 2019 at 18:26
  • thanks for the link. I agree with points 1-5. 6 is problematic and I think unenforceable. I ran into issues with 6 in a different SE because the group consensus, with moderator agreement was that the quality of an answer did not depend on the professional skill level of the answerer. I then disagreed but since have come around. An argument from authority is still a logical fallacy. BTW my entire immediate family, including me, are professional engineers and scientists. X-D
    – empty
    Commented Apr 3, 2019 at 18:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .