How is the community doing at deleting answers flagged as Not an Answer?

Question

We've spent the last few weeks talking about what it takes to make an answer that meets the site's guidelines for quality and I think that these discussions are valuable and will help us decide where to go when we're ready to make the next step forward in setting expectations for answer quality... but I'm not sure that the wider community - the users not active on IPS Meta but active in the review queues - share the opinions of the users who are... and it seems that even the users who are active here on Meta aren't necessarily of the same opinions.

So... here's what I'm thinking... this comment says:

Believe it or not, that's already happening @Beofett - for the past 6 months, a majority of answers here have been deleted by vote, not by their authors and not by moderators. Similarly, low-quality review handles over 70% of VLQ and NAA flags. This wouldn't have had a chance of working back when we first discussed it; now it actually might: the greater community of IPS has the ability to do this sort of moderation, they just need to decide that they want to. Shog9♦

Because of this comment and some other discussion, it looked like there would be general support for creating a post notice that moderators could use to indicate when an answer failed to meet our back it up guidelines, so I requested it and now, a couple of weeks later it's been mentioned that we're not actually ready for this since the site isn't enforcing the policy.

Yeah, there's a reason I haven't commented on that, @Catija; a post notice should follow from what's working on the site already, in spirit if not in word. And we're a ways away from that. Shog9♦

I admit that I pretty much agree with this and that's without actually looking at any data. We have a queue that sees many NAA flags that are disputed by the community and later flagged again. Since NAA and VLQ flags only go through the LQP queue once (if that - accepted answers never go through it at all), it falls on the moderators to determine whether an answer is "sufficient" or not.

And many of these flags sort of sit around for a long time because, as Shog says, "we're a ways away" from knowing what the community wants us to delete as insufficiently supported.

---

What numbers can we get that will compare the quantity of answers that are flagged as VLQ or NAA that are:

1. Deleted by non-moderators
2. Deleted by moderators
3. Disputed (and never reflagged)
4. Disputed and reflagged and deleted
5. Disputed and reflagged but declined
6. Declined

There may be other things that are relevant that I'm not thinking of - deleted by users/moderators but never flagged, perhaps? Score of the deleted/disputed/declined posts?

I'm also not quite sure how to use the data after we have it - what does it tell us? How are we trending? Is there a way we can sort through the data to determine why an answer was deleted and whether it failed to meet our guidelines or something else.

This all sounds really complex but maybe I'm making it more complicated than it needs to be... so, maybe we should start simple and then look deeper after a bit.

Answer 1

Context is everything.

My response to Beofett was in the context of the extra load enforcing a "back it up" policy would impose on moderators. My response was meant to indicate that moderators wouldn't be handling most of that load.

My response to you was in the context of moving forward with enforcement by way of a post notice. Post notices require moderators to add them; there's currently no way to use post notices without involving moderators.

You can probably already guess how these come into conflict, but... First, let's have some baseline info on how NAA flags are handled here.

In the past 30 days, 297 Not an Answer flags have been handled on Interpersonal Skills Stack Exchange, distributed over 203 answers. Of these...

• ...260 flags were handled by the community (non-moderators via review or vote or some other means). These flags were distributed across 183 distinct answers.
• ...flags on 110 of those answers were marked helpful
• ...flags on 94 of those answers were handled via review
• ...flags on 71 of those answers were disputed
• ...flags on 9 of those answers were retracted by the flagger
• ...and a total of 93 of those answers were deleted by the community in the process of handling those flags.

Meanwhile...

• ...37 Not An Answer flags were handled by moderators on 30 answers.
• ...flags on 28 of those answers were marked helpful
• ...flags on 2 of those answers were declined
• ...and 26 of those answers were deleted at the time they were handled.

So, if you're getting bored just reading that, just think how bored I am writing it. SUPER bored. But maybe this'll wake you up: did you know that answers can go through Low Quality review twice? Because I didn't. Just recently updated a FAQ to stress the point that they couldn't; now I gotta update that again. There's even a comment in the relevant code to the effect of "don't let flags send this through twice". But they can. Not to get too far off-topic here, but... If the answer gets edited, it's fair game for review again. And that happened a few times too.

Anyway... Turns out there are a bunch of complicated scenarios where multiple batches of NAA flags can get handled on a given answer. Stuff like...

• ...Goes through review, existing flags get disputed, new flags get raised, mod handles new flags.
• ...Goes through review, existing flags marked helpful but answer not deleted, new flags get raised, mod handles new flags.
• ...Mod dismisses flags, new flags get raised, gets handled by review.
• ...Mod dismisses flags, new flags get raised, non-mods just delete it by vote.

...and a bunch of other combinations. But... These only affect like 18 answers in total during the past 30 days and they break down into really small pieces, so I kinda think they're all just a distraction.

Instead, let's just look at how the mods handled the mod-only flags raised when review doesn't delete something it handles, but maybe should have. There are two relevant flags here:

• A high-scoring answer gets enough "recommend deletion" reviews that it would be deleted if not for the score

  There were 7 of these answers, and mods deleted all 7 of them.

• An answer gets a sufficient number of "Looks OK" reviews to dispute the Not an Answer flags, but also got a non-trivial number of delete reviews.

  There were 2 of these, and mods deleted both of them too.

Conclusion

The community here is more than able to handle whatever they decide "isn't an answer" without putting a large burden on the mod team. However...

The community here is currently very conflicted about what answers should be. That will resolve itself eventually. But it will take time, and it will take a fair bit of work.

More on that elsewhere...

Answer 2

There may be other things that are relevant that I'm not thinking of - deleted by users/moderators but never flagged, perhaps? Score of the deleted/disputed/declined posts?

I'd personally be interested in:

• How many answers got a comment outlining the need to be backed up and were subsequently edited and not deleted?

• How many answers got a comment outlining the need to be backed up and were subsequently not edited and then deleted?

• How many answers got delete votes but were subsequently edited and then not deleted?

• How many of their delete votes does a regular user who can vote to delete typically use?

• How quickly do LQP review tasks get completed after creation?

• How many regular users are responsible for handling review (How many different users regularly review stuff)?

I have a feeling that comparatively few users are responsible for the bulk of review handling and I'm not sure we can draw meaningful conclusions because of that.