This one weird trick gets rid of all off-topic comments. Let's use it!

Question

This is a site that gets a lot of off-topic comments, comments that really are answers, chatty comments that should be posted in chat, etc. I frequently flag these comments, but it's like I'm fighting an uphill battle. Every time I flag one comment thread, I find another. It really would be nice if there's a simple trick that we could use to get people to stop leaving off-topic comments.

Normally, there are no simple fixes. But in this case there is one. One of the community managers changed the text of the "add comment" button on Area51 to "suggest improvements", which is what comments are supposed to be used for. And, surprise surprise, off-topic comments on Area51 have all but vanished.

Is there any chance that we could get this feature? It would take at most a day to implement, we need it, and we could test it out to see if it should be implemented on the rest of the Stack Exchange sites.

Answer 1

A quick update on this request:

Yes, the Community Team has submitted a request to have this feature change implemented using Interpersonal SE to test it.

Acceptance by the dev team is still pending, but with the pent up demand and backlog of feature-requests, it is still unclear how this will be prioritized. If it is accepted, the date of completion is still to be determined so I do not have any further information at this time.

Stay tuned.

Answer 2

June 25: Experiment has ended

Ok! It's been about two months since this started, and I've turned off the altered comment text. This should be enough to give us a reasonable idea of what effect was had by this change.

With less to watch on TV, I've had time to fix a few serious errors in my initial query and also adjusted the method for associating edits and comments to avoid weird effects from edits made long after an irrelevant comment. With that done, I've updated the results in the section below, and now for a wrap-up I'll add another 30 days fore and aft.

The table below, contrasts 4 time periods: 60-30 days prior to the change, the 30 days before the change, the 30 days after the change, and the 30 days after that:

Messages
Description                     2/22-3/24  3/24-4/23   4/23-5/23  5/23-6/23   
------------------------------- ---------- ----------  ---------- ----------
Comments / post ratio                 4.45       4.31        3.40       3.79
Flags / comment ratio                 0.21       0.37        0.30       0.29
Pct comments deleted                 49.05      51.14       48.89      48.48
Comment upvotes / comment ratio       2.02       2.05        1.90       1.81
Pct comments followed by edits       11.16      11.00       14.13      13.15

The most noticeable effect here for me remains the last: a higher percentage of comments are followed by edits. That does not seem to be caused by an increase in constructive comments or edits - rather, it's caused by the reduction in comments overall (as indicated by the first metric). With fewer non-constructive comments being posted (and deleted), the portion of useful comments represents a larger portion of the total.

This clearly isn't a dramatic improvement. An awful lot of comments are still getting flagged and deleted... But, it is still an improvement.

The next question we need to answer... Can we replicate this success elsewhere?

June 3rd: Experiment results

Just as promised, I've collected the resul...

...Ok, so I said "In 30 days" when I probably should have said "after 30 days". It was a busy 30 days, ok? I had a LOT of stuff on NetFlix to watch. Clearly, the first thing learned from this experiment is "be more vague about schedules".

Anyway. Let's see how far we got in terms of achieving those 5 goals Jon suggested. Here are the numbers for each in the 30 days after April 23rd, contrasted with those for the 30 days leading up to April 23rd:

Description                     Before     After      PctChange  
------------------------------- ---------- ---------- ---------- 
Comments / post ratio                 4.31       3.40     -21.12 
Flags / comment ratio                 0.37       0.30     -20.27 
Pct comments deleted                 51.14      48.89      -4.40 
Comment upvotes / comment ratio       2.05       1.90      -7.21 
Pct comments followed by edits       11.00      14.13      28.38 

So...

1. Fewer comments. Yes!
2. Fewer comment flags. Yup
3. Lower ratio of deleted comments. Indeed.
4. More upvotes per comment. ABJECT FAIL.
5. More edits following comments. YES!

In conclusion: this is not a dramatic change; there are noticeably fewer comments, and they would appear to be aimed more at generating improvement, but an awful lot of them still get flagged and deleted. Also, they seem to be less popular with readers... Which, in spite of failing one of the goals, may not actually be a bad sign: if folks are spending less time soapboxing and more time suggesting improvements, there may just be fewer polarizing opinions to garner votes.

It's worth keeping in mind that some of this may be due entirely to the novelty of the change. So with that in mind, I'm going to leave the experiment to run for another month and see if any noticeable change in behavior is still detectable at the end of that time. This may also be a good time to try and replicate the results on another site.

April 23rd: Ok, this experiment is live:

From now until the end of the experiment, "add a comment" will no longer be available here on Interpersonal Skills; authors and those with the "comment everywhere" privilege will only be able to "suggest improvements". Kudos to Tim for reviving this request and Kasra for whipping up the functionality needed to toggle it!

There are no doubt tons of rough edges here, as my reference to the privilege illustrates; the help center, flags, user profiles and of course tons of meta posts will all still reference "comments". The goal here is just to see if this one weird trick actually works - if it turns out that it does, then we can see about making it less... weird.

Tests for success

So, how do we know if it works? Well, Jon Ericson suggested the following goals:

1. Fewer comments.
2. Fewer comment flags.
3. Lower ratio of deleted comments.
4. More upvotes per comment.
5. More edits following comments.

All of those seem like good things to me, so let's call it a success if during the duration of this experiment any of them improve significantly over the current status quo... Unless, of course, the ones that don't improve somehow get significantly worse.

We picked this site to test on because... Well, because y'all post a LOT of comments. And a significant volume of those comments are not useful. Heck, let's be honest: an awful lot of the comments here are awful, just awful, just awfully, awfully awful. I guess what I'm saying is, there's TONS of room for improvement - so if we're gonna see this little change have an effect anywhere, it's gonna be here.

In 30 days, we'll end this experiment and analyze the results.

Answer 3

Yes Yes Yes. This is a great idea.

I know your pain, I have 232 comment flags, 28 alone today and yesterday are waiting for review and 188 helpful in 27 days! This is madness. I feel it's even more important on this website because everyone has some form of interpersonal skills at all different levels and have opinions on other people's interpersonal skills in comparison to their own.

This is only a small way of improving the way people comment! So yes, I would love for this to be implemented. There is a lot of noise on this SE and this would not only help towards keeping this ever so strange addition to the exchange's Q&A format (and lack thereof discussion), it'll alleviate the mods a little on flags, so they can do the important stuff.

Answer 4

If Robert wants to test it here, I'm willing to be a guinea pig but I'm not going to hold my breath that it will work.

Area 51 does not reflect an actual site, the types of users, and the types of comments. Until it's actually tested on a regular site, I don't have any reason to think it will actually do anything.

Changing the wording on A51 and having it work does not mean that it will actually work here. People on the actual SE sites have been using comments for so long that it's unlikely that they'll even notice the verbiage change. It may reduce the number of new users utilizing comments incorrectly but old users have been writing comments for (potentially) years, even though they usually know that comments are 1. not for answers and 2. not for off-topic chatter.

The verbiage may help but I doubt it.

I think that other MSE solutions like comment locks or requiring local reputation to comment on protected questions could be more effective - though I'd prefer they have expiration dates if they were to be implemented. These would actually make it impossible to comment either at all or with insufficient local reputation.

Answer 5

Are we taking this just a bit too far?

There are many times when the comments are as engaging and delightful as the answers, even though they aren't direct questions/suggestions on how to improve the answer. Things like "Ha, that reminds me of a time when..." I'll suggest that this sort of tangential marginalia does add value in an indirect way.

Honestly, I know I'm bonking my head against a brick wall on this, but I have to say it. I predict that relentlessly removing tangential and non answer-improving comments will make the IPSE experience fussy and tedious.

Therefore ...

Let's do it. Prove me wrong! "Life is the best teacher", as they say.

Answer 6

Great idea!

I just looked at this and with mouse over the following text is visible. I guess it would be good to change the wording on this as well. Otherwise people will ask where the comments are - "Use comments"