June 25: Experiment has ended
Ok! It's been about two months since this started, and I've turned off the altered comment text. This should be enough to give us a reasonable idea of what effect was had by this change.
With less to watch on TV, I've had time to fix a few serious errors in my initial query and also adjusted the method for associating edits and comments to avoid weird effects from edits made long after an irrelevant comment. With that done, I've updated the results in the section below, and now for a wrap-up I'll add another 30 days fore and aft.
The table below, contrasts 4 time periods: 60-30 days prior to the change, the 30 days before the change, the 30 days after the change, and the 30 days after that:
Description 2/22-3/24 3/24-4/23 4/23-5/23 5/23-6/23
------------------------------- ---------- ---------- ---------- ----------
Comments / post ratio 4.45 4.31 3.40 3.79
Flags / comment ratio 0.21 0.37 0.30 0.29
Pct comments deleted 49.05 51.14 48.89 48.48
Comment upvotes / comment ratio 2.02 2.05 1.90 1.81
Pct comments followed by edits 11.16 11.00 14.13 13.15
The most noticeable effect here for me remains the last: a higher percentage of comments are followed by edits. That does not seem to be caused by an increase in constructive comments or edits - rather, it's caused by the reduction in comments overall (as indicated by the first metric). With fewer non-constructive comments being posted (and deleted), the portion of useful comments represents a larger portion of the total.
This clearly isn't a dramatic improvement. An awful lot of comments are still getting flagged and deleted... But, it is still an improvement.
The next question we need to answer... Can we replicate this success elsewhere?
June 3rd: Experiment results
Just as promised, I've collected the resul...
...Ok, so I said "In 30 days" when I probably should have said "after 30 days". It was a busy 30 days, ok? I had a LOT of stuff on NetFlix to watch. Clearly, the first thing learned from this experiment is "be more vague about schedules".
Anyway. Let's see how far we got in terms of achieving those 5 goals Jon suggested. Here are the numbers for each in the 30 days after April 23rd, contrasted with those for the 30 days leading up to April 23rd:
Description Before After PctChange
------------------------------- ---------- ---------- ----------
Comments / post ratio 4.31 3.40 -21.12
Flags / comment ratio 0.37 0.30 -20.27
Pct comments deleted 51.14 48.89 -4.40
Comment upvotes / comment ratio 2.05 1.90 -7.21
Pct comments followed by edits 11.00 14.13 28.38
- Fewer comments. Yes!
- Fewer comment flags. Yup
- Lower ratio of deleted comments. Indeed.
- More upvotes per comment. ABJECT FAIL.
- More edits following comments. YES!
In conclusion: this is not a dramatic change; there are noticeably fewer comments, and they would appear to be aimed more at generating improvement, but an awful lot of them still get flagged and deleted. Also, they seem to be less popular with readers... Which, in spite of failing one of the goals, may not actually be a bad sign: if folks are spending less time soapboxing and more time suggesting improvements, there may just be fewer polarizing opinions to garner votes.
It's worth keeping in mind that some of this may be due entirely to the novelty of the change. So with that in mind, I'm going to leave the experiment to run for another month and see if any noticeable change in behavior is still detectable at the end of that time. This may also be a good time to try and replicate the results on another site.
April 23rd: Ok, this experiment is live:
From now until the end of the experiment, "add a comment" will no longer be available here on Interpersonal Skills; authors and those with the "comment everywhere" privilege will only be able to "suggest improvements". Kudos to Tim for reviving this request and Kasra for whipping up the functionality needed to toggle it!
There are no doubt tons of rough edges here, as my reference to the privilege illustrates; the help center, flags, user profiles and of course tons of meta posts will all still reference "comments". The goal here is just to see if this one weird trick actually works - if it turns out that it does, then we can see about making it less... weird.
Tests for success
So, how do we know if it works? Well, Jon Ericson suggested the following goals:
- Fewer comments.
- Fewer comment flags.
- Lower ratio of deleted comments.
- More upvotes per comment.
- More edits following comments.
All of those seem like good things to me, so let's call it a success if during the duration of this experiment any of them improve significantly over the current status quo... Unless, of course, the ones that don't improve somehow get significantly worse.
We picked this site to test on because... Well, because y'all post a LOT of comments. And a significant volume of those comments are not useful. Heck, let's be honest: an awful lot of the comments here are awful, just awful, just awfully, awfully awful. I guess what I'm saying is, there's TONS of room for improvement - so if we're gonna see this little change have an effect anywhere, it's gonna be here.
In 30 days, we'll end this experiment and analyze the results.