This is later than I would have liked but having a couple of months of data rather than one or two does give us a bit more information. Could wait until Friday and have another month but... eh... For moderation stats, Shog posted some info here that's worth looking at, though I'm guessing most of y'all have read it already.
To start out, here are some numbers that compare the period since the removal from the HNQ this year to that period last year:
Period Questions Answers Good questions Good answers 3rd-party edits Votes Votes on >30-day-old posts Comments
-------------------------- --------- ------- -------------- ------------ --------------- ----- -------------------------- --------
Oct 16 2017 - Feb 1st 2018 857 3611 417 2404 1370 53054 5258 18641
Oct 16 2018 - Feb 1st 2019 500 934 242 608 684 9862 2853 4467
There are a couple of things that I think are particularly interesting, though not unexpected.
- Questions are down by ~40% but answers are down ~75%.
This makes sense, since most HNQ traffic pulls in answers from users around the network. It's still somewhat startling to see how much of the activity is from HNQ, though.
- Votes are down ~80% but votes on >30-day-old posts are down ~45%
Again, this makes sense as most of the vote volume is on HNQs which are under 30 days old. That said, when you're relying on mostly traffic from search engines - as IPS is right now - this is a stat you would generally hope wouldn't really drop much at all. Most of those voters would be getting referred here from Google and then hopefully poking around on the site a bit before wandering off. That doesn't seem to be happening - likely because many of these users can't vote.
I got Shog to create this cool plot of voting on the site. He'd done one previously for comments (for emphasis, the red arrow indicates when the site was pulled from the HNQ):
What I'm seeing here is actually a bit less worrisome. We've cut out most of the huge outliers that earned hundreds of upvotes due to HNQ attention but the median votes are only slightly down from before the site was dropped from the HNQ and there are still posts regularly getting a score >20. The regular users of the site are still around and still voting on posts. This is great! You are the heart of this site and what are keeping it going.
Here are some more charts from the mod analytics tool, which is much more powerful than the regular site analytics - oh, and these numbers are for the lifetime of the site, to better compare year-over-year:
Questions & Answers by month
New users by month
I put these two back-to-back because I thought it was interesting how closely the new users count lines up with the number of answers - except in May of 2018. I'd love to know what happened in May to cause that.
Page views by month
Google Analytics was broken for a while before some time in February of 2018, hence the lack of data.
Visits by new users
Google Analytics data can be a bit wonky and untrustworthy, as I mentioned with the lack of data for a big chunk of time, assuming this is correct, though, it's interesting that it seems to say that people GA considers "new" to the site have stayed stable. Like the relative stability of votes on 30-day-old posts (compared to all posts), this is somewhat expected as the most likely candidates for these new visits are referrals from Google, which haven't be impacted by the removal from the HNQ.
So far there's one thing I notice. While the site took a decided drop in most metrics in October, there was a slight downward trend overall since that big peak in August. You can see that in the graphs Ælis posted in their answer, too. There was one place where there was an encouraging slight increase in activity up until October, though:
Visitors on at least one or five days
The site was building up more visitors in general but also more people who were returning five or more times over the course of a month.
Together, these charts definitely show that there's been a big downturn in activity on the site. Whether this has been good or bad for IPS as a beta site working through some important foundational discussions is something worth thinking about. Yes, there's less traffic but has it allowed you to have some room to breathe (room we'd been asking for since the site went live) to come to some decisions on scope or post quality (etc) or has it just been a loss of traffic. Have you used the four and a half months to iron out some of the concerns?
One more thing before I go... let's look at the effect on comments, specifically, some of the stuff we looked at last spring - comment noise:
Description Before After PctChange
------------------------------- ---------- ---------- ----------
Comments / post ratio 3.71 3.05 -17.81
Flags / comment ratio 0.31 0.29 -8.23
Pct comments deleted 54.62 37.68 -31.01
Comment upvotes / comment ratio 1.67 0.89 -46.66
Pct comments followed by edits 13.97 15.05 7.68
For easy reference, here's the same data from the comment text change:
Description Before After PctChange
------------------------------- ---------- ---------- ----------
Comments / post ratio 4.31 3.40 -21.12
Flags / comment ratio 0.37 0.30 -20.27
Pct comments deleted 51.14 48.89 -4.40
Comment upvotes / comment ratio 2.05 1.90 -7.21
Pct comments followed by edits 11.00 14.13 28.38
I think this is pretty big, though not unexpected. There are clearly fewer comments, so fewer need to be deleted, hence the huge drop in comment deletion percentage. There's a lower percentage of flagged comments, too, and a higher percentage of edits after comments. The change in the last of those isn't as big but if you look at the actual percentages rather than the percentage changed, 15.05% is a point higher than the 14.13% from the other experiment.
There are some changes on the horizon - and soon - for the Hot Network Questions and once those changes are implemented, IPS will have the choice of whether y'all want to get back on the list or not. I know you've had a couple of discussions about it already but those are with the assumption that the HNQ is the way it currently is. The changes we're making aren't life-altering but they will address some major issues and give us the ability to learn about the types of questions the HNQ features so that we can further improve the feature moving forward.
Thank you so much for your patience. Let me know if you have any questions or if there's anything I missed.
Thanks and credit goes to Shog for pulling most of the data and helping me understand it.
