13

Note: This isn't a request for SE to add this site back to HNQ. Rather, it's a question asking about what evaluation has been made about this site and HNQ overall, and if this site can be cleared for potentially contributing to HNQ should the community agree on it later.

According to the answer to https://interpersonal.meta.stackexchange.com/a/3335/11936:

This removal from the HNQ is temporary. How temporary will depend on a few things. It will probably last at least until next year but I'm hoping to turn a negative into a positive and use this to really see what effect the HNQ has on this site. More on this in a bit.

Well, next year has come by.

Later on in the post, it mentions that this can be turned into an experiment in order to compare data:

Now that we're over a year in and have some data to compare to, I'm excited to turn this into an experiment to see what the site does when it doesn't have a constant influx from the HNQ eyeballs. I've already got plans for the Community Team to look at some post-mortem numbers in a month or two with the goal of seeing how this affects a few things...

Has this evaluation been done? How are the numbers like now? I can tell subjectively that quite a few changes have happened (e.g. site traffic going down tremendously), but I'd like to see some numbers and the Community Team's evaluation of said numbers.

Additionally, it would be great to know if SE has or may clear this site for potential future re-inclusion, so that if the community discusses and decides that this site should be re-included, the request won't be rejected by SE.

8

This is later than I would have liked but having a couple of months of data rather than one or two does give us a bit more information. Could wait until Friday and have another month but... eh... For moderation stats, Shog posted some info here that's worth looking at, though I'm guessing most of y'all have read it already.

To start out, here are some numbers that compare the period since the removal from the HNQ this year to that period last year:

Period                     Questions Answers Good questions Good answers 3rd-party edits Votes Votes on >30-day-old posts Comments 
-------------------------- --------- ------- -------------- ------------ --------------- ----- -------------------------- -------- 
Oct 16 2017 - Feb 1st 2018 857       3611    417            2404         1370            53054 5258                       18641    
Oct 16 2018 - Feb 1st 2019 500        934    242             608          684             9862 2853                        4467     

There are a couple of things that I think are particularly interesting, though not unexpected.

  • Questions are down by ~40% but answers are down ~75%.
    This makes sense, since most HNQ traffic pulls in answers from users around the network. It's still somewhat startling to see how much of the activity is from HNQ, though.
  • Votes are down ~80% but votes on >30-day-old posts are down ~45%
    Again, this makes sense as most of the vote volume is on HNQs which are under 30 days old. That said, when you're relying on mostly traffic from search engines - as IPS is right now - this is a stat you would generally hope wouldn't really drop much at all. Most of those voters would be getting referred here from Google and then hopefully poking around on the site a bit before wandering off. That doesn't seem to be happening - likely because many of these users can't vote.

I got Shog to create this cool plot of voting on the site. He'd done one previously for comments (for emphasis, the red arrow indicates when the site was pulled from the HNQ):

Whisker plot showing median vote counts along with outliers over a 90 day period

What I'm seeing here is actually a bit less worrisome. We've cut out most of the huge outliers that earned hundreds of upvotes due to HNQ attention but the median votes are only slightly down from before the site was dropped from the HNQ and there are still posts regularly getting a score >20. The regular users of the site are still around and still voting on posts. This is great! You are the heart of this site and what are keeping it going.

Here are some more charts from the mod analytics tool, which is much more powerful than the regular site analytics - oh, and these numbers are for the lifetime of the site, to better compare year-over-year:

Questions & Answers by month

chart showing the Question & Answer count by month

New users by month

chart showing the number of new users by month

I put these two back-to-back because I thought it was interesting how closely the new users count lines up with the number of answers - except in May of 2018. I'd love to know what happened in May to cause that.

Page views by month

Google Analytics was broken for a while before some time in February of 2018, hence the lack of data. chart of page views by month

Visits by new users

enter image description here

Google Analytics data can be a bit wonky and untrustworthy, as I mentioned with the lack of data for a big chunk of time, assuming this is correct, though, it's interesting that it seems to say that people GA considers "new" to the site have stayed stable. Like the relative stability of votes on 30-day-old posts (compared to all posts), this is somewhat expected as the most likely candidates for these new visits are referrals from Google, which haven't be impacted by the removal from the HNQ.

So far there's one thing I notice. While the site took a decided drop in most metrics in October, there was a slight downward trend overall since that big peak in August. You can see that in the graphs Ælis posted in their answer, too. There was one place where there was an encouraging slight increase in activity up until October, though:

Visitors on at least one or five days

Visitors on at least one or five days

The site was building up more visitors in general but also more people who were returning five or more times over the course of a month.

Together, these charts definitely show that there's been a big downturn in activity on the site. Whether this has been good or bad for IPS as a beta site working through some important foundational discussions is something worth thinking about. Yes, there's less traffic but has it allowed you to have some room to breathe (room we'd been asking for since the site went live) to come to some decisions on scope or post quality (etc) or has it just been a loss of traffic. Have you used the four and a half months to iron out some of the concerns?

One more thing before I go... let's look at the effect on comments, specifically, some of the stuff we looked at last spring - comment noise:

Description                     Before     After      PctChange  
------------------------------- ---------- ---------- ---------- 
Comments / post ratio                 3.71       3.05     -17.81 
Flags / comment ratio                 0.31       0.29      -8.23 
Pct comments deleted                 54.62      37.68     -31.01 
Comment upvotes / comment ratio       1.67       0.89     -46.66 
Pct comments followed by edits       13.97      15.05       7.68 

For easy reference, here's the same data from the comment text change:

Description                     Before     After      PctChange  
------------------------------- ---------- ---------- ---------- 
Comments / post ratio                 4.31       3.40     -21.12 
Flags / comment ratio                 0.37       0.30     -20.27 
Pct comments deleted                 51.14      48.89      -4.40 
Comment upvotes / comment ratio       2.05       1.90      -7.21 
Pct comments followed by edits       11.00      14.13      28.38

I think this is pretty big, though not unexpected. There are clearly fewer comments, so fewer need to be deleted, hence the huge drop in comment deletion percentage. There's a lower percentage of flagged comments, too, and a higher percentage of edits after comments. The change in the last of those isn't as big but if you look at the actual percentages rather than the percentage changed, 15.05% is a point higher than the 14.13% from the other experiment.


There are some changes on the horizon - and soon - for the Hot Network Questions and once those changes are implemented, IPS will have the choice of whether y'all want to get back on the list or not. I know you've had a couple of discussions about it already but those are with the assumption that the HNQ is the way it currently is. The changes we're making aren't life-altering but they will address some major issues and give us the ability to learn about the types of questions the HNQ features so that we can further improve the feature moving forward.

Thank you so much for your patience. Let me know if you have any questions or if there's anything I missed.


Thanks and credit goes to Shog for pulling most of the data and helping me understand it.

6

Here are some data that I got from the Site analytics page (>5k users only).

The data are from the 2018-Feb-10 to the 2019-Feb-07 and each point represents a week (not a day).

Data about posts

Graphic of the posts data

Data about votes

Graphic of the votes data

Data about traffic

Graphic of the traffic data

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .