2

This is a question about site statistics.

Background:

I love sports and statistics and I'm well aware of the beneficial ways that 'game theory' has been used on Stack Exchange websites. To standardise sporting statistics for differing volume and duration of career performance, average scores per game are typically considered the index of quality.

I appreciate and am motivated to write better quality posts by seeing how other users build reputation. So I was reviewing users' reputation points recently and was struck by the fact that certain high-reputation members have actually written relatively few answers, but most of those answers were highly upvoted by the community.

It means they were excellent answers, which shows that some users have been contributing more quality with each answer, as measured by the number of upvotes broadly reflected in rep score.

'Average reputation per post' and 'quality impact':

The highest reputation members are obviously making the biggest volume of contribution, usually by writing the most number of answers on IPS. But not all members can devote a lot of time to creating posts here, so many a medium-rep member has actually made a big quality impact with multiple excellent answers.

For any number of users with similar reputation score, I would crudely assess that the user who earned that rep from fewest answers has made the strongest 'impact of contribution' here. To measure this impact I have been considering a crude index called average reputation score per post which is simply calculated as total reputation score divided by total number of posts, where 'post' = questions and answers.

Question:

  1. So is average rep earned per post a good (if crude and simple) statistical index of quality impact contributed by a member's 'body of questions and answers'?

  2. If yes, what is the data query which would yield this calculation as a table for all IPS users who have written at least 8 answers and earned at least 2400 reputation points to date?

  3. Finally, can you think of any other simple index to measure 'strength/impact' of contribution here, apart from total rep score?


Notes:

  1. User @Daniel wrote a brilliantly statistical meta answer on English.SE that gives pointers to the data query approach to calculate averages on IPS.

  2. My own 'average rep score per post' is currently 3602 ÷ 50 = 72, which is quite weak compared to many other members here.

  • 3
    can you think of any other simple index to measure 'strength/value' of contribution : I don't get it. What would this be useful for? What is your point here? – OldPadawan Oct 16 '17 at 9:15
  • This is a question about site statistics. What is the point of reputation? Somebody wrote somewhere on SE that the highest rep members are usually those who have participated most on the site by writing the highest number of answers. So total rep represents volume of contribution. Some people don't have the time. My point is some middle-reputation members who have collected many rep points per post could be understood to be contributing a lot of value on a consistent basis, which would be good to know! Whoever they may be, the quality their work represents is a motivation for other members. – English Student Oct 16 '17 at 9:23
  • Please do not use comments for extended discussion; this conversation has been moved to chat. Please continue your discussion there. Thank you. – Robert Cartaino Oct 17 '17 at 15:16
  • @EnglishStudent it sounds like you're calculating efficiency by rep/post, yes? Hmm ... for impact, have you considered 'weighted rep', that is reputation * views calculated on a per-page basis. That might represent the number of people you've impressed with your answer. Must retreat and think on this. – akaioi Oct 20 '17 at 5:41
  • Page views is confusing because I found some pages are viewed more than others mainly depending on the topic and not the quality of the posts. Political and sensational topics get more eyeballs. A niche topic or an Asian cultural context gets less views. However a SE expert has suggested page views as a measure of impact. Please think over all angles at your convenience and all your suggestions are most welcome @akaioi! – English Student Oct 20 '17 at 5:44
  • Honestly I'm not sure what purpose these metrics would serve anyhow? The gamification feels nice at times, but we already have a whole lot of this stuff from rep, to badges, helpful flags, edits, views, etc... Like we already have a never ending positive reinforcement system in place that encourages people to interact with the system and the community. What purpose would yet another metric serve? – apaul Oct 21 '17 at 4:23
  • @apaul a little deja-vu going on ;D Short-short: extra stats are just fun, like creating our own badges. For english-student I put some SQL below. – akaioi Oct 21 '17 at 4:31
  • @akaioi I just wanted to ping English Student on that as well. – apaul Oct 21 '17 at 4:33
  • I guess I'd rather see people participating because they enjoy the subject matter and want to help someone? Like the ego stroking of having your pov accepted or upvoted feels nice, but I guess I would prefer people get involved and stay involved for more altruistic reasons. – apaul Oct 21 '17 at 4:39
  • Quite apart from reputation reward, upvoting/downvoting and acceptance of answers also constitute useful feedback @apaul. – English Student Oct 21 '17 at 6:13
  • @apaul I'm not seeing a contradiction. People won't come and post here just for internet points & badges... there are 170 other SE sites. The bells, whistles, and insane statistical analysis just make it more fun. There are a few badges (populist et al) which do make one stop and ponder for a moment. – akaioi Oct 22 '17 at 2:03
  • @EnglishStudent recommend that once you have settled on the stats you really want, you start a new question whose answers could be the SQL statements to fetch the data. I'm willing to pitch in. – akaioi Oct 22 '17 at 20:01
  • Good point, @akaioi! I am still studying the matter and hope to get more community inputs here. Then that SQL question is a really good idea. – English Student Oct 22 '17 at 20:02
1

Disclaimer: This answer only looks at questions and answers and ignores other aspects of a user's impact, such as votes, comments, many edits, chat-usage, flags, Area 51 proposal participation and of course other (often less visible) moderation as well as whatever else I've forgotten to mention.

What comes to mind when the word impact is used?

The simple 'definition' given by Google "a marked effect or influence" is hardly much use, so searching for the nearest dictionary gives (in noun form)

a powerful effect that something, especially something new, has on a situation or person.

So, in this context, "the effect of a user on the site and other users of the site" is a reasonably solid definition to begin with.

There's clearly two different parts to the above, so let's try splitting things up a bit: we have "impact on the site" and "impact on other users" as two different types of impact. Conveniently, site discussions are held on the relevant Meta, while issues affecting users are, well, put on the main site as questions. This helpfully gives an impact [on other users] and a meta-impact [affecting the site].

These are a little better, although the best way to measure these impacts still seems very difficult and subjective...

Measuring the subjective

Before getting into technical details of how to measure impact, we still need to figure out what to measure that gives some sort of number at the end. As such, if we want a number, and the aim is to improve our own questions and answers, then (as mentioned in the disclaimer) a number of metrics have to be ignored. This does narrow down what we'll end up with when we get a number for 'impact' (especially meta-impact), but does make computing things easier as well as put the focus on question and answer quality, which is the ultimate goal in the case.

So, how do we measure question and answer impact? Again, Stack Exchange is very helpful in that it already has some measures for this. Reputation and score (upvotes-downvotes) are particularly good and easy to find measures of question and answer impact. These can also be adapted to get relative reputation and score, where an answer is compared with all other answers given to a question; or a question/answer is compared with all other questions/answers in a given tag used in the question (or even the whole site). In more detail, these question/answer impacts are:

  • Reputation: Explained in this help post. Divide this by 10. (this allows for easier comparisons with score as well as calculating the impact of a user, as explained below)
  • Score: +1 for every upvote, -1 for every downvote
  • Relative reputation/score: reputation of that question/answer over (sum of reputations of questions/answers over total number of questions/answers)
  • Number of views: Only for questions, although answer impacts can also be scaled by number of views of the question

While this may initially seem a bit excessive, the usefulness of using all of these is that reputation and score can be compared to find controversial questions and answers (high rep, low score) as well as allowing for comparisons between tags or even stack exchange sites using relative reputation or score to measure impact.

Having now got a method to measure the impact of single questions and answers (as well as meta-impact of the same on, well, Meta sites), we need to turn this into impact of a user.

If you look around for inspiration on how to measure the impact of someone, it won't be long before you come across impact in academia, which has been around for a lot longer than Stack Exchange and so has some reasonably good methods for measuring this:

The Impact (and meta-impact!) of a user

While there are a number of methods used to calculate the impact of a person, three more common ones are:

  • (average) reputation/score count: sum of question/answer impact. Disadvantage: It's an average, so is strongly effected by extremal values (such as a few lucky posts)
  • h-index: h questions and answers with a question/answer impact greater than h. Disadvantage of taking a while to build up (a low h-index can mean either few posts, such as a new user would have; or few good posts)
  • i-10 index (the 'nice' index): Number of questions and answers with a question/answer impact greater than 10 (i.e. with a nice question/nice answer badge). Naturally extends to the 'good' and 'great' index. Disadvantage of the number 10 being somewhat arbitrary, so may exclude some or include too many people.

Each of the above measures can then be used, along with the relative versions of each, which helps differentiate between those new users with a few really good questions and answers and any who have just been here for a long time and perhaps got lucky a number of times.

This gives a large number of ways to measure a user's impact. Individually, they measure fairly similar things, with average versions being good for comparing between tags and users and comparisons between rep and score being useful for looking at more controversial posts as well as those that were popular enough to rep-cap. None of these a perfect and all have slightly different disadvantages, so all (or at least, many) need to be looked at for a more detailed picture.

Of course, the nice/good/great index brings another measure in mind: the number of bronze/silver/gold badges. These three additions partially help avoid the problems of ignoring everything in the disclaimer, although as such, reduce the importance of the impact of questions and answers, which is what we're trying to measure with the aim of improving. There's also the straightforward total reputation and proportion of accepted answers.

That's quite abstract

We've gathered different ways of measuring the impact, both within a question and of a user. Of course, the best way to understand this is with an example, so (as this is my first post on IPS) I'll use my profile on Worldbuilding to calculate some impacts as an example:

As there are only 6 questions tagged Martial-Arts, let's look at What would be required for a large mammal to realistically throw its own kind a significant distance? This one is reasonably straightforward - rep and score impacts of 3, relative to the question of 5*7/11 = ~2.9, relative to the tag of 5*48/480 = 0.5 (Hmm...)

On a bigger scale, I have 17 answers (3 accepted), no downvotes (at the minute anyway!) and 154 upvotes which gives answer impacts of: rep: 158.5; score: 154; average rep: ~9.3; average score: ~9.1; h-index: 7; nice-index: 5; average-nice-index: 5/17 ~0.29; good-index: 1; average-good-index: 1/17 ~0.06; great-index: 0.

This is considerably better than my equivalent impacts on physics (Average-rep-impact of 2...) and roughly comparable to my impacts on Sci-Fi and Fantasy.

Of course, all of this is for the purposes of fun/learning statistics/improving question and answer quality and needs some serious analysis to see if it's of any real use. One actual use for this could be to compare differences in voting on different stack exchange sites as well as across different tags on one site.

  • Thank you so much for expertly giving us a lot to think about @Mithrandir24601 -- also for the information that other SE sites are interested in such indices. I need to read a lot more about it and will get back to you if I have any more questions. – English Student Oct 17 '17 at 1:06
  • I now realized I'm not really looking at the impact of a high reputation member, because high reputation is its own impact that is indisputable.What I am looking for is a tool to identify the low-medium rep user or users with a disproportionately high impact compared to other users at their reputation level (though it was your answer and my intensive discussion with 3 other members that helped me understand this point.) Why? Because such work is model & inspirational, but also because we need to nurture such high impact users and encourage them to contribute more high quality posts here! – English Student Oct 17 '17 at 5:18
  • @EnglishStudent In that case, you'll want to take the average impacts of users below a cut-off rep (to take out high-rep users) and compare that with tag/site-wide/high-rep user averages. If the average impacts of a low+mid-rep user are higher, then you've found yourself someone with a disproportionately high impact! – Mithrandir24601 Oct 17 '17 at 8:42
  • Thanks, I will try that out @Mithrandir24601. – English Student Oct 17 '17 at 8:44
0

You have to decide whom you want to impact, first off...

The main metric I'm interested in is which answer OP drops a comment on saying "Thanks, this helped with my problem!" Hey, there is a feature request lurking in there somewhere. [Edit: Wiser heads than mine remind me of the accept button...]

I mean, why do you post? Is it to show off your mad IP skills? Is it to read insightful answers? Is it to read clever, funny comments? Is it to help OP? Is it to minmax rep? That's a question worth answering, if just to oneself.

  • "I mean, why do you post?" __ It's meant to help OP and future readers @akaioi. If OP feels that a particular answer best helped their situation then they are encouraged to signal their choice to the community by accepting that answer as I informed/reminded a new OP at the end of my answer to this question: interpersonal.stackexchange.com/a/5627/381 -- so thanks for giving me one other metric to consider for impact: average number of accepted answers per post! – English Student Oct 20 '17 at 4:58
  • @EnglishStudent where is my head today? OP-acceptance is what I wanted, I somehow forgot that that button was there... ;D – akaioi Oct 20 '17 at 5:03
  • Average number of accepted answers per post is a really impactful metric that we somehow missed before your this answer; +1 @akaioi! – English Student Oct 20 '17 at 5:05
0

Let's look at some more things we can measure

I'd mentioned already that the average measures acceptances/post rep/post etc really measure efficiency rather than impact. I'll suggest that if the goal is to winkle out and highlight mid-rep users who have contributed value without getting swamped by the huge-rep users, we might consider the following:

  • Calculate rep/time and acceptances/time. That is, how many reps (ha!) per day is someone contributing. Make sure you don't include time before the person joined! Note how I'm trying to divorce total impact from efficiency. I'm emphasizing the guy who swings and misses a lot, but also knocks a lot out of the park.

  • Calculate rep/time and acceptances/time again, now take the standard deviation. Now you can identify bands of contribution. You can highlight who is close to the average, who is >1 std deviation above average, >2, etc.

So that gets you the all-time hall of fame. Not bad, not bad. But you know the famous question... "so what have you done for us lately?" You can take these same measurements but restrict down to a specific month or week. (Note SE tracks high gross rep earners per time period like this).

Okay, you have got all this data. Cool, we all like data. How to share it? Maybe a special post (like that happy 100 days business) every time-period which has a report on results.

Hey, here's one real simple thing

Per my obsessive focus on acceptances, another nice thing might be a monthly report: "Thank you User-X, User-Y, User-Z. All of you have had an answer accepted during the month of Somethingember!"

Logistics

Why didn't you guys tell me it was straight-up SQL? Sheesh! We can have some fun with this. Here's a free one; who are the people who had the most questions accepted this past September...

select 
     u.DisplayName, count(u.DisplayName) cc
from
    Posts p
inner join
    Posts answers on answers.id = p.AcceptedAnswerId  /* answers.x is the accepted answer*/
inner join
    Users u on u.id = answers.OwnerUserId
where p.PostTypeId = 1  /*  a question */
and p.AcceptedAnswerId is not null
and ( 
    /*  Only for this September */
    datepart(year, answers.CreationDate) =2017 
    and datepart(month, answers.CreationDate) =9 )
group by u.DisplayName
order by cc desc
  • 2
    Eh, I'm not sure that accepted answers are a terribly useful thing to measure, they only indicate that the OP found the post helpful. Often times the highest voted answer is not the accepted answer, and worse sometimes the accepted answer is only accepted because it told the OP what they wanted to hear. – apaul Oct 21 '17 at 4:02
  • @apaul this hearkens back to the "are we Dear Abby" from another Q ... I like the count of accepted answers because it's the closest thing I could find to measuring the answers that actually helped someone with a problem. I recognize the danger you point out, but imagine that at some point we have to trust OP to know good advice when he sees it. Oh hey, you're near the top of September's list, btw... ;D – akaioi Oct 21 '17 at 4:07
  • Also it doesn't account for all the questions with no accepted answer. To use the baseball analogy, it wouldn't account for the total games played, because many games never technically ended. – apaul Oct 21 '17 at 4:13
  • @apaul I see where you're coming from, but tend to disagree. I'm suggesting that answers whose question's OP took the effort to hit the accept button count for something. I'm less interested in some of the other metrics like efficiency ... that seems more IPSE-serving than OP-serving, if you catch my drift. Anyhoo, English Student appears to be running this stats show, I'm hoping to give him some options. – akaioi Oct 21 '17 at 4:17
  • Honestly I'm not sure what purpose these metrics would serve anyhow? The gamification feels nice at times, but we already have a whole lot of this stuff from rep, to badges, helpful flags, edits, views, etc... Like we already have a never ending positive reinforcement system in place that encourages people to interact with the system and the community. What purpose would yet another metric serve? – apaul Oct 21 '17 at 4:21
  • @apaul oh as to that ... it serves the same purpose as the other bells and whistles, badges etc -- makes it perhaps a bit more fun for question-answerers. Think of it as us being able to define our own badges. Plus [shameful confession time] ... I like writing SQL statements. – akaioi Oct 21 '17 at 4:25
  • That is shameful, although I'm glad someone enjoys it so I don't have to. – apaul Oct 21 '17 at 4:26
  • Great insights, thanks @ akaioi -- you helped me understand that 'average reputation score per answer' is a measure of efficiency rather than impact -- it reflects good selection if questions to answer, as earlier noted by @OldPadawan, and I could easily boost my average simply by deleting 10 answers that scored no upvotes! Accepted answers ratio to total answers is relevant. In short I learned that 'impact' is a mysteriously complex thing to measure, and shall be reading these 4 excellent answers together to get a better understanding of which parameters would make up the 'perfect recipe.' – English Student Oct 21 '17 at 5:58
  • "Honestly I'm not sure what purpose these metrics would serve anyhow?" -- Whenever I see huge rep scores on SE sites I think, what a lot of helpful activity! What about the users who don't really have that much time? What about users who have joined relatively late? Their names won't show up in the top 20 list of major websites or the top 10 of new growing sites like IPS. High rep is big impact so I don't need to apply these statistical indices to high rep users. But who are the big-impact users in the lower-middle segment? Perhaps it matters only to statistically minded meta users @apaul. – English Student Oct 21 '17 at 6:07
-2

So is average rep earned per post a good (if crude and simple) statistical index of quality impact contributed by a member's 'body of questions and answers'?

Here's an incomplete list of the factors that determine how much fake internet points a post generates:

  1. Questions that get more attention (e.g. through the HNQ) get more votes and thus more points.
  2. Even if a post gets a lot of downvotes, it can still generate positive points for the person writing it. For example, if an answer has been downvoted four times and upvoted once, that writer of that answer gains 2 points, even though the answer has a score of -3.
  3. Points don't take into account people without accounts on the site who find the content useful (e.g. visitors from Google).
  4. Using points as a measure of quality assumes that the opinions of the people voting are a good measure of quality. This isn't always the case.

Points work as a way to decide who gets access to privileges such as closing questions. Points also work as a number to sum up someone's activity on Stack Exchange. The caveat here is that other numbers, such as the number of posts someone writes, would probably work OK--not as good as points of course, but not badly. And of course, points and badges do have an aspect of "gamification" to them. As someone who has turned off the points/badges notification, I still feel the same gamification effect when I get a notification telling me that someone responded to my comments, so points and badges aren't the only source of gamification, but gamification is obviously something that they do.

But other than those three things, points are a lot less important than people make them out to be.

There's also a very weird tendency to view points as the Stack Exchange equivalent to money, e.g. they're the economic system that motivates contributions. Let me assure you that points are very different from money in some very important aspects. (Here's a hint: can you buy anything with points? Here's another hint: are points subject to things like inflation?)

Now the real question is, if not average rep per post, then what is a good index of user impact @Hamlet?

If you're looking for an objective measure of user impact there isn't one. Determining someone's impact is a subjective question. You can make up a number, but the number is still subjective. For example, if you are using points as a number, then you are making the subjective decision that the opinions of anonymous users, or the opinions of users that don't have the voting privilege, aren't useful in calculating user impact. When coming up with a numerical measure of something subjective, you have to make subjective decisions about how to calculate that number, so your numerical measure is also subjective.

Just because you're using a number doesn't mean you're being objective.

other than not being able to vote, and thus reducing sample size, do you have any reason to believe that anonymous visitors and non-voting members would substantially change the assessment of a member's entire body of work individually posted and voted upon over weeks, months and years?

Note that I used the fact that points don't take into account the opinions of anonymous users as an example of the assumptions that go into the concept of points. There are many assumptions that go into points; see if you can spot them (I talk about at least four assumptions at various points during this answer). But sure, I'll answer the question (but unfortunately this will have to be my last response).

Imagine a question that gets an answer that sounds like a plausible solution to a problem. It gets upvoted. The question starts showing up in Google results and people with the same problem arrive from Google. But the people coming from Google leave because the solution doesn't work. The people who upvoted it did so because the answer was plausible, but because they didn't have experience with the problem, they didn't know that the answer was plausible but incorrect.

  • Thanks @Hamlet. Please note that these variable factors that can affect the score of individual posts is exactly why we're looking at the average score per post, which is a crude randomization. If a member has written at least 8 main site answers and scored at least 2400 points I think effects of random variables will begin to cancel out. The higher the total rep & number of posts, the more randomized will be the index. So any user who scored 4000 points over 40 posts (say 30 Q and 10 A) has made a greater quality impact than a 4500 rep member with 100 posts. Please tell me a better idea! – English Student Oct 16 '17 at 13:15
  • @EnglishStudent that is not how averages work. If you have one post with 200 fake points and five posts with 10 fake points, then the 200 fake point post is going to skew that average. – user288 Oct 16 '17 at 13:18
  • That is certainly true, @Hamlet. Extreme variations in score disturb statistical treatment. But we will be made aware of it just by looking at the profile page. – English Student Oct 16 '17 at 13:19
  • @EnglishStudent And of course, fake points are not a measure of quality impact: they often just mean that some drive by voters from the HNQ found the content interesting. So someone with a higher fake points per post measure is usually just someone good at playing the HNQ game. That's all you're measuring here. – user288 Oct 16 '17 at 13:20
  • How many (% proportion) of a person's posts usually make the HNQ @Hamlet? – English Student Oct 16 '17 at 13:21
  • @EnglishStudent that would actually be an interesting statistic that would tell you something useful. – user288 Oct 16 '17 at 13:22
  • In my experience only a few of my 50 posts ever got on the HNQ and only a few of those netted me a lot of upvotes. I think that effect will get largely cancelled out by all the regular posts. Many highly upvoted answers on IPS have come on Q that didn't reach HNQ. Moreover, how does a user actually play the HNQ game? Now the real question is, if not average rep per post, then what is a good index of user impact @Hamlet? – English Student Oct 16 '17 at 13:27
  • I completely agree that we have no reason to see fake internet points as any equivalent of money @Hamlet, and I would look at it more in terms of giving people virtual status, prestige and influence; though the rep system does give an interested user access to various voting and moderator tools of which the most (over)zealously utilised seems to be close-voting! – English Student Oct 16 '17 at 13:31
  • You've been continuously improving this answer, thanks @Hamlet. "if you are using fake internet points as a number, then you are making the subjective decision that the opinions of anonymous users, or the opinions of users that don't have the voting privilege, aren't useful in calculating user impact." __ other than not being able to vote, and thus reducing sample size, do you have any reason to believe that anonymous visitors and non-voting members would substantially change the assessment of a member's entire body of work individually posted and voted upon over weeks, months and years? – English Student Oct 16 '17 at 13:41
  • @EnglishStudent "You've been continuously improving this answer" then why does it have a score of negative 2? #reputationIsAnObjectiveMeasureOfQuality – user288 Oct 16 '17 at 13:45
  • @EnglishStudent note that the assumption that anonymous users opinion's aren't useful is only one of the many assumptions that goes into the concept of fake internet points. That assumption was the one that I happened to use as an example. – user288 Oct 16 '17 at 13:47
  • 4
    Your overuse of fake internet points is over 9000!. – NVZ Oct 16 '17 at 13:51
  • 5
    Do you have to always be so controversial? Is it really necessary to repeatedly call reputation "fake internet points"? I get it. You think reputation is stupid. There's no need to smash us on the head about it repeatedly. – Catija Oct 16 '17 at 14:43
  • 4
    I'd like to point something out... when you argue in a way that is so overt about feeling one way or another, people who disagree with you stop reading. So, people who have fun collecting reputation decide that you just don't like reputation and assume that the rest of your answer is invalid or incorrect simply because you don't like reputation. If you take a more neutral approach, more people will read what are, in general, valid points. – Catija Oct 16 '17 at 14:54
  • 2
    +1 @Catija : If you take a more neutral approach, more people will read what are, in general, valid points. ... which seems, at least in my POV, a very wise approach when it comes to IPS. Where being very controversial in order to argue leads to... to... uhm... to... (shrug) – OldPadawan Oct 16 '17 at 15:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .