Is average reputation score per post a good index of 'impact of contribution' on IPS.SE? What else is, apart from total score?

Question

This is a question about site statistics.

Background:

I love sports and statistics and I'm well aware of the beneficial ways that 'game theory' has been used on Stack Exchange websites. To standardise sporting statistics for differing volume and duration of career performance, average scores per game are typically considered the index of quality.

I appreciate and am motivated to write better quality posts by seeing how other users build reputation. So I was reviewing users' reputation points recently and was struck by the fact that certain high-reputation members have actually written relatively few answers, but most of those answers were highly upvoted by the community.

It means they were excellent answers, which shows that some users have been contributing more quality with each answer, as measured by the number of upvotes broadly reflected in rep score.

'Average reputation per post' and 'quality impact':

The highest reputation members are obviously making the biggest volume of contribution, usually by writing the most number of answers on IPS. But not all members can devote a lot of time to creating posts here, so many a medium-rep member has actually made a big quality impact with multiple excellent answers.

For any number of users with similar reputation score, I would crudely assess that the user who earned that rep from fewest answers has made the strongest 'impact of contribution' here. To measure this impact I have been considering a crude index called average reputation score per post which is simply calculated as total reputation score divided by total number of posts, where 'post' = questions and answers.

Question:

1. So is average rep earned per post a good (if crude and simple) statistical index of quality impact contributed by a member's 'body of questions and answers'?

2. If yes, what is the data query which would yield this calculation as a table for all IPS users who have written at least 8 answers and earned at least 2400 reputation points to date?

3. Finally, can you think of any other simple index to measure 'strength/impact' of contribution here, apart from total rep score?

---

Notes:

1. User @Daniel wrote a brilliantly statistical meta answer on English.SE that gives pointers to the data query approach to calculate averages on IPS.

2. My own 'average rep score per post' is currently 3602 ÷ 50 = 72, which is quite weak compared to many other members here.

Answer 1

Disclaimer: This answer only looks at questions and answers and ignores other aspects of a user's impact, such as votes, comments, many edits, chat-usage, flags, Area 51 proposal participation and of course other (often less visible) moderation as well as whatever else I've forgotten to mention.

What comes to mind when the word impact is used?

The simple 'definition' given by Google "a marked effect or influence" is hardly much use, so searching for the nearest dictionary gives (in noun form)

a powerful effect that something, especially something new, has on a situation or person.

So, in this context, "the effect of a user on the site and other users of the site" is a reasonably solid definition to begin with.

There's clearly two different parts to the above, so let's try splitting things up a bit: we have "impact on the site" and "impact on other users" as two different types of impact. Conveniently, site discussions are held on the relevant Meta, while issues affecting users are, well, put on the main site as questions. This helpfully gives an impact [on other users] and a meta-impact [affecting the site].

These are a little better, although the best way to measure these impacts still seems very difficult and subjective...

Measuring the subjective

Before getting into technical details of how to measure impact, we still need to figure out what to measure that gives some sort of number at the end. As such, if we want a number, and the aim is to improve our own questions and answers, then (as mentioned in the disclaimer) a number of metrics have to be ignored. This does narrow down what we'll end up with when we get a number for 'impact' (especially meta-impact), but does make computing things easier as well as put the focus on question and answer quality, which is the ultimate goal in the case.

So, how do we measure question and answer impact? Again, Stack Exchange is very helpful in that it already has some measures for this. Reputation and score (upvotes-downvotes) are particularly good and easy to find measures of question and answer impact. These can also be adapted to get relative reputation and score, where an answer is compared with all other answers given to a question; or a question/answer is compared with all other questions/answers in a given tag used in the question (or even the whole site). In more detail, these question/answer impacts are:

• Reputation: Explained in this help post. Divide this by 10. (this allows for easier comparisons with score as well as calculating the impact of a user, as explained below)
• Score: +1 for every upvote, -1 for every downvote
• Relative reputation/score: reputation of that question/answer over (sum of reputations of questions/answers over total number of questions/answers)
• Number of views: Only for questions, although answer impacts can also be scaled by number of views of the question

While this may initially seem a bit excessive, the usefulness of using all of these is that reputation and score can be compared to find controversial questions and answers (high rep, low score) as well as allowing for comparisons between tags or even stack exchange sites using relative reputation or score to measure impact.

Having now got a method to measure the impact of single questions and answers (as well as meta-impact of the same on, well, Meta sites), we need to turn this into impact of a user.

If you look around for inspiration on how to measure the impact of someone, it won't be long before you come across impact in academia, which has been around for a lot longer than Stack Exchange and so has some reasonably good methods for measuring this:

The Impact (and meta-impact!) of a user

While there are a number of methods used to calculate the impact of a person, three more common ones are:

• (average) reputation/score count: sum of question/answer impact. Disadvantage: It's an average, so is strongly effected by extremal values (such as a few lucky posts)
• h-index: h questions and answers with a question/answer impact greater than h. Disadvantage of taking a while to build up (a low h-index can mean either few posts, such as a new user would have; or few good posts)
• i-10 index (the 'nice' index): Number of questions and answers with a question/answer impact greater than 10 (i.e. with a nice question/nice answer badge). Naturally extends to the 'good' and 'great' index. Disadvantage of the number 10 being somewhat arbitrary, so may exclude some or include too many people.

Each of the above measures can then be used, along with the relative versions of each, which helps differentiate between those new users with a few really good questions and answers and any who have just been here for a long time and perhaps got lucky a number of times.

This gives a large number of ways to measure a user's impact. Individually, they measure fairly similar things, with average versions being good for comparing between tags and users and comparisons between rep and score being useful for looking at more controversial posts as well as those that were popular enough to rep-cap. None of these a perfect and all have slightly different disadvantages, so all (or at least, many) need to be looked at for a more detailed picture.

Of course, the nice/good/great index brings another measure in mind: the number of bronze/silver/gold badges. These three additions partially help avoid the problems of ignoring everything in the disclaimer, although as such, reduce the importance of the impact of questions and answers, which is what we're trying to measure with the aim of improving. There's also the straightforward total reputation and proportion of accepted answers.

That's quite abstract

We've gathered different ways of measuring the impact, both within a question and of a user. Of course, the best way to understand this is with an example, so (as this is my first post on IPS) I'll use my profile on Worldbuilding to calculate some impacts as an example:

As there are only 6 questions tagged Martial-Arts, let's look at What would be required for a large mammal to realistically throw its own kind a significant distance? This one is reasonably straightforward - rep and score impacts of 3, relative to the question of 5*7/11 = ~2.9, relative to the tag of 5*48/480 = 0.5 (Hmm...)

On a bigger scale, I have 17 answers (3 accepted), no downvotes (at the minute anyway!) and 154 upvotes which gives answer impacts of: rep: 158.5; score: 154; average rep: ~9.3; average score: ~9.1; h-index: 7; nice-index: 5; average-nice-index: 5/17 ~0.29; good-index: 1; average-good-index: 1/17 ~0.06; great-index: 0.

This is considerably better than my equivalent impacts on physics (Average-rep-impact of 2...) and roughly comparable to my impacts on Sci-Fi and Fantasy.

Of course, all of this is for the purposes of fun/learning statistics/improving question and answer quality and needs some serious analysis to see if it's of any real use. One actual use for this could be to compare differences in voting on different stack exchange sites as well as across different tags on one site.

Answer 2

You have to decide whom you want to impact, first off...

The main metric I'm interested in is which answer OP drops a comment on saying "Thanks, this helped with my problem!" Hey, there is a feature request lurking in there somewhere. [Edit: Wiser heads than mine remind me of the accept button...]

I mean, why do you post? Is it to show off your mad IP skills? Is it to read insightful answers? Is it to read clever, funny comments? Is it to help OP? Is it to minmax rep? That's a question worth answering, if just to oneself.

Answer 3

Let's look at some more things we can measure

I'd mentioned already that the average measures acceptances/post rep/post etc really measure efficiency rather than impact. I'll suggest that if the goal is to winkle out and highlight mid-rep users who have contributed value without getting swamped by the huge-rep users, we might consider the following:

• Calculate rep/time and acceptances/time. That is, how many reps (ha!) per day is someone contributing. Make sure you don't include time before the person joined! Note how I'm trying to divorce total impact from efficiency. I'm emphasizing the guy who swings and misses a lot, but also knocks a lot out of the park.

• Calculate rep/time and acceptances/time again, now take the standard deviation. Now you can identify bands of contribution. You can highlight who is close to the average, who is >1 std deviation above average, >2, etc.

So that gets you the all-time hall of fame. Not bad, not bad. But you know the famous question... "so what have you done for us lately?" You can take these same measurements but restrict down to a specific month or week. (Note SE tracks high gross rep earners per time period like this).

Okay, you have got all this data. Cool, we all like data. How to share it? Maybe a special post (like that happy 100 days business) every time-period which has a report on results.

Hey, here's one real simple thing

Per my obsessive focus on acceptances, another nice thing might be a monthly report: "Thank you User-X, User-Y, User-Z. All of you have had an answer accepted during the month of Somethingember!"

Logistics

Why didn't you guys tell me it was straight-up SQL? Sheesh! We can have some fun with this. Here's a free one; who are the people who had the most questions accepted this past September...

select 
     u.DisplayName, count(u.DisplayName) cc
from
    Posts p
inner join
    Posts answers on answers.id = p.AcceptedAnswerId  /* answers.x is the accepted answer*/
inner join
    Users u on u.id = answers.OwnerUserId
where p.PostTypeId = 1  /*  a question */
and p.AcceptedAnswerId is not null
and ( 
    /*  Only for this September */
    datepart(year, answers.CreationDate) =2017 
    and datepart(month, answers.CreationDate) =9 )
group by u.DisplayName
order by cc desc

Answer 4

So is average rep earned per post a good (if crude and simple) statistical index of quality impact contributed by a member's 'body of questions and answers'?

Here's an incomplete list of the factors that determine how much fake internet points a post generates:

1. Questions that get more attention (e.g. through the HNQ) get more votes and thus more points.
2. Even if a post gets a lot of downvotes, it can still generate positive points for the person writing it. For example, if an answer has been downvoted four times and upvoted once, that writer of that answer gains 2 points, even though the answer has a score of -3.
3. Points don't take into account people without accounts on the site who find the content useful (e.g. visitors from Google).
4. Using points as a measure of quality assumes that the opinions of the people voting are a good measure of quality. This isn't always the case.

Points work as a way to decide who gets access to privileges such as closing questions. Points also work as a number to sum up someone's activity on Stack Exchange. The caveat here is that other numbers, such as the number of posts someone writes, would probably work OK--not as good as points of course, but not badly. And of course, points and badges do have an aspect of "gamification" to them. As someone who has turned off the points/badges notification, I still feel the same gamification effect when I get a notification telling me that someone responded to my comments, so points and badges aren't the only source of gamification, but gamification is obviously something that they do.

But other than those three things, points are a lot less important than people make them out to be.

There's also a very weird tendency to view points as the Stack Exchange equivalent to money, e.g. they're the economic system that motivates contributions. Let me assure you that points are very different from money in some very important aspects. (Here's a hint: can you buy anything with points? Here's another hint: are points subject to things like inflation?)

Now the real question is, if not average rep per post, then what is a good index of user impact @Hamlet?

If you're looking for an objective measure of user impact there isn't one. Determining someone's impact is a subjective question. You can make up a number, but the number is still subjective. For example, if you are using points as a number, then you are making the subjective decision that the opinions of anonymous users, or the opinions of users that don't have the voting privilege, aren't useful in calculating user impact. When coming up with a numerical measure of something subjective, you have to make subjective decisions about how to calculate that number, so your numerical measure is also subjective.

Just because you're using a number doesn't mean you're being objective.

other than not being able to vote, and thus reducing sample size, do you have any reason to believe that anonymous visitors and non-voting members would substantially change the assessment of a member's entire body of work individually posted and voted upon over weeks, months and years?

Note that I used the fact that points don't take into account the opinions of anonymous users as an example of the assumptions that go into the concept of points. There are many assumptions that go into points; see if you can spot them (I talk about at least four assumptions at various points during this answer). But sure, I'll answer the question (but unfortunately this will have to be my last response).

Imagine a question that gets an answer that sounds like a plausible solution to a problem. It gets upvoted. The question starts showing up in Google results and people with the same problem arrive from Google. But the people coming from Google leave because the solution doesn't work. The people who upvoted it did so because the answer was plausible, but because they didn't have experience with the problem, they didn't know that the answer was plausible but incorrect.