Disclaimer: This answer only looks at questions and answers and ignores other aspects of a user's impact, such as votes, comments, many edits, chat-usage, flags, Area 51 proposal participation and of course other (often less visible) moderation as well as whatever else I've forgotten to mention.
What comes to mind when the word impact is used?
The simple 'definition' given by Google "a marked effect or influence" is hardly much use, so searching for the nearest dictionary gives (in noun form)
a powerful effect that something, especially something new, has on a situation or person.
So, in this context, "the effect of a user on the site and other users of the site" is a reasonably solid definition to begin with.
There's clearly two different parts to the above, so let's try splitting things up a bit: we have "impact on the site" and "impact on other users" as two different types of impact. Conveniently, site discussions are held on the relevant Meta, while issues affecting users are, well, put on the main site as questions. This helpfully gives an impact [on other users] and a meta-impact [affecting the site].
These are a little better, although the best way to measure these impacts still seems very difficult and subjective...
Measuring the subjective
Before getting into technical details of how to measure impact, we still need to figure out what to measure that gives some sort of number at the end. As such, if we want a number, and the aim is to improve our own questions and answers, then (as mentioned in the disclaimer) a number of metrics have to be ignored. This does narrow down what we'll end up with when we get a number for 'impact' (especially meta-impact), but does make computing things easier as well as put the focus on question and answer quality, which is the ultimate goal in the case.
So, how do we measure question and answer impact? Again, Stack Exchange is very helpful in that it already has some measures for this. Reputation and score (upvotes-downvotes) are particularly good and easy to find measures of question and answer impact. These can also be adapted to get relative reputation and score, where an answer is compared with all other answers given to a question; or a question/answer is compared with all other questions/answers in a given tag used in the question (or even the whole site). In more detail, these question/answer impacts are:
- Reputation: Explained in this help post. Divide this by 10. (this allows for easier comparisons with score as well as calculating the impact of a user, as explained below)
- Score: +1 for every upvote, -1 for every downvote
- Relative reputation/score: reputation of that question/answer over (sum of reputations of questions/answers over total number of questions/answers)
- Number of views: Only for questions, although answer impacts can also be scaled by number of views of the question
While this may initially seem a bit excessive, the usefulness of using all of these is that reputation and score can be compared to find controversial questions and answers (high rep, low score) as well as allowing for comparisons between tags or even stack exchange sites using relative reputation or score to measure impact.
Having now got a method to measure the impact of single questions and answers (as well as meta-impact of the same on, well, Meta sites), we need to turn this into impact of a user.
If you look around for inspiration on how to measure the impact of someone, it won't be long before you come across impact in academia, which has been around for a lot longer than Stack Exchange and so has some reasonably good methods for measuring this:
The Impact (and meta-impact!) of a user
While there are a number of methods used to calculate the impact of a person, three more common ones are:
- (average) reputation/score count: sum of question/answer impact. Disadvantage: It's an average, so is strongly effected by extremal values (such as a few lucky posts)
- h-index: h questions and answers with a question/answer impact greater than h. Disadvantage of taking a while to build up (a low h-index can mean either few posts, such as a new user would have; or few good posts)
- i-10 index (the 'nice' index): Number of questions and answers with a question/answer impact greater than 10 (i.e. with a nice question/nice answer badge). Naturally extends to the 'good' and 'great' index. Disadvantage of the number 10 being somewhat arbitrary, so may exclude some or include too many people.
Each of the above measures can then be used, along with the relative versions of each, which helps differentiate between those new users with a few really good questions and answers and any who have just been here for a long time and perhaps got lucky a number of times.
This gives a large number of ways to measure a user's impact. Individually, they measure fairly similar things, with average versions being good for comparing between tags and users and comparisons between rep and score being useful for looking at more controversial posts as well as those that were popular enough to rep-cap. None of these a perfect and all have slightly different disadvantages, so all (or at least, many) need to be looked at for a more detailed picture.
Of course, the nice/good/great index brings another measure in mind: the number of bronze/silver/gold badges. These three additions partially help avoid the problems of ignoring everything in the disclaimer, although as such, reduce the importance of the impact of questions and answers, which is what we're trying to measure with the aim of improving. There's also the straightforward total reputation and proportion of accepted answers.
That's quite abstract
We've gathered different ways of measuring the impact, both within a question and of a user. Of course, the best way to understand this is with an example, so (as this is my first post on IPS) I'll use my profile on Worldbuilding to calculate some impacts as an example:
As there are only 6 questions tagged Martial-Arts, let's look at What would be required for a large mammal to realistically throw its own kind a significant distance? This one is reasonably straightforward - rep and score impacts of 3, relative to the question of 5*7/11 = ~2.9, relative to the tag of 5*48/480 = 0.5 (Hmm...)
On a bigger scale, I have 17 answers (3 accepted), no downvotes (at the minute anyway!) and 154 upvotes which gives answer impacts of:
rep: 158.5; score: 154; average rep: ~9.3; average score: ~9.1;
h-index: 7; nice-index: 5; average-nice-index: 5/17 ~0.29; good-index: 1; average-good-index: 1/17 ~0.06; great-index: 0.
This is considerably better than my equivalent impacts on physics (Average-rep-impact of 2...) and roughly comparable to my impacts on Sci-Fi and Fantasy.
Of course, all of this is for the purposes of fun/learning statistics/improving question and answer quality and needs some serious analysis to see if it's of any real use. One actual use for this could be to compare differences in voting on different stack exchange sites as well as across different tags on one site.