The first Evaluation Panel member is a lot more lenient than the second, who gives much lower scores. If your idea was scored by the first Evaluation Panel member, it would have a much higher total score than if it was assigned to the second.
To fix this issue, we utilize a mathematical technique relying on two measures of distribution, the mean and the standard deviation.
The mean takes all the scores assigned by an Evaluation Panel member, adds them and divides them by the number of scores assigned, giving us an average score. So, if an Evaluation Panel member is easy, he will have a much higher average score than a harsh one.
Formally, we denote the mean like this:
The standard deviation measures the “spread” of an Evaluation Panel member’s scores. So, maybe two of them both give the same mean (average) score, but one assigns a lot of zeros and fives, while the other gives a lot of ones and fours. We can see how it wouldn’t be fair to you if we didn’t consider this difference.
Formally, we denote standard deviation like this:
To ensure that the scoring process is fair, we rescale all the scores to match the population of Evaluation Panel members. In order to do this, we measure the mean and the standard deviation of all scores across all of the Evaluation Panel members. Then, we change the mean score and the standard deviation of each Evaluation Panel member to match.
We rescale standard deviation like this:
Then, we rescale mean like this:
Basically, we are finding the difference between both the distributions for an Evaluation Panel member and those for all of the Evaluation Panel members combined, then adjusting each score so that no one is treated unfairly according to which Evaluation Panel members they are assigned.
If we apply this rescaling process to the two Evaluation Panel members in the example above, we can see the outcome of the final resolved scores; they appear more similar, because they are now aligned with typical distributions across the total population of Evaluation Panel members.