shows that virtually everyone is extremely
overconfident before the training (e.g.,
over a large number of trials, when they
say they are 90 percent certain, they may
have less than a 60 percent chance of
being correct). But HDR also found that
about 80 percent of individuals can be
trained to be nearly perfectly calibrated
(they are right just as often as they expect to be). In other words, they can be
trained in about half a day to be as good
as a bookie at putting odds on uncertain
events. This skill becomes critical in the
process of quantifying someone’s current
uncertainty about a decision.
4. Calculating information values
avoids “the measurement inversion.”
A defined decision should always be
the objective of measurement. Uncertain variables in such a decision have a
computable expected value of information
(EVI); that is, what is it worth if we had
less uncertainty about this? When HDR
compared the EVI to clients’ past measurement habits, virtually always what got
measured and what needed to be measured were very different things. With the
third edition, HDR has conducted more
than 80 major decisions analysis, and the
results are consistent with earlier findings:
This phenomenon appears to pervade
every industry and profession from software development to pharmaceuticals,
A NA L Y T I C S
real estate to military logistics, and environmental policy to technology startups. It
appears that the intuition managers follow
to determine what to measure routinely
leads them astray; they tend not to measure the very things for which they have
the poorest information and would therefore benefit most from more data. Hubbard calls this practice “the measurement
inversion,” and it appears that the best
guarantee to avoid this problem is simply
to know the information values of uncertainties relevant to a decision.
5. A philosophical dilemma: Does
probability describe the object of
observation or the observer?
When someone says, “but how do I
know what the exact probability is?” they
are implicitly adopting a particular definition
of the word “probability.” Since the author
observed the challenges some readers
were having with this issue, the newest
edition of “How to Measure Anything” expands more on it. We generally take a
Bayesian position on the interpretation of
probability – that is, probability is used to
quantify the uncertainty of an observer, not
a state of the thing being observed. This
stands in contrast to the “frequentist” point
of view, which treats a probability as a kind
of idealized frequency of occurrence in
some objective system. Somewhat ironically, the validity of applying subjective
M A Y / J U N E 2 014
|
31