OpEd: ROI vs. Metrics

By Jay Cross

Learning about ROI seems to be enjoying a renaissance in the training industry. Workshops and certificate programs abound. However, the courses I've looked at teach things that no business manager would buy. Here, let me tell you why I feel this way.

ROI is often a mask for uncertainty or an attempt to quantify cost-benefit with accounting principles that don't count people as assets. I contend that the business return on an e-learning investment should be so obvious that you can figure it out on the back of a napkin.

Traditionally, executives assume training has little or no impact on revenue, so they measure training benefits in terms of cost savings. This works against e-learning, in which increases in top-line revenue generally exceed reduced expenses by a wide margin. Enter metrics.

Metrics are broader than ROI

Metrics are measurements that matter. Decision-makers use metrics to

• choose the best course of action
• supplement gut feel with a framework of logic
• assess project failure or success
• monitor progress
• uncover ways to make improvements
• divine ways to do better next time
• focus attention on profitable activities.

The Industrial Age is over, so measures that fail to account for intangibles are misleading. And often, metrics are in the eye of the beholder. They aren’t simply the application of a rote formula or accounting rule, instead they’re subject to interpretation. This is what makes metrics worthy of discussion.

What does “proof” really prove?

How I wish I had $10 for every time I've heard a training manager lament that they cannot separate the impact of training from everything else that was happening in the organization. I’ve even heard some trainers suggest using a control group, having a number of employees go untrained. Forget that idea; the Hawthorne effect* would skew the results. (*In a classic experiment in the 30s at Western Electric's Hawthorne Works, researchers found that workers were more productive when they cut the lights up. Also, when they cut the lights down! Conclusion: Workers are more productive if you pay attention to them. Placebos work.)

And anyway, a control group would lose its usefulness in the corporate world because business is not a precise science. Deciding whether to invest in more training or increasing bonuses is not some physics experiment requiring six-place accuracy. Consider John Wanamaker's regret: "I know only half my advertising is effective. If I only knew which half."  Wanamaker didn't become a department store mogul by cutting his ads; he did what his gut told him to do, as should you.

Think about this: Your sponsor decides whether an iffy investment, like Wanamaker's ads or your training program, is worthwhile. The sponsor is your client, and it could be anybody: the CEO, the VP of sales, the head of HR, and so forth. The point: The sponsor has the strongest influence over decisions on how to spend money, so the sponsor decides what markers constitute proof.

Performance agreements

The best proof describes quantitatively the link between your learning initiative and business results, using assumptions your sponsor will buy into and explicitly stating the what-ifs and maybes. Do this in writing, in a performance agreement that

• provides a shared understanding of the problem to be solved
• describes what you intend to provide in its solution
• estimates the expected increase in profit and the step to get there
• sets out a way to assess whether the goal was accomplished or not
• lays the foundation for solving the next problem.

This sort of agreement will show that you understand the business and that you're on the same page as your sponsor.

Beware of bad numbers

Present-day accounting is an anachronism. Invented half a millennium ago to maintain accurate shipping records, double-entry bookkeeping helped Venice dominate its part of the world. Formal accounting worked well when you could go out to the warehouse to count your assets. But in the Information Age, it's an inappropriate yardstick to measure most modern business assets. Most assets drive home every night.

In a nutshell, the basic problem is that accounting recognizes nothing but physical entities. Intangibles are valued at zero. Vast areas of human productivity--ideas, abilities, experience, insight, esprit de corps, and motivation--lie outside the auditor's field of vision.

In that way, ROI or cost-benefit analysis is relative, rather than some absolute value like the speed of light. CEOs rarely care about learning objects or an LMS. Line managers focus on the performance of their unit rather than the overall corporation. Training directors don't allocate resources to business transformation. One size does not fit all. In other words, where you stand depends upon where you sit.

Time matters

Before you get too far into metrics, ask yourself, "Does it matter?"

One of the few aspects of accounting that I like is The Principle of Materiality. This principle says that if it doesn't matter, don't worry about it. For example, if Chevron-Texaco’s accountants uncover a $32,000 error in the sales department’s expense budget, they don't make Chevron-Texaco note the error in its annual report. Chevron rakes in $100 billion a year, so $32,000 is a drop in the bucket. It's immaterial. But if the accountants find a $32,000 discrepancy in your personal expense report, that's material. If that’s you, send us a postcard from the slammer.

Keep in mind that it’s nearly impossible to measure everything. Therefore, seek to measure the important things and let everything else coast. Don't fritter away time on the small stuff--because time matters most of all.

Even though training directors have different objectives than CEOs, everyone in the business world shares one need: They want it NOW. Benefits that go undetected for two years are hardly benefits at all. Therefore, an appropriate metric for most e-learning is time-to-proficiency. How long will it take until your people are performing competently? By competent, I mean able to meet or exceed the expectations of customers, be they internal or external to the organization.

And the largest cost of all maybe foregone opportunity.

Indeed, again and again, I've found that the largest overall cost of any corporate learning endeavor is the cost of people's time. I'm not talking about salaries and benefits; I refer to the value they would have created had they not been tied up in training. Opportunity cost per hour is not a fixed amount. A salesperson's time during working hours in peak buying season is worth much more than the same individual's time after closing time in non-peak season. E-learning often enables the employee to shift learning to those non-peak hours.

There's more

To be sure, I could go on for another ninety pages, but I’ll leave you to think over what I’ve said so far…

Published: April 2004