E-Learning Analytics
By Josh Bersin
For decades companies have struggled with the real costs, benefits, and return-on-investment of training. Now that e-learning and blended learning has become a major new initiative, even more investment is needed in the training organization. Using new tools and technology now available, is it finally time for companies to truly apply business analytics to understand the activity, effectiveness, and impact of e-learning and training? The answer is, of course, yes. Enter analytics. Here’s what leading companies and suppliers are doing to make training analytics a reality.
Why measure e-learning? The answer to that question is simple: Because training is a large investment.
Corporations invest large amounts of money, resources, and time in training. According to the 2002 ASTD State of the Industry Report, which surveyed more than 375 major corporations, companies spend between one and three percent of their total payroll on training. On a per-person basis, the average spent on training is more than US$700 per employee per year, and in the leading-edge companies, that figure doubles to more than $1400 per employee per year.
Across a large enterprise, these numbers can become staggering. In 2001, for example, the average profitability of the S&P 500 companies was less than nine percent. If training expense is viewed as a percent of profits, the training budget represent as much as five-20 percent of total corporate profits.
“In today's results-oriented business climate, it's more important than ever for trainers to demonstrate to management the value and return on investment we bring to the organization. This means creating programs and initiatives clearly linked to overarching business objectives -- and having effective tools in place to measure their success,” says Whitney Shelley, director of learning and development for Kinko’s.
What to measure: efficiency, effectiveness, compliance
In training, there are three critical issues which managers face: efficiency, effectiveness, and compliance. Every major decision within the training function falls into one of these three areas. Fortunately, practitioners can establish benchmarks and measures in each (see Figure 1). As I talk with training organizations throughout the world, however, I find that many aren’t benchmarking themselves--primarily because accessing the data they need is too difficult.
| Efficiency |
Effectiveness |
Compliance |
| Training budget as percentage of revenue
Training budget per employee
Training budget per offering
Revenue per training program or per customer
Cost of training per student hour or student day
|
Enrollment rates, completion rates
Scores and certification rates
Improvements in sales, reduction in turnover, improvement in quality tied to specific training programs
|
Completion rates for mandatory programs
Certification acquisition rates
Percent certified in each business unit
Risk of falling out of compliance, days to expire
|
The e-learning dashboard
“If you can’t measure it, you can’t manage it.”
--Nolan & Norton Consultants
As the Nolan & Norton quote drives home, it’s impossible to improve and optimize the training function if you can’t measure its operations. Nolan & Norton advocate the creation of benchmark measures that can be established as guidelines to measure the efficiency and effectiveness of any business operation. Indeed, iIn the mid 1990s consultants Nolan and Norton pioneered the concept of a business scorecard (often called the Balanced Scorecard) that would set specific metrics by which a company could measure itself. Scorecards monitor specific business metrics that are important to the company itself. With an e-learning analytics system, these dashboards are easy to create.
The dashboard is designed to help organizations monitor and distribute the measures within a company. For example, in addition to other metrics, Honeywell measures Six Sigma certifications. According to Mark Sullivan, director of learning technology at Honeywell, “Honeywell is a metrics-driven company. Our business is based on precision results--and for precision results we need precision training. Our CEO has a mandatory initiative for 20,000 engineers to become Six Sigma certified by year-end. Six Sigma defines how we measure quality. We have to measure training results just like we measure product quality.”
Some examples of generic dashboards are shown below:
Measurement challenges
Unfortunately, few organizations consistently measure their training efforts. The ASTD 2002 SOIR found that only one-third of companies profiled try to measure learning effectiveness, and that 12 percent or less try to measure job and business impact.
Why is that? A 2002 Bersin & Associates study of more than 30 training organizations found that the leading reason that companies fail to measure training more rigorously is that they lack the experience, tools, and infrastructure to do so. It isn’t lack of interest or importance that slows or stops measurement initiatives.
If you consider the total training investment per person in an organization, how much should you spend on measurement? One percent? Five percent? Ten percent? According to the most recent ASTD survey of best practices, most companies spend 40 percent to 50 percent of their training dollars on content development, eight percent to 10 percent on infrastructure, and the remaining resources on salaries and facilities cost.
Without question, it’s time to add measurement systems to training the budget. CLOs should consider allocating a small but fixed percentage of the budget to providing the tools and systems to measure the effectiveness of the overall investment. Bersin & Associates research has found that organizations that spend US$2-$10 per learner on a training analytics system notice improvements in the measurability and ROI of their training.
But this sounds like just another expense, doesn’t it? To be sure, in the current economy all organizations must cost-justify every new investment--and measurement is no exception. So, how can companies cost-justify the tools and procedures to measure training? They need to identify the business impact and risk of not training.
For example, Mark Sullivan of Honeywell justifies measurement by linking training to compliance issues: “If we’re out of compliance on critical safety training, we can face large fines. These costs justify our investment in the infrastructure to manage and carefully measure our training results.”
According to Whitney Shelley, Kinko’s links training to customer service issues and product knowledge, “The investment in analytics is far outweighed by the risks of inadequate training. Our success hinges directly on our team members' understanding of our products, services, operations and policies. Given the nature of our business, team members must be thoroughly trained in legally sensitive areas such as customer confidentiality, non-disclosure and copyright, and we must be able to track the deployment of that training.”
Making analytics easy
Analysis requires data. If your organization has implemented some kind of LMS which houses enrollments, fees, completion data, scores, and certifications, then you have the basis for an analytics system. It’s important to make sure that your LMS and e-learning content implements standards, which allows easy tracking of completion stats and test scores. If your organization hasn’t implemented an LMS, you still need to track down this same sort of data.
It’s important here to note the difference between analytics and reports. Most LMS systems have built-in reports. Although reporting tools are important, they alone do not provide the information for building a compelling case for training. Instead, a report should be the result of an analysis. Before you can create a report, you have to dive into the information and separate or distinguish the component parts of training initiatives and results to discover inner relationships. From there you can create a report to view that information regularly.
This concept of analytics is to provide a software solution that lets any business person--not just a PhD statistician or SQL programmer--understand what’s going on in his or her training and e-learning operations. To do so, the solution should answer basic business questions, such as:
-
How much did something cost?
-
What were the components of the cost?
-
Who took or completed a learning offering?
-
What can we do to improve it?
In addition, the training analytics solution gives different users the information they need to make decisions. To that end, Bersin & Associates has identified three categories of analytics users or information consumers. Depending on their jobs, they use information for different purposes.
| Audience |
Analytics needs |
Decisions they make |
| Executives |
Overall metrics, financial information, compliance, and training efficiency |
Drive compliance, ensure training is efficient, make sure that employees are developed |
| Line managers |
Training compliance, skills development to meet their job needs, completion of mandatory programs. |
Drive compliance, develop people, ensure training costs are reasonable, ensure that people are completing and learning something. |
| Training executives and managers |
Training volumes, completion rates, vendor effectiveness, facility efficiency, program effectiveness, overall costs and financial efficiency. |
What courses should be offered? What media and programs work? Which audiences consume the most? Which audiences learn best? Which vendors’ content performs well? How do we optimize scarce resources? What should we stop doing? What should we do more of? |
It’s important to keep in mind that these three different groups need slightly different views of information. Executives want dashboards or charts. Line managers typically need tabular reports and charts designed around their audience and programs. Training managers and executives need the ability to slice, dice, drill down, and filter information on a continuous basis.
Training analytics architecture
The diagram above, adapted from a Saba report entitled “LMS and Other Systems,” shows the basic architecture of a training analytics system.
In most companies, the first step is to analyze existing information, which is the easiest and lowest cost solution. If your organization has an LMS and it has been implemented for six months or longer, you most likely have the right information to get started.
As your analysis needs grow, though, you want to analyze additional information regarding costs, revenue, or organizational hierarchy, which is typically located in other systems. The analytics system must be open so that information can be imported from other sources. Most important, business measures, such as sales data, may be integrated to illustrate direct correlations between learning to business measures.
Packaged analytics solutions
Recently, Saba and Docent announced turnkey analytics solutions. According to their PR, these systems offer out-of-the-box analysis that will let organizations quickly understand the activity, efficiency, and effectiveness of your training. Saba’s system, for example, gives users immediate views of training for any or all programs (see below). Users can drill down by manager or organization and identify completion, student hours, fees, and other important metrics by manager all the way down to the individual learner.
Analytics example
Shown below is a good example of a training analytics system. It gives this company a drill down view of compliance with critical certifications, which are viewable by manager, program, and geography. This program ensures compliance with critical programs in sales, leadership, and technology, and it gives all managers the ability to view their reports and peers.
Benefits to the enterprise: closing the loop
A recent survey, conducted by Bersin & Associates in partnership with IDC, asked 30 companies, “How many of you would like to spend more time and money measuring your training investments?” The answers were enlightening: Seventy percent of respondents wanted more or much more in the way of measurement efforts.
Clearly there’s a demand for more analysis of training today. Companies want to “close the loop” between training investments and business results. The time has come for Training Analytics.