Reviews ASTD Members Only

Docent Analytics

An early look at a new addition to the learning management toolkit.

Docent’s new Analytics is designed to improve an organization’s ability to measure the impact of learning. It rounds out the Docent product suite, which includes learning, content, and performance management. For organizations focused on measuring the results of learning initiatives, the release of Analytics signals the beginning of a concentrated effort in the market to resolve learning management’s dirty little secret: The built-in capability for reporting and analysis is woefully weak.

Many organizations that implement learning management infrastructure share a common vision: The ability to create that snappy chart that finally informs the organization’s executive team exactly how learning investments contribute value to a strategic effort in sales, manufacturing, or R&D. The vision is quickly deflated, unfortunately, when it becomes clear that the path to that chart involves additional spending to develop custom reports; employ report-writing tools that no one really understands; or export the LMS data into Microsoft Excel so someone in the organization can spend hours slicing and dicing it—and then do it all over again next month after the data changes.

A learning management infrastructure can capture an incredible amount of data about learning initiatives. So why is it still so hard to manipulate that data into meaningful analysis to show business impact? An early look at Docent Analytics gives hope that the tide is changing.

This article explains the Docent product and provides an overall view of its utility and quality. See Jeff Merrell’s TMR report, “Docent Analytics,” for a detailed evaluation of the product’s features and an explanation of the evaluation framework.

How analytics fits into an LMS

A learning technology project should create business value in one of two ways. First, it should support the creation of strategic advantage. (Strategic learning creates value by driving competitive advantage through the capabilities of employees, suppliers, or customers.) Second, the learning technology should further operational excellence in learning and development functions (design, development, delivery, and management of learning activities). Those efforts help manage costs by improving efficiency in both strategic learning and “core” learning, for instance learning that may be required by the business but does not necessarily create strategic advantage.

A learning management solution directly or indirectly influences all of the critical elements required to build an infrastructure to support strategic learning and operational excellence:

• user environment and user management
• learning and development process and resource management
• content authoring, content management, and learning delivery
• reporting and analysis.

Docent Analytics should be viewed as a new extension of LMS reporting and analysis. In general, reporting and analysis tools should help organizations turn data into information and actionable insight. Without such tools, the LMS is a big data bucket filled with transactional information (user starts, user gets a score, user completes). Reporting and analysis tools should support the transition from a collection of transactional data bits to a system of integrated metrics that show the influence of learning on strategic value and operational excellence.

The journey can be mapped against a matrix that helps organizations answer several key questions about any learning initiative:

• Does the learning project increase operational excellence or add strategic value?
• Can the intended impact be measured?
• Is the measure a single, independent metric (namely, one-time cost reduction)?
• Is it one that contributes to a value chain? (Do we understand the contribution of each link in the chain?)

Docent Analytics is positioned to help organizations develop independent and integrated metrics that differentiate between operational and strategic impact for learning and development projects.

Current state of LMS reporting and analysis

Even for an organization that has sorted out how to measure the value of learning to the business, the long road to turning data into information and insight usually begins with the discovery that LMS reporting and analysis capabilities are limited. Even the best systems fall short here, in part because they are designed that way.

Standard LMS reports typically cover the most common areas of interest for training administrators (specifically, enrollment or registration management, resource utilization, and financial transactions). Reporting capabilities may also support the interests of business managers and individual learners. Managers may have access to reports on the progress of their teams in completing learning activities; individual learners may view summary reports on their personal progress.

In all of those cases, the system defines the data included in each standard report and the structure (rows and columns) of the report. Users have a limited ability to modify the structure.

To move beyond that limitation, most LMS providers offer one or more of the following alternatives:

• Create additional reports during the LMS implementation phase that use the standard LMS interface but with custom data and structure specified by the client.
• Create an interface between the LMS database and one of several popular, desktop report-writing tools. Users who are familiar with the report-writing tool can then manipulate LMS data into new reports.
• Export LMS data into another tool for manipulation and analysis. This capability can range from exporting to familiar office productivity tools or into more sophisticated enterprise databases constructed specifically for data mining.

The situation is lose-lose for everyone involved. Organizations focused on measuring the true value contribution of learning discover they have two options if they wish to achieve any level of analytical sophistication: Provide funding for a database guru to create new report views every time someone thinks of a new way to look at LMS data, or hope that the IT organization won’t drain the entire learning and development budget when asked if LMS data can be pumped into corporate analytics tools.

LMS providers also face a conundrum. They can build more standard reports and still never satisfy every client requirement, or they can offer to build custom reports during implementation and risk client backlash at the typical fees for such efforts: It will cost how much?!

Docent Analytics value proposition

Docent now enters into this situation, declaring that the same class of tools businesses used to understand the impact of customer relationship management, finance, or supply chain management should also be applied to analyzing the business impact of organization learning efforts. To understand the context of this approach, it is important be familiar with two database technology terms:

• Online Transaction Processing (OLTP) encompasses database technology tuned to processing transactions. An LMS database is really a transaction engine processing registrations, test scores, activity results, and so on.
  
• Online Analytic Processing (OLAP) encompasses database technology designed for dealing with analytic processes as opposed to transactions. It’s more multidimensional than relational (tables, columns, rows) and incorporates specialized technology for calculations and queries.

These two database approaches establish the foundation for the continuum of solutions organizations can use to grapple with the issues of reporting and analysis in learning management. (See Exhibit 2.)

[Exhibit 2 A Simple Continuum of Analytical Solutions

Docent is carefully positioning its new offering as not only providing the underlying technical functionality to perform data analysis more easily (OLAP), but also the predefined parameters for performing learning activity analysis in different environments. Training analysis in a sales environment, for example, will not only need to include typical training elements (for example, learning activities, scores, completions) but also sales data (specifically, region, district, branch, representative, product category, product, and SKU).

In short, this isn’t just an add-on to create more types of reports. Docent is positioning Analytics as a whole new class of business analysis tools.

Solution packaging and pricing

Analytics is an optional, separately priced product within the Docent Enterprise suite of solutions. It is a database application that is distinct from the Docent LMS and requires a dedicated server. Two types of users are defined in the system and priced differently: Basic User and Power User.

Pricing, as of May 2003, is as follows:

• reporting and analysis database: US$50,000
• annual license fee per Power User: $350
• annual license fee per Basic User: $20

You should take these prices as rough benchmarks. As the market develops, Docent and other entrants will be refining their pricing strategies. Professional services fees will vary based on the scope of the implementation.

Recommendation

Docent is to be commended for establishing a leadership position by declaring that corporate learning and development warrants its own set of analytics applications. As more organizations implement balanced scorecards or use Key Performance Indicators (KPI’s) to drive organizational performance across functions, the ability to analyze information, establish metrics, and create actionable insights is a critical success factor for business performance. And this capability is even better if it can be distributed to local levels within the organization rather than held in a central corporate think tank.

Docent Analytics has a few soft spots, but the momentum is beginning, and it seems reasonable to expect rapid development of templates and tools in the four business areas Docent has earmarked for attention.

Any existing Docent LMS customer and any organization currently searching for a learning management solution should seriously consider Analytics as an element of a learning management infrastructure.

	Docent Analytics
User environment
Delivery	NA
Content authoring and management	NA
Learning management	NA
Reporting and analysis
Enterprise integration
Implementation management
Overall rating