Challenge: The U.S. Navy employs e-learning tools to teach leadership skills to thousands of Petty Officers and others stationed aboard ships and offices around the world. It needs to know precisely how effective that training is.

Solution: The Navy selected the Situational Leadership® II e-learning course from Ninth House Inc. as a primary leadership training vehicle for duty stations, and then carefully measured its effectiveness in a comprehensive pilot analysis, reportedly the first ever study of the behavioral impact of online training.

Among the many benefits of computerized learning technologies is the ability of large organizations to train widely dispersed workforces, and to save money doing it. But it isn’t always easy to document the effectiveness of e-learning courses to ensure that critical soft skills are being improved. Measuring behavioral change can be a daunting assignment.

In the case of the U.S. Navy, currently undergoing a "revolution in training," that information is vital. The Navy has thousands of sailors stationed around the world, many of them frontline personnel in key leadership posts. Under its new decentralized decision-making process, these individuals often must make vital split-second decisions in combat situations. The U.S. Navy must quickly adapt training to meet their needs.

Three years ago, the Navy selected Ninth House Inc., a seven-year-old blended learning provider, as a primary supplier of online learning courses. They include its Situational Leadership® II interactive course featuring The One-Minute Manager author Ken Blanchard. The course teaches coaching skills to help managers develop their employees’ competence and lead them to become peak performers.

The six-hour self-paced course includes practical simulations, web-based reinforcement modules and tailored progress measurement. It typically includes both classroom and CD-ROM instruction, but the Navy agreed that the classroom portion was impractical for deployed sailors, and so decided to rely entirely on the web-based portion.

Although the Navy’s Chief of Naval Education and Training (CNET) is a major proponent of e-learning, and employs an armada of content vendors as well as the THINQ TrainingServer LMS, it selected the Ninth House leadership course for an exhaustive year-long study to determine its effectiveness. Because it was the Navy’s first comprehensive analysis of advanced training methods, interested observers extended well beyond the immediate participants.

"The Navy needed the documentation for future large-scale training implementations," says Jeff Snipes, Ninth House’s founder and CEO. He says it needed to know if the Ninth House course was improving the leadership skills of Chief Petty Officers, and by how much. To conduct the study, the Navy tapped the Center for Naval Analysis, a division of the CNA Corp., an independent non-profit research and analysis institution based in Alexandria, Virginia.

The CNAC pilot monitored 3,968 individuals selected to become CPOs during 2004, with special emphasis on those specializing in aviation and IT. (CPOs are the Navy’s equivalent of middle management. They are responsible for managing 80 percent of personnel aboard ships.) It employed a series of tools, including 360-degree assessments of participating CPOs and others who observed their behavior prior to and three months after the training, to determine whether the acquired leadership skills were being performed. A retrospective satisfaction survey (Kirkpatrick Level I) was conducted six months later.

The study compared pre- and posttest scores from a test that measures learning, and focus groups of the CPO selectees who had taken the training to assess the outcome of their learning experiences. In addition, the study compared the effectiveness of traditional classroom training to Ninth House’s blended online curriculum.

The CNAC’s analysis examined performance improvement on eight leadership categories that are considered essential in the Navy, but also are valued in the private sector. Among them were communicating with supervisors, peers, or subordinates; coordinating the activities of others; developing objectives and strategies; organizing, planning and prioritizing work; training others; and coaching subordinates

Individuals who completed the Ninth House program increased their knowledge and retention of SL II material from 63 percent to 91 percent—a 44 percent improvement—and demonstrated a significant behavioral change in key leadership areas, according to CNAC. Significant improvement was indicated in all vital leadership categories specific to their fields, including team communication, critical organizing, planning, training, and motivating others. In short, sailors learned the course material, successfully transferred what they learned to their job roles, and improved in key leadership areas.

Evidence of cost efficiencies was also significant. Ninth House reports that the program costs 94 percent less than traditional classroom training and could be completed in one-tenth of the time. It projected that the Navy was able to save three days of classroom training plus two travel days for 3,892 active duty and 800 Reserve CPO selectees, resulting in an anticipated cost savings of more than $2.7 million per year.

CNAC research analyst Robert C. Hausmann says the study demonstrates that sailors "learn the material, transfer learning to the job, and that they improve in key leadership areas." Equally important, says Hausmann, it demonstrates that the Navy can effectively deliver training and assessment programs to a diverse workforce across the world. He also said the study identified indicators of performance improvement in critical aspects of the aviation and information technology jobs.

CNAC recommended that the Navy continue its use of SLII for leadership training, and that the course be offered earlier in a sailor’s career. It said sailors at the Private First Class rank and above demonstrated the greatest learning improvements from the course and learned the material as well as, or better than, those at senior pay grades.

Snipes says he believes the study is the first to use multirater assessment to evaluate behavioral change based on an online course alone. "Traditional studies measure cost effectiveness and the reduction of travel time, but fail to measure whether behavior is actually changed. The question is, have they become better leaders by taking this course?" He says that as a result of the study, both the Ninth House approach and the testing methodology are now proven.

The Ninth House chieftain says the study is equally significant for the private sector, which must also reach distributed populations via e-learning amid constantly changing environments. "We see conflicting orders and priorities where training can easily be pushed to the bottom of the list," says Snipes. "As the military has learned, that is not a viable approach. Organizations simply must work on leadership of their managers. They can’t afford not to have people making the right decisions."