Take two courses and call me in the morning.

In recent years, interest in examining technology's potential to meet training needs for public health professionals has generated new initiatives, such as the Enhancing Data Utilization Skills through Information Technology program.

EDUSIT is a distance education program that trains, via a Web-course, maternal and child health professionals in state and local health departments to collect, analyze, and interpret data. The course is offered through the Department of Maternal and Child Health and the School of Public Health at the University of North Carolina at Chapel Hill in collaboration with the school's Center for Distance Learning.

The overall goal is to train maternal and child health professionals to apply data collection and analysis and technology skills to improve access to health care services for mothers and children. At the outset of the project, three primary outcomes reflecting the overall goal of the project were identified:

• improve participants' information technology literacy
• improve participants' data collection and analysis skills
• motivate participants to complete the full year-long course.

Those goals were pursued by developing and implementing a year-long six-module course presented in a self-paced Web-based format.

The people

Learners. In general, learners were well-educated and their last degree was awarded more than ten years prior to the project. As a cohort, the pilot group had positive attitudes about using technology and data, but confidence in their own abilities to use data and demonstrate technology skills was low. Participants were expected to complete the entire course; six hours of graduate credit was offered as an incentive.

Developers. Because the Department of Maternal and Child Health had no experience in the area of distance education prior to obtaining funding, the Center for Distance Learning (cdlhc.sph.unc.edu) was a critical collaborator. CDL provided support staff, including the center director and assistant director, an instructional designer, a Webmaster, and other technical staff such as audio assistants and software specialists. Academic Technology and Networks, UNC's technology support office, provided consultation and training for the software used to administer the course. One instructional designer from CDL was assigned to the project long-term.

Instuctional staff. Instructors and facilitators were selected for their expertise in teaching specific data skills in traditional classrooms or through consultation venues. A different instructor developed each module and none had experience with Web-based training. To facilitate understanding of the Web-based format and to develop instructors' expertise with various approaches, training was provided via a Website and meetings. For example, posted on the Website were guidelines on how to successfully monitor a discussion forum because each module had at least one forum. Meetings were held throughout the development phase of the project. Instructors were geographically dispersed, so conference calls were used simultaneously with online material.

A survey of instructors revealed that they couldn't rely on material developed for the classroom but needed to develop and organize new material for their modules. Completing material within the scheduled timeframe was critical because it needed to be turned over to the instructional designer for Web preparation, which included designing the layout and activating all quizzes and interactive segments. Instructors were dependent on the instructional designer for guidance in deciding the most appropriate format to use for various topics, augmenting print material, making material more interactive, creating slide presentations, and developing useful examples. In addition to the instructional designer, instructors needed administrative assistance obtaining copyrights and scanning articles, as well as technical assistance for recording the audio portions of the modules.

The program assistant supported instrructors by providing immediate consultation to learners and teaching technology skills. Questions or issues that couldn't be tackled at this level were referred to CDL staff, who then worked with learners or program staff to resolve the issue. Project staff assisted in forging a relationship between a particular agency's LAN manager and the learners; subsequently, on-site technical assistance was available when needed.

The course

CDL staff participated with EDUSIT in the initial design discussions about the appropriate delivery method. Decisions about technology delivery were based on the geographic distribution of the audience and characteristics of the learners, which included participants from 13 state health departments. Videoconferencing, the technology that CDL had the most experience using, was quickly eliminated because the audience was so dispersed and unlikely to meet simultaneously. Video and CD-ROM were not considered because of the need to manipulate data and revise and update materials in response to learner needs. Consequently, the course was entirely Web-based. During the development phase of the course, weekly meetings were held between EDUSIT and CDL to ensure that both content and technology were on track.

A course Website was created to provide a classroom-like atmosphere and a site where all course information could be obtained. The Website included an overview of the course; a course user's manual; photos and biographical sketches of instructors, project staff, and participants; a bulletin board for activities and updates; and a list of resources. The password-protected course and its newsletter (inaugurated with a prize for the learner who came up with the best title) were accessible from the Website.

The EDUSIT course included the following components:

Precourse skills. Potential learners were provided a description of the technical requirements, including software needed to participate in the course. Baseline survey data revealed a wide variation in participants' technology skills. Consequently, follow-up phone interviews were conducted and 10 Internet skill-building exercises were developed that related directly to the skills and modes of electronic communication needed for the course.

Because the course had been well-defined, we were able to identify specific technology skills that the learners needed and provide training and practice exercises to develop them. Exercise topics included using email and its various features, completing an efficient Internet search, downloading Adobe Acrobat Reader and RealPlayer, participating in discussion forums, and subscribing to mailing lists. After completing related activities, learners were required to pass a short skills test to obtain password access to the course.

By providing technologically appropriate training prior to the course, we prepared learners for success. One learner sums up the importance of good preparation: "I can truly say that without those activities, the convenience of email and the use of the Internet would still be giant mysteries and monsters for me! Now I feel comfortable with both."

The six-module training. The content for the online course emerged from experiences in traditional training initiatives conducted by the Department of Maternal and Child Health and from recommendations from program directors in the states participating in the EDUSIT program. Content focused on quantitative and qualitative data collection, analysis, and interpretation.

Various presentation modes were used throughout the modules, including online tutorials and slides with audio and video. Interaction was encouraged through online discussion forums, chats, and space for participants to post written assignments and for faculty to give feedback. Online self-tests and quizzes with instant feedback were available.

Each module took approximately fifteen to twenty hours to complete. The modules were field-tested by graduate students and then modified according to field testers' evaluations before being released for training. Significant changes were made in many of the modules based on field test results.

State-specific health status problem. In each state, participants worked in teams to apply the skills they learned to a health status problem that was significant in their state. Learners developed reports on their state-specific health status problems and posted them on the Internet. Reports covered a variety of topics with Webpage designs that ranged from simple to complex. For example, one group presented a two-page report summarizing state and local data on childhood asthma with links to relevant resources.

Evaluation. The evaluation model, including formative, summative, and impact measures, was implemented at various points in the course. Initially, participants completed a baseline survey assessing their data and technology knowledge, beliefs, self-efficacy, and current practices, as well as agency factors and demographic characteristics. Six months after completion of the training, the survey was readministered to determine whether particpants experienced changes in their knowledge, attitudes, and use of computers and information technology.

The formative evaluation was administered after each module to determine what worked. Learners rated the value of each topic within a module using a Likert scale, indicated successes and limitations, and offered recommendations to be considered in the development of subsequent modules.

The summative evaluation included questions on knowledge of the module content, beliefs about topic usefulness, self-efficacy, and current practices.

The final evaluation focused on the impact of the initiative and the benefits of training to the state health departments, skills developed, directors' willingness to institutionalize training into their program, and benefits to community-level programs.

Lessons

We elicited ongoing feedback from participants through module evaluations and by creating an atmosphere that encouraged feedback. We responded to learners' suggestions for revisions that would make the course more appropriate for the intended audience. For example, the original syllabus wasn't useful to learners, who suggested that a list of activities and tasks for each module would be more appropriate. Consequently, a checklist was added to the syllabus. The flexibility of course instructors and designers appeared to be an important factor in learners' continued investment in the course. Indeed, from the project's earliest stages, changes were made based on lessons learned.

Some insights gained during the development phase of the modules include

• Allow plenty of time for development. Consider doubling initial estimates to make sure that materials are completed on time.
• Involve an instructional designer from the beginning to the end of the development phase to reduce the number of unexpected problems.
• Give instructors with no experience teaching online the time and the skills training to learn how to be effective on the Internet.

During the course's implementation, the designers realized that a year-long course wasn't an appropriate model for training the target population. Professional staff move in and out of the system at both state and local levels. Thus, efforts to build a training community within a cohort of participants were consistently undermined. Also, as in any business, the healthcare workload ebbs and flows. Professionals have more leisure at one time than at another to complete course-related assignments. Even though directors had given their support--and in some cases attempted to protect time for staff to compete course-related activities--the immediacy of day-to-day tasks took precedence over coursework. Supervisory support for training in the workplace was tempered by realities of the job. Finally, not all data-skills modules were of equal interest or provided new material to all participants. We learned that the characteristics of the work environment need careful consideration. Modules were too long and lacked natural stopping points, so they were disruptive to the flow of normal work responsibilities. Shorter, just-in-time modules taking two to four hours to complete would have been more useful.

Because distance learners don't necessarily have the experience of group interaction, we soon learned the value of including activities that bring learners in contact with one another. For example, requiring participants to work in teams on a health issue of relevance in their state gave them an opportunity to discuss course content with colleagues and to apply what they had learned to an actual problem.

We also learned that responsibilities shift when using the Internet for training. For the learner, the freedom of scheduling a convenient course time comes at the price of having to secure and maintain an operating computer system. Some learners who had never before been hooked up to the Internet realized the low speed at which they were able to access information. In that case, learners had to negotiate with their supervisors to get a faster machine. Other learners discovered they were unable to engage in discussion forums or chats due to a firewall. While some learners were able to engage in discussion forums through a third-party, others could only access the archived chat transcript.

The learners might feel that the responsibility for infrastructure is piled on their shoulders, but the provider institution isn't immune from costs and responsibility. For the provider, the control over designing a common space (classroom) is replaced by the need for a support system for learners who work increasingly at nontraditional times. For example, 24/7 help desks have become common elements of distance education programs. It became clear that planning for staffing for a distance education project must include not only the consultation of experts for course content but also the expertise of an instructional designer and technical experts.

Successful features of the course included development and implementation of the precourse training and the content of the six module course. We know from the participants that both the precourse training and several of the modules provide opportunities for the application of new skills on the job. More important, we acquired valuable information about the needs and norms of the public health workforce. However, the implementation of the course revealed limitations that provided important lessons for future e-learning projects.