[Reprint (PDF) Official Layout of this Article]

Deborah Glik, ScD¹; Emil Berkanovic, Ph.D.¹; Fiona Macpherson, MSc¹; Kathleen Strone, MPH¹; Darcy Ratner²; Marcy Connell Jones³;  
¹ UCLA School of Public Health; ² Ratner & Associates; ³ California Department of Health Services, Immunization Branch

Corresponding author: Deborah Glik, ScD; UCLA School of Public Health; phone: 310.206.9548; fax: 310.794.6480; e-mail: DGLIK@UCLA.EDU.

This paper describes the dissemination and utilization of a 6th grade immunization curriculum, Immunization Plus!, distributed to schools and school districts throughout California in 1997-98. Using a tracking system and a follow-up survey, this dissemination analysis assessed how the curricula were distributed, what different contact methods cost, how the curriculum was used, and barriers to use. Of copies tracked, only 33.5% could be traced to actual users in the classroom, teachers or nurses. Only 23.3% of teachers and nurses had used the curriculum. The most effective methods of dissemination included outreach with personal contact.

Dissemination research describes the diffusion and adoption of specific programs and interventions by key individuals, agencies, and institutions, whereas implementation research is the study of how programs are used in specific sites (Basch, 1984). Large scale health promotion and education campaigns lend themselves well to dissemination and implementation research. This study describes the distribution and subsequent utilization of a 6th grade immunization curriculum, Immunization Plus!, in classrooms throughout California. Data presented assess the effectiveness of different methods of curriculum distribution, whether teachers and school nurses who received the materials used them, and the barriers to use of this curriculum.

Immunization Plus! was developed as a comprehensive curriculum on communicable diseases with an emphasis on their control through immunizations. Curriculum development was based on formative research with target audiences including students, teachers, curriculum development specialists, and administrators. Five thematic modules include topic areas of communicable diseases, immunizations, the immune system, herd immunity, and personal and community responsibility for health (Glik, Stone, McNeil, Berkanovic, Jones, Richardes, Mirocha, 1997). The curriculum stresses participatory and skill based learning, which is integrated into other subject matter areas. Activities are designed to meet grade level requirements for Mathematics, Science/Health and Language Arts. When taught for 10-12 hours, it has been shown to be an effective tool to change knowledge, attitudes and self reported behaviors of students (Glik, Berkanovic, Macpherson, McNeil, Stone, Gill, Jones, in press).

This curriculum was developed in response to recently revised national guidelines concerning infectious disease control among young adolescents(Centers for Disease Control, 1996). It is now recommended that young adolescents receive the three dose Hepatitis B vaccine and a tetanus diphtheria booster (Td). As well they should receive the varicella immunization, if they are among the 20% of population that has not had childhood chicken pox, and an additional dose of the measles, mumps and rubella vaccine (MMRII), if they have not received it at an earlier age (CDC, Advisory Committee on Immunization Practices, 1996). Current estimates are that only 10-20% of young adolescents are fully immunized in accordance with current recommendations. Barriers to immunization coverage in this age group include infrequent use of medical care, lack of medical insurance, and lack of knowledge about the new recommendations

In response to changes in recommended immunizations, in 1996 the Department of Health Services of the state of California launched a strategic campaign to improve immunization coverage among adolescents. School based educational initiatives have been a viable means to increase awareness about the importance of these preventive measures (Centers for Disease Control, 1996; Boyer-Chuanroong, Woodruff, Unti, Sumida,1997; Averhoff, Brink, Pollard, et al. 1997). The development and dissemination of the Immunization Plus! curriculum for the middle grades was one element in the California campaign. Other strategies were improved provider education, availability of free Hepatitis B vaccine for 11-12 year olds at county health clinics, and a new state law (AB 381) mandating that all students entering the 7th grade have started the three part Hepatitis B series by July 1, 1999. One goal of this 6th grade curriculum was to help improve awareness of these new requirements.

Dissemination Research

Dissemination evaluation research is mainly linked to large scale, regional or statewide attempts to distribute health programs, curricula, or communications materials to groups and organizations (Basch, 1984; Parcel, Eriksen, Lovato, Gottlieb, Brink, & Green1989; Paulussen, Kok, Schaalma, Parcel, 1995). While dissemination evaluation shares some methodological commonalities with program monitoring, monitoring is usually more narrowly focussed on programs with a smaller number of locations. That is, program monitoring assesses target population coverage and integrity of the program within specific sites (Sechrest, 1981; Kamb, Dillon, Fishbein, Willis, 1996; Rossi, Freeman, & Lipsey, 1999). In contrast, dissemination evaluation documents the diffusion of activities over a larger target population, and uses explanatory models such as diffusion of innovations (Rogers, 1983; Dignan, Tillgren, Michielutte, 1994) or mass communications and information processing (McGuire, 1972).

A number of studies have tested the effectiveness of specific strategies to disseminate new programs and curricula to schools. Studies guided by diffusion theory have assessed characteristics or situations of persons more likely to adopt new programs or curricula (Paulussen, Kok, Schaalma, Parcel, 1995), characteristics of the innovation that makes it more acceptable, methods of dissemination that are more successful(Brink, Levenson-Gingiss, Gottlieb, 1991; O'Hara, Brink, Harvey, Harrist, Geen, Parcel, 1991), and barriers to dissemination and utilization (Parcel, et al, 1989; Brink, et al, 1991). In this study we focus on the latter two aspects: methods of dissemination and barriers to dissemination and utilization.

Two methods used to assess dissemination are surveys of end users, and monitoring systems of the distribution process itself. For example, a study that documented the adoption of HIV/AIDS curricula among Dutch secondary school teachers examined how teachers' subjective norms, perceived instrumentality of the curriculum, collegial interactions, and networks increased teachers' adoption of different curricula (Paulussen, et al, 1995). Other studies have used similar methods (Parcel, et al, 1989; Brink, et al, 1991). The other method is to use a tracking system to assess how and through what channels materials are distributed (Dignan, et al, 1991).

Studies rarely document both distribution processes and dissemination outcomes. For example, in a study that surveyed teachers to assess how they used curriculum, the majority of teachers had not received the materials being studied. As there was no simultaneous tracking system for distribution it was difficult to specify the barriers to getting materials into teachers' hands (Brink, et al, 1991).

The present study used both distribution tracking and follow-up survey methods to document the distribution and utilization of Immunization Plus!. Tracking methods were used to assess where materials went, how many teachers and school nurses received materials, and what the best method of distributing materials to end users were. Survey methods were then used to assess, among teachers and nurses who received the material, whether and how the materials were used, how materials were evaluated, and what some of the barriers to receipt and use were. In addition cost data linked to the actual distribution methods were also collected. Diffusion theory (Rogers, 1973) and theories about the effectiveness of communications campaigns (McGuire, 1972) guided this study. Both theories predict that, after initial dissemination efforts, only a small percentage of the target audience, in this case teachers and school nurses, will actually acquire and use the product disseminated.

Distribution Activities, Targets and Methods

The dissemination of the first 3000 copies of the Immunization Plus! curricula took place in 1997- 1998. During this time 2,975 copies were distributed. A second printing in 1998 provided 2500 additional copies, however the dissemination analysis documents only the first wave of distribution. At outset it was estimated that there were approximately 10,700 6th grade teachers in California. Limited resources did not allow a strategy that specifically targeted every 6th grade teacher. Moreover as the study progressed it became clear that school nurses were a second important target group, as they teach health in many districts. As the goal of this analysis was to document how effective different contact methods were in bringing the curriculum into the classroom, lack of materials for every potential target is not considered critical to this analysis.

Seven basic methods were used to promote the distribution of the Immunization Plus! curriculum. First, there was a category called Cold Send, where persons in supervisory or decision making roles in counties or districts were sent an unsolicited copy of the curriculum. For example in each of each of 59 counties one copy was sent to the County Office of Education and one was sent to the County Health Department. A second category of contact were persons known to the statewide Advisory Committee. A third contact method was attendance at local and statewide educational and health conferences: persons at these meetings ordered copies. A fourth contact category was mailers, one with telephone follow-up and one without telephone follow-up, sent to generate orders. A fifth category was mailers sent to curriculum reform projects within academic education departments. A sixth category was a county-wide teacher in-service training session in Humboldt County. The seventh method was advertising in a statewide immunization coalition newsletter, IZ Update, to generate orders. Thus promotion methods fall into two general categories : 1) personal networks and outreach (advisory committee, conferences, teacher in-service) and 2) impersonal publications (Cold Sends, mailers, newsletters) with mailers followed by phone call a mixed method.

Evaluation methods

To monitor the dissemination process, a computerized management information system (MIS) was established to track all contacts and orders for the curriculum. This provided counts and identification information of teachers, nurses, schools, districts and counties who ordered or who were sent the Immunization Plus! curriculum. Except for the initial Cold Sends, the remainder of the copies were sent to those who requested them free of charge.

Information for the MIS was first collected from the program side to obtain names, addresses, professional position, and affiliation of those who got the material through one of the seven major distribution channels. Then databases were then sorted according to those who received more than one copy of the curriculum and those receiving just a single copy. To track distribution of the curriculum, all persons who received more than one copy were asked to fill in a Log Sheet to provide information about what happened to their copies of Immunization Plus!.

Copies of the curriculum were distributed from June 1997 through December 1997. From September 1997 through December 1997, multiple waves of written requests for the return of completed Log Sheets were sent out based on initial distribution lists. As this yielded a low rate of return (35%), in January 1998 data collection of Log Sheets by telephone and fax was initiated. The large bulk of Immunization Plus! curricula were sent in sets to persons in a position to distribute them to others.. Of the initial 2975 copies distributed, approximately 378 were Cold Send (with no specific order) to counties and to individuals. Another 2597 copies were sent to 191 individuals each of whom ordered two or more. Of persons who ordered multiple copies, follow-up log sheets were completed among 153 persons representing 2297 copies of the curriculum. Thirty-eight persons representing 300 copies of the curriculum were lost to follow-up, as were 378 individuals who initially received the curriculum prior to the implementation of the MIS. Therefore actual follow up was achieved for 77% of the curriculum copies.

The purpose of the cost analysis was to put the distribution of the curriculum into monetary terms. Cost data for each method of distribution of the curriculum was based on the cost of the labor to generate the orders, track the requests, and respond to special requests; costs associated with printing mailers, fliers, advertisements, and correspondence; telephone and fax costs associated with distribution; and staff travel costs for presentations, meetings and conferences. Not included in the cost estimates were actual printing costs of the curriculum and postage costs to send them out (this was handled by a mail order company). The analysis of these data compared the total number of curricula that were sent out (general distribution), or those that were in the possession of teachers or nurses that could be tracked through the MIS, with the average cost of each distribution method.

The MIS was also designed to generate lists of individuals who could provide information on implementation. Thus the follow-up survey, conducted in the Spring of 1998, was based on names and address of 610 teachers and nurses on the original contact list and the log sheet information provided by distributors. This sample of names is less than complete as some log sheets were returned stating that the binder was given to teachers or nurses but no name or school site was given. A telephone survey was conducted with this sample to obtain basic information about curriculum use. Respondents were called up to seven times before being excluded from the sample. Questionnaire items asked whether the individual received their copy, used it, how much they used it, and their rating of the material. The interviewer was able to contact and interview 382 persons or 62.6% of the total sample.

Initial findings based on counts of curricula distributed show that over time most copies were distributed in the summer months prior to September, 1997, with 2,774 copies sent out. The remaining 201 copies were sent out during the remainder of 1997. Interest in Immunization Plus! clustered in several counties. While every county in California received at least two copies of the curriculum, the variation in numbers received ranged from 2 to 909. Specifically the counties of Los Angeles, Orange, Riverside, Sacramento, San Joaquin, Santa Clara, and Tulare all received more than 100 copies, with Alameda, Fresno, Santa Barbara, San Diego, and Ventura counties receiving between 50 and 100 copies. Los Angeles County received 909 copies, the largest amount for any single county, possibly attributable to the high population base, the large number of school districts, and county officials involved in immunization programs.

From an institutional perspective the distribution was split between County Health Departments, County Offices of Education, school districts, and individuals working at the school level throughout the state. The greatest number of copies distributed to any one type of institution was 968 going to the numerous school districts. This was followed by the county offices of education, and schools themselves, at 692 and 601 respectively. The distribution of the curriculum to the levels of organizations varies a great deal over the counties. However in most cases the bulk of each county's supply of curriculum was in the hands of the education sector with few exceptional counties having greater involvement by the county health department (Sacramento, Los Angeles, Santa Clara, San Joaquin, Nevada).

Considering the overall distribution by contact method, the largest number of copies distributed (n= 830), or 25%, were through contacts at conferences.

This was followed by mailers with phone call follow-up (n= 653) or 21.7%, and then orders generated by listing in the newsletter (n= 436) or 14.5%. Each of the other four contact methods accounted for less than 10% of total distribution.

When teachers and nurses were seen as units of analysis a somewhat different picture emerges. From attempts to contact the 191 people who ordered multiple copies (representing 2597 copies), the status of 2204 copies were tracked by telephone follow-up and returned logs. The overall effectiveness of the distribution effort shows that in fact curricula were received by only 430 teachers and 308 nurses, those most likely to use the curriculum in the classroom. Thus of curricula tracked, only 33.5 % of those curricula were in teachers and nurses possession by the end of the distribution effort. When these totals were combined with those teachers and nurses who had ordered directly (n = 175), 913 copies out of the 3000 produced, or 30.4%, went to professionals who could take Immunization Plus! into the classroom.

Of the copies that did not reach a teacher or nurse, 833 copies, or 37.7% of those distributed which could be tracked, were in county, district or principal's offices, had been given to non teaching staff, or were in the process of being evaluated for future use. Another 633 or 28.9% were unused, warehoused, or lost. Thus more 50% of the copies sent out were not in use a year in the year following distribution.

These findings lead to two inter-related questions. First, of the seven contact methods used, which were the most effective in getting Immunization Plus! to teachers or nurses? For teachers, a teacher in-service was by far the most successful means to distribute curricula, with 85% of participants getting a curriculum as can be seen in Table 1 in Column II. Mailers and curricula sent out to persons on lists achieved a lower rate of success defined as getting curricula to teachers and nurses. For nurses, mailers seemed to be more important at getting the word out and receiving a curriculum.

Table 1. Copies Received by Teachers and Nurses Compared to Copies Tracked by Contact Method
	# copies to Teachers/ Total Tracked	% to Teachers/ Total Tracked	# copies to Nurses/ Total Tracked	% to Nurses/ Total Tracked
How Distributor was Contacted
Cold send	62/279	25.3%	38/351	10.8%
Advisory Committee	25/260	9.7%	------	------
IZ Update ( Newsletter)	60/428	14.1%	53/428	12.4%
Mailer w/phone call	113/478	23.7%	44/478	9.2%
Mailer w/ no phone call	25 / 87	28.2%	31/38	81.5%
Conferences	125/603	20.7%	136/609	22.5%
County Teacher In - Service training	6/7	85.8%	-----	-----
TOTAL	430/2204	19.5%	302/1904	17.7%

The second question is, given the discrepancy between the number of copies sent out and those actually reported received by teachers and nurses, which gatekeepers who ordered the curricula were most likely to pass the materials on to the target populations? As seen in Table 2, for teachers, those most likely to pass on curriculum were other teachers, passing along 43.4% of copies they received. Principals (33.4%) and then nurses (26.4%) followed this. Persons with the worst rates of dissemination to teachers were school superintendents (8.5%), other school personnel (10%) and other health personnel (20%). For nurses, other nurses (21.6%) and the County Immunization coordinators (22.8%) were the most likely to pass the curriculum along, with other health personnel (1.1%), principals (6.7%), and superintendents (1.7%) having the lowest rates.

Table 2. Copies Received by Teachers and Nurses Compared to Copies Tracked by Type of Distributor
	# copies to Teachers/ Total Tracked	% to Teachers/ Total Tracked	# copies to Nurses/ Total Tracked	% to Nurses/ Total Tracked
Position of Distributor
Other Health Personnel	60/376	16.0%	4/376	1.1%
District Coordinators	97/476	20.4%	82/476	17.3%<
Nurses	95/361	26.4%	78/361	21.6%
Teachers	62/143	43.4%	9/143	11.4%
Immunization Coordinator	-----	-----	47/101	22.8%
Other	27/270	10.0%	-----	-----
Other School Personnel	79/398	19.9%	80/398	20.1%
Principal	5/15	*33.4%	1/15	*6.7%
Superintendent	5/59	*8.5%	1/59	*1.7%
TOTAL	430/2204	19.5%	308/2075	14.8%

Table 3 shows the cost per copy distributed for the seven major distribution methods used. For general distribution to all contacts in California and out of state, not taking into account secondary distribution by recipients, the IZ Update newsletter had the lowest cost per quantity distributed at $0.88 per copy. The connections of the Advisory Committee also proved to be a relatively effective method to create orders and distribute curricula to a general audience. The next least costly methods are statewide conferences for teachers, nurses, and health workers, and the mailer with a follow-up telephone call. However these two latter methods cost more as they are labor intensive.

When considering cost of methods used to contact of teachers and school nurses, the parameters change as the cost of distributing curricula to professionals who teach the material is greater than general distribution.

The least costly method to reach teachers and school nurses was the newsletter, followed by the Advisory Committee contacts and attendance at conferences (Table 3). The most costly means of getting curricula to teachers and nurses were mailers with no phone calls and sending out unsolicited copies to identified gatekeepers (Cold Send). A countywide in-service training was only moderate costly and quite effective in getting curricula to teachers. (See Table 1)

Table 3. Cost analysis of Immunization Plus! General Distribution and Teacher /Nurse Distribution
	General Distribution				Teachers and Nurses
How Distributor was Contacted	# Copies Distributed	Overall Cost of Method	Effectiveness- Cost Ratio	Cost per copy Distributed	# Copies Distributed	Overall Cost of Method	Effectiveness- Cost Ratio	Cost per copy Distributed
Cold send	604	$10,166	.059	$16.83	111	$10,166	.001	$91.16
Advisory Committee	230	$520	.440	$2.26	29	$520	.060	$17.93
IZ Update ( News letter)	436	$384	1.13	$0.88	60	$384	.150	$6.40
Mailer w/phone call	662	$4,831	.130	$7.29	192	$4,831	.040	$25.16
Mailer w/ no phone call	212	$7,238	.030	$34.14	95	$7,238	.013	$76.18
Conferences	831	$8,438	.090	$10.15	373	$8,438	.044	$22.05
County Teacher In - Service training	24	$655	.036	$27.29	24	655	.036	$27.29

This study describes the dissemination of a 6th grade curriculum on infectious disease and immunizations to teachers, nurses, school district, and county personnel in California. Out of an initial set of 2975 copies, approximately 2204 were successfully tracked over the first year, revealing that only 738, or 33.5% of the known copies, were in teachers' or nurses' possession. Many copies were warehoused, not distributed to teachers or nurses, or otherwise unused after the first year of the project. Yet unless these curricula get into the hands of teachers and nurses they will not be used. Among the teachers and nurses who had received the curriculum and were followed up by a survey, less than a quarter had actually used the materials.

From a theoretical standpoint such a low level of adoption is predictable. Specifically based on diffusion theory, documentation of the dissemination process for the initial phase of a campaign can be likened to assessing the initial stage of the adoption cycle, where utilization is low and adopters are more innovative than non-adopters. As both the curriculum and subject matter were relatively new, it is likely that those most interested in acquiring or using the curriculum were "early adopters" (Rogers, 1973). As well, the curriculum dissemination study occurred at the beginning of a larger statewide social marketing campaign to promote adolescent immunizations. From anecdotal reports in the years since this study, and as more copies have been circulated, more widespread adoption and use seems evident. As well, results fit McGuire's theories as regards to the probability of changing behavior through mass marketing of materials and messages: actual rates of behavior change are relatively low based on large scale diffusion efforts and until some degree of market saturation has been achieved (McGuire, 1972). Findings from other studies also suggest that low rates of utilization of school based curricula are common. (Parcel, et al, 1989; Brink, et al, 1991).

A number of substantive issues are suggested by the findings. There are multiple barriers both internal and external to schools that impede adoption of new curriculum. While school districts and teachers have some autonomy in selection of materials, there are profound disincentives at present for many teachers to adopt much that is new, especially in health related areas where they have not been trained in the subject matter. Basic curricular demands, lack of time, job stress, and lack of incentives may preclude their motivation to adopt new material. The time frame for adoption and use of new material is often a longer process than a year as in many school districts materials need to be approved by the School Board and then incorporated into lesson plans, well in advance of the school year. Thus sufficient lead time must be allowed.

Another issue has to do with the notion that some teachers do not feel comfortable teaching about health, unless they receive training. Nurses expressed an interest in teaching the subject matter but are also often too busy with regular duties to venture into the classroom, and when they do it is often for short-term teaching. Despite these barriers, for those teachers who had taken the time to assess the Immunization Plus! curriculum, the response to teaching this subject matter was overwhelmingly positive. This group of teachers however may be somewhat self-selected in regards to their interest in health issues as well as ability to try new things.

As shown in the tracking and follow-up process, many of the barriers to getting curricula into teachers and nurses hands are organizational. Often persons who ordered the curriculum were not teachers and nurses: ideally they would pass these materials on to those who would teach them but our data seems to indicate that often this transfer did not occur in a timely way. The issue of layers of gatekeepers holding up the distribution of materials has been found in other school dissemination studies(Brink, et al, 1991).

This study has given some insight into methods of distribution that do not have the desired effect. Some of the least successful results were linked to sending materials to persons with no personal contact. The best methods of promoting the Immunization Plus! curriculum involved personal contacts. This has been shown elsewhere to be advantageous in the promotion of curricula (Roberts-Gray, Solomon, Gottlieb, Kelsey, 1995). A second method that worked was sending mailers to schools and school districts targeting school personnel and district coordinators of health programs, especially when followed up with phone calls. Personal contact, even as minimal as a follow-up telephone call, proved effective especially when this contact was the source where educators obtain their information. Finally the traditional training model must still be considered as the most effective way of getting a curriculum to teachers and nurses, and helping to motivate persons to use it, even though training may not be the most cost efficient method over the short term.

The study suggests a number of concerns when promoting health curricula in schools. First, curricula were ordered more frequently by persons in large urban school districts, possibly a function of having more resources available to scout out and order new curricula. Second, most teachers and school personnel did not seem to be aware of current immunization requirements for adolescents. Thus, it is not surprising that the content covered was not a priority for most school systems. As a non-mandated program it was seen to be extra or optional, and was in fact in direct competition with Drug, Alcohol and Tobacco education programs and curricula which are mandated by the state for the middle grades.

Another issue that became apparent was the very different cultures of schools and health departments in most locales (Glik, et al, 1997). Goals and organizational structures of the two systems are often incompatible, and sending curricula to county health departments did not often translate into entree into local school systems. Many teachers and districts did not see non-critical health topics as a primary educational mandate. Moreover while state educational organizations can set policies or create guidelines about curricula, local school boards and districts retain the rights to decide what should be taught and how. Thus marketing and promotion of curricula does have to focus at the local level, a process that is resource intensive and difficult.

In sum, disseminating materials to schools is a complex process with multiple layers of entree. While there is a great deal of resistance to curricula, there is also a great deal of acceptance: essentially we were able to get orders for almost all curricula we had printed within 6 months. To disseminate curricula effectively resources need to be put into teacher training or other interpersonal means of advocating for increased utilization of health promotion materials.