IEJHE, Vol. 3(1), 44-54, January 1, 2000, Copyright © 2000

Charting Your Course: Formative Evaluation of a Prostate Cancer Treatment Decision Aid

Corresponding author: Susan G. Brink, DrPH; Principal, HealthMark Multimedia, LLC; 1870 Wyoming Ave, NW Washington, DC 20009; phone: 202.265.0033; fax: 202.328.3210; email: SGBRINK@HEALTHMARKMULTIMEDIA.COM.

Men with prostate cancer are faced with making a treatment decision. Charting Your Course is an interactive CD-ROM that educates patients and facilitates treatment decision-making. This paper reports formative evaluation results leading to a prototype. Prototype test results indicate an increase in knowledge about brachytherapy and staging, and in perceived self-efficacy in discussing brachytherapy. The fully developed CD-ROM is expected to facilitate discussion and shared decision-making among patients, their partners and their physicians.

Key Words: prostate cancer, health care, evaluation

Prostate Cancer in the United States

For U.S. men, cancer of the prostate is the most commonly diagnosed noncutaneous cancer and the second leading cause of cancer death (American Cancer Society, 1998). A fifty year-old male in the U.S. today has a 40 percent chance of developing prostate cancer and a nearly three percent chance of dying from it (Garnick, 1993). Recent changes in screening and diagnosis of prostate cancer have led to more and younger men being diagnosed with this disease. These changes have begun to shift the perception of prostate cancer away from that of an "old man's disease" to a life-threatening disease that can strike any man.

Prostate cancer, especially in its early stages, is a treatable disease. Survival has increased over the past 25 years, with a dramatic difference in survival by stage at diagnosis (Stanford et al., 1998). Nearly all men diagnosed with local or regional tumors survive 5 years, compared to only a third of men diagnosed with distant metastases (Stanford et al., 1998). Diagnosing men in the early stages of prostate cancer allows them treatment choices that can improve their survival.

Treatment Options & Side Effects

Treatment options vary by the stage of the cancer and the age of the patient, and involve sequelae that impact on both medical and personal life issues, such as impotence, incontinence and pain management (Fowler, 1995; Williams & Love, 1996). The optimum treatment for localized prostate cancer is often elusive (Coleman & Kaplan, 1993; Palmer & Chodak, 1996). Options for tumors contained within the prostate capsule include "watchful waiting," hormonal treatments, surgery and at least two forms of radiotherapy (ACS, 1998; Stanford et al., 1998). Aggressive treatments, such as surgery and brachytherapy (radiation seed implants), that are intended to remove or kill the cancer cells before they have spread beyond the prostate capsule are less appropriate for distant-stage tumors that have reached the lymph system or other tissues (Garnick, 1993). Disease at these stages is controlled through the use of hormonal therapies, palliative radiation and some experimental chemotherapies. Each of these treatment options can cause significant and potentially long-term side effects, including incontinence, impotence, loss of libido, depression, and chronic pain (Garnick, 1993).

Patient Education & Treatment Decision-making

With shifts in the affected population and available treatments comes an increased need to educate patients and to help physicians facilitate their patients' treatment decisions. Once diagnosed, patients must receive comprehensive and understandable information in a manner that encourages thoughtful dialogue and shared decision-making between patient and physician (Coulter, Entwistle, & Gilbert, 1999; Garnick, 1993; Fowler, 1995; Mazur & Merz, 1996; Litwin & deKernion, 1994; Wagner, Barrett, Barry, Barlow, & Fowler, 1995). When making treatment decisions, it is important that the patient be aware of all possible treatments, including their respective benefits, risks, side effects and personal preferences regarding both good and bad outcomes be explored (Fowler, 1995; Mazur & Merz, 1996; Garnick, 1993). With recommendations and support from their physicians, patients can-- many would argue should-- weigh their options and arrive at a treatment decision, taking into account their medical situation and personal preferences (Davidson & Degner, 1997; Mazur & Merz, 1995, 1996; Litwin et al., 1995; Montie, 1993).

Multimedia Approaches to Patient Education

Multimedia tools, such as videodiscs and CD-ROMS, have been used to educate adults in health promotion and disease prevention, as well as to facilitate preventive health and treatment decision-making (Wagner et al., 1995; Schapira, Meade, & Nattinger, 1997; Onel et al., 1998; Kim & Watson, 1995; Krishna, Balas, Spencer, Griffin, & Boren, 1997; Barry, Fowler, Mulley, Henderson, & Wennberg, 1995). Randomized trials of computerized patient education materials indicate that such interventions increase patients' knowledge and ability to ask questions, especially when used to supplement face-to-face interactions with physicians (Krishna et al., 1997). Multimedia decision-support systems have been shown to have high patient acceptance and to provide increased flexibility to both the patient and the physician (Kim & Watson 1995). In addition to educating the user, such a system can assist medical personnel with counseling during diagnosis and treatment. After patients have used multimedia educational products, patients and their physicians report increased sophistication in their discussions of treatment options, moving beyond description of options to focus on the risk and benefit tradeoffs of treatments (Onel et al., 1998).

An interactive decision-support system can not only help men with prostate cancer learn about their disease and its treatments, but can tailor information to each patient's medical characteristics and personal preferences. Decision-support activities can help patients sort through treatment options and their medical and personal concerns, improve their interactions with physicians, and arrive at a decision that balances their concerns. One videodisc-based system has been developed to assist patients with treatment for benign prostatic hyperplasia (Barry et al., 1995; Wagner et al., 1995). Unfortunately, videodisc technology is both uncommon and expensive. Patients and their physicians need an inexpensive, accessible product that allows patients to fully explore many treatment options and manage a complex array of treatment-related decision factors.

This paper describes the formative evaluation process for a CD-ROM prototype that helps men recently diagnosed with prostate cancer learn about the disease and its diagnosis and treatments. It describes how the data collected in early stages of the process were incorporated, adapted and refined throughout the development process. It also raises and addresses a myriad of questions that arise as patients and their family's struggle with complex treatment decisions. The evaluation process also uncovered additional issues that will be addressed in a full and final version of the CD-ROM.

Charting Your Course: Prostate Cancer Charting Your Course: Prostate Cancer is a 30-minute CD-ROM prototype that establishes the user interface and a prototype decision-support tool. It also provides in-depth information in two primary content areas: stages and grades of prostate cancer and one treatment option (brachytherapy, or radiation seed implants). The latter includes medical criteria and procedures for the treatment, pros and cons of the treatment, one patient's personal story, and practical advice on "what to expect" before, during and after treatment.

The program opens with narration by "Joe," a prostate cancer survivor. Joe captures the user's attention with a brief description of his experience in learning about prostate cancer and the importance of taking the time to explore treatment options. Leading a "cast" of other patients and several medical providers, Joe serves as the narrator throughout the product, introducing topics, providing navigation cues, and offering plain-language explanations of medical information. The prototype decision-making aid prompts users to enter relevant information about their own medical situation and personal preferences, and provides an assessment of brachytherapy as a treatment option under those conditions.

The final product will offer patient-targeted information on the entire spectrum of treatment options, as well as support material such as a hypertext glossary, a gateway to Internet resources, personalized lists of questions for doctors and partners, and an index of cancer-related organizations and other resources. The interactive decision-support component will allow users to compare all treatment options based on the personal and medical information they provide.

Based on a model of treatment decision-making and a broad knowledge of the outcomes and side effects of prostate cancer treatments, a schematic of the full CD-ROM was developed. Through an iterative process involving both quantitative and qualitative data collection and analysis, the scope of content, presentation level, nature of the user interface, and usability of navigational elements were determined and refined.

Prostate cancer support group facilitators, several of whom were also cancer survivors, were invited to provide input on the concept of a patient-focused multimedia decision-support tool, to react to the model and schematic, and to identify issues to be addressed with patients and physicians during the formative evaluation process. These informants also offered direction on patients' education and decision-making needs, and the proposed content of the CD-ROM.

To help shape the CD-ROM, two waves of data-gathering activities were conducted with prostate cancer patients and their partners. In both waves, subjects participated in semi-structured qualitative interviews or focus groups, and also were asked to complete a brief written survey about their information-seeking strategies and decision-making process. Combining qualitative with quantitative data, broad parameters for the product scope and presentation of information were established. A second wave of data collection captured subjects' reactions to the overall user interface, preliminary design elements, and content for several key components of the CD-ROM. Table 1 summarizes the formative evaluation process, in which messages and themes generated by subjects were developed, tested and refined within prototype components. The table also indicates how elements will continue to be refined in the final CD-ROM.

Patient & Partner Focus Groups & Interviews

The protocol for the first wave of focus groups and interviews elicited qualitative data about the experiences, information needs and decision-making processes of newly diagnosed prostate cancer patients and their partners. It also explored preferences related to the tone and style of presentation for a CD-ROM. A total of 23 men and 13 women participated, 13 of whom were African American, 1 Mexican American, and the remaining 22 were Caucasian.

All focus groups were audiotaped. A non-participating observer recorded detailed notes. Interviewers using essentially the same protocol kept their own detailed notes. Notes prepared by each observer/interviewer were compiled into a single data set and analyzed for patterns in information and decision-making needs, and for preferences related to the user interface.

Information Needs and Decision-Making Process

Many participants identified a need to gather information on multiple treatment options and from multiple sources. Inherent in this theme, particularly for men, was a desire for balanced, "unbiased" information across treatment options. Specifically, respondents recommended that the CD-ROM encourage patients to seek medical opinions from physicians of different specialties (e.g., urologist, oncologist, radiologist), acknowledging that each brings a unique perspective to the discussion of treatment options. Many informants urged inclusion of links or references to additional information resources, including specific documents, information, support and advocacy organizations, and the Internet.

Both men and partners indicated that the most important factor they considered when making treatment decisions was survival rates. However, on reflection of their own experiences, they encouraged development of elements that motivate users to carefully consider the significance and day-to-day implications of potential side effects, like incontinence and impotence, during the decision-making process. Several informants indicated that lack of knowledge about multiple treatment choices and their side effects prior to decision-making and feeling rushed to make a decision ultimately reduced their acceptance of the side effects they encountered.

Informants of both genders raised a variety of issues related to communication and involvement of partners and other family members during the decision-making, treatment and recovery process. A nearly universal recommendation was to present information on the experiences of other patients and their partners. Both men and women recommended including multiple personal stories from patients to complement similar information presented through other methods. Women, in particular, recommended that the CD-ROM address the practical aspects of care-taking and anticipating the patient's needs before, during and after treatments. Many informants agreed that survivors, friends and acquaintances provide a valuable perspective on side effects, partner issues and practical advice on "what to expect" with treatments and progression of the disease.

Tone, Theme and Style of Presentation.

Informants generally agreed on issues of tone and style of presentation. Both men and women tended to recommend a hopeful, non-threatening tone, including: light/bright, but realistic, colors; stories of successful treatment decisions and solutions to overcoming side effects; active, but not frenetic, pace; and a clean-cut, professional appearance. Several informants preferred to be able to control the pace and navigation of the CD-ROM, such as through "Replay" and "Pause" features. All informants preferred a straightforward presentation style that is neither "sugar-coated" nor alarming, especially when presenting frightening or disheartening information.

Testing & Refining the Product Concept

Based on results from the initial wave of data collection, the schematic for the full CD-ROM was modified and sample CD-ROM screens (i.e., electronic storyboards) were developed. The second-wave focus group protocol included open-ended questions about the theme and user interface, design elements, content, and navigation of the storyboards. Participants were men who had already researched and made initial treatment decisions, and their female partners. Forty (40) men and 16 women participated in 12 focus groups. Thirty-one (31) participants were Caucasian and 25 were African American. Reactions to the storyboards were remarkably consistent across race and gender. Again, focus groups were observed and detailed notes recorded. Notes were compiled and analyzed for patterns in feedback to the content and usability of the storyboards.

Overall Theme. Participants generally liked the color scheme, but felt it should be lighter, brighter and more contrasting. Participants were split on their reaction to the nautical navigation theme. Many perceived it as masculine and analogous to the information-seeking and decision-making processes. Some, however, were not inspired by the imagery and felt the design should be more powerful.

Design Elements. A wide variety of specific design elements were tested. For instance menus, buttons/bars, and text were presented in several different styles, formats and layouts. Participants generally agreed that the layout and style of navigational elements should be consistent throughout the CD-ROM to the extent possible. Buttons should be clearly defined and labels should make the purpose of navigational elements obvious (e.g., "Next" button rather than "¨").

Content. The primary focus of the sample screens was the initial treatment decision-support tool. As proposed, the CD-ROM would prompt the user for specific medical and personal information and the computer would generate a graphic indicating treatment options that best fit the user's situation. The general concept of rating treatment options based on personal characteristics was appealing to most participants. However, a majority of both men and women disliked the graphical interface and had difficulty interpreting the treatment rankings. Nearly all participants recommended a graphical interface with a more "visible" and logical process of gathering personal information and arriving at possible treatment options.

Navigation. Navigation was the most problematic aspect of the initial storyboards. Informants identified numerous navigational elements that were missing (e.g., lack of voice-over instructions and beeps) or inherently difficult to use and understand (e.g., illogically placed buttons or arrows, unclear labels and unlabeled symbols). Participants in all groups also expressed difficulty in understanding the use of the pop-up menus, buttons, and other elements to move between screens, particularly within the decision-making storyboards. All groups offered suggestions for text, audio, and structural changes to improve the user's ability to navigate logically through the information and decision-making process.

Technical Review & Utilization Interviews

In addition to pursuing input from patients and their partners, advice was sought from four physicians (e.g., two urologists/surgeons, a radiation oncologist, and a family practitioner), who served as technical advisors and provided input on strategies for integrating the CD-ROM into clinical practice. The physicians reviewed the script for technical accuracy, as well as for style and level of presentation. After the prototype was programmed and debugged, they also reviewed the prototype and participated in an open-ended interview that further explored their preferences regarding content and presentation, identified factual errors and inappropriate or inflammatory presentation of content, and solicited their recommendations for additions and modifications. The interview also included issues related to physicians' intentions, motivation and process for using a CD-ROM patient education tool in the clinical setting.

After exploring the prototype, all four physicians approved of the product and suggested modifications. The physicians particularly appreciated the multimedia presentation of the Gleason Score, which they judged to be simplified and yet accurate, suitable for the needs of patients. Physician-suggested modifications included removal of tests that are no longer used, increased specificity in word choice, modified wording concerning PSA "screening," and increased labeling on diagrams. The physicians offered their personal intent to use the CD-ROM when it is completed, in addition to providing suggestions for integrating its use into their institution or clinical setting. Remarks included: "All of our physicians should see this", "We can get this into the cancer education center"; "We should have computers on the hospital and clinic floors so patients can access these materials."

Testing the Prototype with Patients

Subjects were given up to 45 minutes to explore the prototype with minimal direction beyond a basic orientation to using the computer. Subjects were observed by one recorder, who also administered a semi-structured usability interview. Pre- and post-tests included questions related to the subjects' perceived self-efficacy in interacting with physicians about treatments, as well as knowledge questions related to two of the prototype's primary content areas (e.g., staging and brachytherapy). Several additional post-test questions and the post-test interview focused on the CD-ROM's usability.

Prototype Test: Outcome Measures & Analysis

Self-Efficacy. Perceived self-efficacy in understanding the treatment of brachytherapy (radioactive seed implantation) and discussing brachytherapy with physician and partner was measured using the procedure described by Bandura (1986). Based on prototype content, six questions, each beginning with "How confident are you thatÉ," were developed. Subjects rated each statement on a five-point scale from "very confident" (1) to "not at all confident" (5). Responses were averaged to form an overall self-efficacy score.

Knowledge. Knowledge of the prototype content was measured through a series of eight True/False questions that addressed specific facts about brachytherapy (4 questions) and about staging (4 questions). The number of correct responses was averaged for each subject to provide a knowledge score for the subject. Mean knowledge scores ranged from 0 to 1.0, with 1.0 an indication of four correct answers and 0 indicating no correct answers.

Program Elements. Subjects were queried their preferences regarding seven program elements: title, colors, navigation, personal story, presentation of visual information, presentation of narrated information, and quantity of information. Subjects rated each statement on a five-point scale from "like it a lot" (1) to "don't like it at all" (5). Frequency distributions for each element were used to determine areas where there was less than 75% agreement on liking the component of the program.

Data Analysis. Change in self-efficacy and knowledge scores pre- to post-test was analyzed with two-tailed paired t-tests, alpha = .05. Frequency distributions were used to examine the post-test ratings on the prototype usability and program components.

Prototype Testing Results

Forty-three (43) subjects tested the prototype, 29 men with prostate cancer and 14 female partners. Forty percent of the subjects were African American (N = 17); 51% were Caucasian (N = 22) and 7% were Hispanic (N = 3) . One subject was under 50 years of age; 33 subjects were between 50 and 69 years; and 8 subjects were 70 years or over. One subject did not report an age. Forty-seven percent (47%) had been diagnosed within the past year. Education level was available for 32 of the respondents. Levels ranged from non-high school graduate to college graduate. Twenty-one percent (21%) of the 32 subjects had no formal education beyond high school; 20% had some college and the remaining subjects were college graduates. Almost half of all subjects indicated that they did not know the stage of their cancer at diagnosis.

Self-efficacy & Knowledge. Table 3 summarizes the results of outcome measures for self-efficacy, brachytherapy knowledge and staging knowledge for all subjects and men only, respectively. All three outcomes were statistically significantly different from pre-test to post-test. The largest differences were in brachytherapy knowledge change, with effect sizes of 53% (all subjects) and 63% (men). Changes in knowledge of staging and self-efficacy were smaller, with effect sizes of 9% (all subjects) and 12% (men) for staging knowledge and 24% (all subjects) and 23% (men) for self-efficacy.

These results led to considerations for the final product and its evaluation. Brachytherapy information was provided by voice-over, bulleted text and diagrams. The staging information was bulleted text and diagrams, with no voice-over. It is clear from the data that the subjects knew more about staging than brachytherapy before the test (Table 2). However, it may also be true that the added impact of the audio increases short-term learning. The complete product will have audio voice-over for all information.

Analysis of the individual components of the self-efficacy measure indicated that three of the questions made the greatest contribution to significance. After using the CD-ROM, subjects were significantly more confident in their ability to "discuss issues around brachytherapy with my doctor" (p <.0001); that they "know how brachytherapy would effect my life" (p < .0001); and that they "understand the side effects of brachytherapy" (p=".001)." No change was found in measures of locating information to make a treatment choice, working with their doctor to make a treatment choice, and discussing brachytherapy with their partner.

These results indicate that the CD-ROM is likely to produce a change in knowledge and that a change in self-efficacy is a real possibility. This was a test of a short prototype, with information confined to four specific areas. These measures necessarily are of short-term gain. All exhibit the response bias inherent in any test where the user is being observed. The field test in Phase II will eliminate these concerns by allowing the subject to use (or not use) the product under unobserved personal conditions and on their own schedule.

Program Elements. The majority of subjects approved of the following elements: the name Charting Your Course^î (95%); the colors (88%); the presentation of information on the screen (91%); the presentation of information by narration (91%); and the amount of information presented (79%). Comments from subjects indicated that they would prefer more information in greater depth, rather than less. Two elements received less than 75% approval, suggesting that there remains room for improvement: 62% liked the personal story, and 62% liked the navigation.

In addition to subjects' written responses, subject observations and responses during the usability interview were analyzed for overall reaction, preferences, and suggested modifications. Many prototype testers suggested improving the personal story concept by portraying of a variety of people and perspectives, taking care to cover all treatment outcomes. Subjects also encouraged the use of video to assure that they were "real people," rather than fictional characters. Although much improved from the comments received on the storyboards, the element most in need of improvement continued to be navigation. Subjects offered suggestions to help users better understand their movement through content areas, such as incorporating an audio cue to signal the end of all voice-overs, adding voice-over instructions in potentially confusing navigational situations, and highlighting menu items after they have been accessed.

There is an upward bias by education in our samples, with 79% of the prototype testers having attended at least some college and 59% having graduated. The prototype testers also tended to be younger than many men diagnosed with prostate cancer (89% were younger than 70 years). However, these biases are both likely to be true of patients who would use a computerized patient education tool, at least for the foreseeable future. The prototype test results also can be expected to exhibit response bias, such as is inherent in any test where subjects are observed.

Qualitative Input

Input from physicians and support group leaders, combined with two waves of qualitative input from patients, resulted in a CD-ROM prototype that was acceptable to both patients and physicians. Design elements and the overall scope of content were based largely on recommendations and ideas proposed by members of the target population.

The multi-wave development process also allowed conceptualization and refinement of a patient-focused treatment decision-making aid. Patients clearly wanted a straightforward mechanism to help them weigh the pros and cons of each treatment option in the context of their personal situation. Patients also want a tool that does not intentionally bias a patient toward any single treatment. Input from both patients and physicians on the concept and interface of the CD's decision-making component is promising, but clearly needs refinement before it will be fully functional. Based on user feedback, the interface used to compare treatments and personal preferences should be more logical and easier to interpret at a glance. Patients and their partners also thought it important for the decision-making tool to prompt the user to consider the impact of treatments beyond survival.

Several important content areas and usability issues uncovered during prototype development will be incorporated into the final CD-ROM. Many women, and some men, who participated in the formative evaluation and prototype test clearly are interested in information on family communication and roles in decision-making, as well as issues related to care-giving. Further development of the CD-ROM will include data collection and development on these issues, as well as issues of lifestyle change, sexual function and intimacy, and support for other family members. Other proposed concepts that received positive feedback from participants were links to other information and support resources, interactive tools for developing lists of questions for doctors and family members. Patients and partners also were interested in more information on emerging and complementary treatments, such as diet supplements, vaccines, and emerging chemo and genetic therapies. We were constantly reminded of their need and desire for clear, in-depth information.

Prototype Test

The prototype test was performed on less than a tenth of the content proposed for the full CD-ROM, with information confined primarily to three content areas (i.e., staging, tumor grading, and brachytherapy). The outcome measures are of short-term gain and duration of the effect is unknown. Nonetheless, these results indicate that the CD-ROM is likely to produce a change in knowledge and that a change in self-efficacy is a real possibility.

Knowledge change was more dramatic for brachytherapy than for staging. As expected due to the relatively recent acceptance of brachytherapy as a treatment option, the prototype testers' knowledge of staging was higher than for brachytherapy before viewing the CD-ROM (Table 2). Furthermore, the brachytherapy information was conveyed in the prototype via voice-over, bulleted text and diagrams. Presentation of staging information included bulleted text and diagrams, with no voice-over. It is possible that the added impact of the audio increases short-term learning. The complete product will have audio available for all information.

The discrepancy observed in self-efficacy scores between well-developed and poorly developed content areas indicates that further development and evaluation is needed to meet objectives of improving patient information-seeking, patient/provider communication, and patient/partner communication.

Conclusions

A prototype for a CD-ROM, Charting Your Course: Prostate Cancer, was developed using qualitative and quantitative formative evaluation methods. The process involved multiple stakeholders, patients and their partners, physicians and support group leaders. The resulting product combines educational content with a decision aide that addresses medical issues and personal life issues important when considering prostate cancer treatment. The prototype test showed significant change pre to post in knowledge and self-efficacy. Physicians who reviewed the prototype indicated interest in using such a product with their patients.