Measuring our Impact: Meet the Evaluators

The Wellness Coalition (TWC) and its partners have implemented six different health interventions over the last four years, as part of a REACH cooperative agreement with the Centers for Disease Control and Prevention (CDC). Each intervention involved the collection and evaluation of data as a way of measuring the program’s impact. A team of evaluation professionals worked with TWC staff and REACH partners to develop the instruments used for collecting data. These instruments often took on the form of surveys and questionnaires designed to collect information from program participants prior to their involvement with the intervention (“baseline data”). Subsequent collections of information from that same participant were also collected after their involvement with the intervention to document changes in behavior and any other measurable outcomes (“follow-up data”). After tabulating this collected data, the REACH evaluation team matched cases of baseline and follow-up data belonging to the same participant, and then determined positive or negative changes experienced by participants.

Don Bogie, PhD, a retired sociology professor and experienced evaluation consultant has been leading the REACH evaluation team in concert with Jerry Ingram, PhD and Stacey Ingram of the evaluation consulting firm Southeast Research, Inc. Retired since 2007 from an academic career that began upon receipt of his doctorate from University of Kentucky, Dr. Bogie has been doing evaluation work since the 1980s on projects with federal agencies that include the Substance Abuse and Mental Health Services Administration, the U. S. Department of Education, the U.S. Department of Housing and Urban Development, and the CDC.

The Montgomery-based firm Southeast Research, Inc., also has a great deal of experience with the evaluation of federally and privately funded projects at various nonprofit organizations. Dr. Ingram, with a PhD from University of Arkansas, and Stacey Ingram, with a degree from Huntingdon College, have served as evaluators for the Council on Substance Abuse, the Family Sunshine Center, the Turning Point Foundation, and Alabama Arise. Like Dr. Bogie, they have a passion for statistics and the power of data to inform and guide.

This evaluation team was recently interviewed about their impressions of the program and their experience with the TWC staff, the CDC, and other REACH partners.

Would you mind letting us know a little bit about your experience working with the CDC as well as with TWC’s staff and partners?

Dr. Bogie: CDC personnel stay in close contact with the grantee and provide expert guidance and direction throughout all phases of a project. TWC staff care deeply about the health needs of the community and possess the training and expertise necessary to generate positive change across diverse sectors of the population. Not to be overlooked is the excellent data entry and processing assistance that has been provided to us by the REACH staff (Jen Prater, Deona Tucker, and CDC Associate, Justin Alexander) and Southeast Research, Inc. (Stacey Ingram). Together, Jen, Deona, and Justin entered data from thousands of questionnaires during the last four years and Stacey has produced hundreds of frequency tables and cross tabulations, as well as scores of reports for the REACH staff, the CDC, and our partners and stakeholders.”

An evaluation of the REACH community action plan involved both “process evaluation” and “outcome evaluation”. Could you define these two types of evaluation and give us some helpful examples?

Dr. Bogie: The process portion of the evaluation has enabled us to track the number of participants in our various interventions and their characteristics. It is important for us to know if we are reaching our target populations and to what degree. Likewise, continuous monitoring of survey results has enabled us to evaluate whether our data collection instruments are measuring what we intended to measure and, if not, where to make adjustments. Basically, it is essential that we know where we are at any given moment in delivering interventions—and ‘process evaluation’ is designed to tell us that.

As important as process evaluation is, the most exciting part of any evaluation project to me is looking at outcomes. Whether an intervention has actually made a difference in the lives of people is the ultimate goal. As an evaluator, it has been especially gratifying to see numerous positive changes in health-related behaviors and characteristics across all six of our interventions. These include the sale of over 20,000 units of fresh fruits and vegetables in 15 corner stores where they had not been previously available; the establishment of 11 new community gardens that helped to increase the availability of freshly-grown produce in several low-income neighborhoods; improvements in blood glucose, blood pressure, and cholesterol levels among patients who participated in our Green Prescription intervention; increases in the number of people with medical insurance, medical homes, and the ability to obtain needed medications for those who took part in our Community Health Worker intervention; increased physical activity for persons who participated in the Living Well Chronic Disease Self-Management program; and weight loss and increased physical activity for those who joined our Decreased-Cost Fitness Center initiative. While many of these changes have been small, they nevertheless indicate that progress is being made.

What barriers did you encounter during your work with REACH and what do you consider to be the strengths of the program?

Dr. Ingram: There were no barriers associated with the REACH project that prevented the achievement of program fidelity. At the same time, if we were starting the REACH project again, there would be some recommended instrument refinements that could improve the data collection process some – but nothing major. The strength of the evaluation process for REACH was actually the hard work and dedication of the TWC team who insisted on quality in the data collection process.

Dr. Bogie: We were able to solve most data-related issues locally through working side-by-side with TWC/REACH personnel, with the CDC as a back-up. Perhaps the greatest challenge we faced during REACH was the coordination of data collection activities for the six REACH interventions all at one time, which entailed developing and pretesting data collection instruments, training staff members and contractors in questionnaire administration and interviewing, coordinating baseline and follow-up data collection, and conducting matched-case data analysis and report writing for each of the six.

What recommendations would you make in terms of areas of improvement?

Stacey: I would like to see more close ties between the various interventions. For instance, linking people participating in the Green Prescription intervention to other programs such as wellness classes, community gardens program, and fitness centers.

Dr. Ingram: Personally, I would like to see more cooperation from the stores involved in the Healthy Corner Store program intervention to stock and display produce items.

Dr. Bogie: I would like to see more time for follow-up data collection. Ideally, an intervention would last long enough for data collection to occur over specified intervals for at least a year, with follow-up data collection approximately six- months thereafter. Unfortunately, time and monetary considerations did not support this model.

All members of the REACH evaluation team found value in the local data that has been and continues to be collected and evaluated. Dr. Ingram put it plainly when he said, “The data… represents documentation that the interventions produced the desired outcomes; they improved the target population’s access to healthy foods and improved healthcare.”

Meanwhile, Stacey foresees the outcomes being shared widely in order to expand the reach of various interventions.

Currently, Drs. Bogie and Ingram are working with TWC staff on the publication of several articles in an effort to share REACH intervention results with health professionals and community stakeholders.

“We want all the good things that have been learned here in the River Region to be used and applied elsewhere,” said Dr. Bogie.

TWC staff members working closely with the REACH interventions value these comments from the REACH evaluation team. From TWC’s staff perspective, there is still much work to be done to interconnect the various interventions to encourage their sustainability. Even so, TWC staff reflected that much effort had been put into linking the interventions, such as the effort of Medical Outreach Ministries to give Green Prescriptions with instructions to weed and harvest at the community garden training program at Restoration Baptist Church.

“Throughout the current REACH community action plan, we’ve been working to link the interventions together and to alert the intervention providers to all that is going on due to the REACH coalition so that their clients and patients could take advantage of healthy opportunities and improve their health,” said Eileen Knott, REACH Community Liaison.

Latrice Lewis, REACH project coordinator, added, “Coalition building takes time and effort and this REACH program has allowed us to build a foundation of mutual support based on the goal of a healthier community. TWC will sustain and strengthen the bonds of the current REACH coalition in years to come and seek further opportunities to expand the scope of healthcare access in the River Region.”

The positive findings of the REACH evaluators will be published and shared with REACH partners this fall. Appreciative of every member of the REACH team, Mrs. Lewis reflected on differences that the REACH interventions made on individual lives. “These results motivate TWC to continue its training and coalition-building efforts,” she said. “We are eager to share this news and celebrate the many community partners who are increasing opportunities for health in the region.”