The Use of Qualitative and Quantitative Methods to Evaluate an
Adult Community Education Program
Christine Langone
F. Richard Rohs
University of Georgia
Determining the impact and success of educational programs
is a challenge for program planners and evaluators. In this age
of accountability, educational program planners must provide
evidence of program outcomes to funding agencies and decision
makers. Knowledge and use of a broad range of evaluation methods
is advantageous to provide an overall view of project activities
and impact. Using traditional quantitative measures of
attitudes toward the program or pre/posttest measures to show
change in beliefs and knowledge provide limited assessment of
program accomplishments. However, through a combination of
qualitative and quantitative methods, a comprehensive view of the
people, processes and impacts can be designed.
Recently, significant resources from private and public
sectors have been channelled at all levels for leadership
development. Such allocations require accountability to assess
the impact of programs (Tait, Scheffert, & Augustinack, 1991).
Limited research to assess program impact in leadership education
has been done. Findings from studies which focus on changes in
behavior are often inconclusive (Tait et al, 1991). Problems in
evaluating leadership programs arise from the multiple
definitions of leadership and the multidimensional nature of
leadership development (Boatman, 1989). This article describes
an approach for evaluation of a statewide community adult
leadership development program using multiple methods throughout
various stages of the program in order to address the problems
stated.
The Need for Leadership
The need for leadership in communities has been cited by
many community development professionals (e.g., Warren, 1972),
and especially for rural communities in the South (Beaulieu,
1990; Soileau, 1990). Leadership development programs can help
ensure an adequate supply of effective leaders who can assist
communities provide quality of life for its citizens (Fear,
Vandenberg, Thullen, & Williams, 1985). Citizen involvement and
leadership are key ingredients in solving the many social,
economic and infrastructure problems which communities face
today. The state of Georgia recognized the critical need for
leadership development as a result of a comprehensive needs
assessment in which communities analyzed local resources and
developed plans for the future. A key finding was the need for a
broader, better trained leadership base. State leaders have
stated that unified, committed local leadership is crucial to
rural development and community survival (Berry, 1986; Niemi,
1991). Although many definitions of leadership exist,
community leadership has a particular arena and focus. A
workable definition for discussing community leadership and
program development has been formulated by the National Extension
Task Force on Community Leadership (1986):
Community leadership is that which involves influence,
power, and input into public decision making over one or
more spheres of activity. The spheres of activity may
include an organization, an area of interest, an
institution, a town, county or a region. Leadership capacity
extends beyond the skills necessary to maintain a social
service and/or activities organization. The leadership
skills include those necessary for public decision making,
policy development, program implementation, and
organizational maintenance.
The nature of the definition of community leadership makes
assessment of program effectiveness difficult due to the nature
of behaviors involved and the diverse arenas for their
application.
Georgia's Community Leadership Program
In response to this critical need for leadership
development, the Georgia Cooperative Extension Service initiated
its Community Leadership: A County Perspective program in 1986.
The Community Leadership program was designed to assist counties
develop and expand their leadership base. The county-based
program was developed and implemented by the University of
Georgia Cooperative Extension Service with co- sponsors from
local chambers of commerce, boards of commissioners, and other
groups interested in community leadership development. This
12-week program is designed to equip local leaders with skills to
manage and direct change in their own towns and cities. The
program provides both an educational setting and a forum for
leaders to discuss current issues. Over the past five years,
this leadership program has been implemented in 90 counties
involving over 3400 participants.
Program emphasis is on basic leadership skill development
for current and emerging leaders. Specific program objectives
were to develop basic leadership skills, identify major local
issues, understand government functions, improve the quality of
life of the county, and help build a solid leadership base for
the county. Thirty instructional hours include topics such as
leadership theory, values, group dynamics, conflict management,
communication, managing community change, understanding
government functions and economic development.
In addition to theory learned, the application of skills in
case studies, simulations, and actual community planning is
stressed. During many class sessions, participants discuss and
prioritize their individual and community concerns. By the last
class, an action plan has been formulated by participants
enabling them to put their skills to work. The Program Evaluation
Design
The Community Leadership program objectives and the
definition of community leadership suggest several directions for
evaluation. Because the real impact of this program is evident
through changes in behavior and ongoing activities of the
participants in their community, a major goal of the evaluation
was to collect data on such activities. A comprehensive approach
to evaluation was used to produce in-depth data from a variety of
sources and has provided a rich description of the myriad program
successes and, in some cases, unanticipated impacts of the
program. Because many program impacts are not evident
immediately upon class completion, a longitudinal view of
participants' activities was needed to fully evaluate program
accomplishments. Documenting program impact of such
community development programs has presented challenges to
evaluators. Frequently quantitative approaches fail to discover
important aspects of program impact. Qualitative methods may
document impact, but often fail to trace direct connections
between programs and impact (Pigg, 1990). In order to use
strengths of both methods, a combination of qualitative and
quantitative methods was used in the design of this comprehensive
evaluation. Each method complements others, and in many cases,
data collected from one stage of the evaluation contributed to
selection and design of later methods.
This evaluation is still being completed and while some data
has been compiled, the focus of this discussion will be on
overall design rather than specific results. The following
discussion outlines the methods of evaluation used, program
components assessed and the types of data collected.
Pre/post test survey
A paper/pencil instrument, used in other statewide
Leadership Development Programs, was modified specifically for
this program evaluation to gain information on:
-number and types of organizations in which class members
participate,
-opinions/attitudes related to county services, facilities
and programs,
-personal assessments of skills, abilities and knowledge
related to leadership and public policy decision making,
-level of involvement in local, state or national political
campaigns or issues, and
-demographic data.
This survey is completed before classes began, after the
last class was completed and one year after completion of the
program. Data from these surveys will provide information about
participant's past and improved experience and abilities related
to leadership skills. Participants are also asked their opinion
of adequacy of community facilities and services. Comparison with
non-participating counties is also being conducted.
Descriptive statistics of central tendency and variability,
T-tests, Pearson Product Moment correlations, and analysis of
covariance will be used to summarize data. Preliminary data
shows levels of change in participation, opinions of community
affairs, and personal opinions of skills and knowledge. Other
areas are less conclusive. Additional data collection methods
can be used to explain these findings. Questionnaire.
An open-ended questionnaire completed by local program
coordinators provided descriptive information about activities of
alumni groups or individuals who have participated in the CL
program. Quantitative and qualitative data includes the number
of alumni groups formed, number of graduates who have run for
office, special activities in which alumni have been involved,
types of community groups formed by alumni, other activities
carried out as a result of participation in CL, number and types
of additional leadership development classes taught, and the
impact of the leadership program on the role/perception of
Extension in the county.
Many of the items on this questionnaire resulted from
preliminary interviews and participant observation of programs,
participants, and alumni. For example, the program was not
designed to enhance the perception of county extension staff
among community citizens; however, repeated comments from agents
and class participants led us to believe that this was a
significant impact. Two questions were included on the
questionnaire to solicit further data on this phenomenon. In
addition, questions relating to the broad range of alumni
activities originated from anecdotes and interviews. Anecdotal
Information.
Through conversations with Extension staff (county and
district), community advisory committees and alumni groups, a
great deal of information has been gained about activities of
program graduates. Much of this activity evolves over time.
Records of success stories are recorded to provide case studies
of individuals, counties or groups. This data was used to design
the open-ended questionnaire and interview questions. Document
Analysis.
A series of program documents provides a picture of
community participation in the programs, on-going activities,
success stories, etc. Such materials provide data about program
processes and outcomes which cannot readily be observed. Further
use of documents is to focus interviews and to assist in the
selection of participants to interview. Program documents
include:
News articles. Counties are requested to send copies of
news articles from their local papers which document program
visibility and participant activities.
Record of local contributions. Each county provides
information about the amount of money contributed to run the
CL program. These figures indicate the level of local
commitment to the program and identify co- sponsors and
donors.
Participant selection checklist. Each county provides its
participant selection checklist which indicates the
age,race, sex, and other demographic information about the
make-up of the class. Overall summaries of this information
will provide a picture of the diversity of clientele involved.
Interviews. Data gained through interviews concerning
participant actions, attitudes, and perceptions of the
program will document grass-roots impact. In addition to
class members, advisory committee members and co- sponsoring
groups will provide their view of the impact of the class.
Interview data will supplement previously collected data, provide
greater depth of knowledge about activities reported in the
questionnaire, provide explanations/ hypotheses relating to
statistical analysis.
Conclusion
The activities of individual graduates or county groups are
evolving and continuing so that documentation is difficult. One
success of the program is evidenced by the fact that alumni are
"doing their own thing" which means that program coordinators
often don't hear about activities and success stories. The above
methods are intended to provide an overall picture of the dynamic
nature of the program and resulting community impact. A
major success of the Community Leadership: A County Perspective
program is evidenced by the continuing leadership efforts in each
county. Through their involvement in the CL program,
participants have further developed their basic community
leadership skills, become better informed on local issues, gained
a broader perspective of their counties, and taken an active role
in improving their communities. References
Beaulieu, L. J. (1990, January 16-18). Building partnerships for
people: Addressing the rural South's human capital needs.
Proceedings of a Regional Conference Revitalizing the Rural
South, Birmingham, AL.
Berry, G. (1986). The secret of economic success in local
leadership. Georgia Trend, 1(5).
Boatman, S. A. (1989, August). Potential and impact: Assessment
and validation in leadership development. Proceedings of the
Leadership Development Seminar: Developing Human Capital
through Extension Leadership Programs, Manhattan, KS.
Fear, F. A., Vandenberg, L., Thullen, M., & Williams, B. (1985,
July 30). "Toward a Literature Based Framework for Community
Leadership Development". Paper presented at Annual Meeting of
the Community Development Society, Logan, Utah.
National Extension Task Force on Community Leadership. (1986).
Community Leadership Development: Implications for Extension,
University Park, PA: Northeast Regional Center for Rural
Development.
Pigg, K. E. (1990). Accountability in community development.
Journal of the Community Development Society, 21(1), 19-32.
Soileau, R. R. (1990, January 16-18). Social and economic
impacts of rural revitalization. Proceedings of a Regional
Conference Revitalizing the Rural South, Birmingham, AL. Tait,
J.
L.Schaffert, D. R., & Augustinack, A. (1991,
September).Leadership program impact on participants
organizational behavior. Proceedings of the Association of
Leadership Educators Annual Conference, Denver, CO.
Warren, R. (1972). The community in America. Chicago, IL: Rand
McNally Publishing Company.