People & Programs in the News

Diane Cooper, Ph.D.

Cooper selected as member of the American College Personnel Association Diamond Honoree Class of 2008

Diane L. Cooper, a professor in the College of Education’s department of counseling and human development services, has received a prestigious national award recognizing her scholarly achievements and leadership in the field of college student affairs.

Cooper Elected to Executive Council of American College Personnel Association

Cooper Receives the 2007 Senior Scholar Award from American College of Personnel Association

COLLEGE STUDENT AFFAIRS JOURNAL, VOL. 20, No. 2, SPRING 2001

Published Research about Supervision in Student Affairs:
A Review of the Literature 1969-1999

Diane L. Cooper
Sue A. Saunders
Martin T. Howell
Jeff M. Bates
The University of Georgia

The purpose of this study was to analyze supervision-related literature over the past 30 years to identify empirical models available to help student affairs professionals improve their supervisory skills.  Articles in the NASPA Journal and Journal of College Student Development were reviewed and divided into three categories - general supervision issues, staff development, and supervising student paraprofessional staff.  A vast majority of the supervision-related literature does not contain a rigorous research methodology or a validated research Instrument which to develop an empirically-based model of supervision.  If it cannot be verified that expensive and time-intensive supervision strategies do not make a difference in employee performance, it may become more difficult to continue 'doing business as usual.' The results indicated that few gains have been made in the last 30 years.

Most student affairs practitioners spend a significant amount of their time in activities that fall under the rubric of supervision.  These activities involve selecting staff, orienting new staff, one-on-one time with subordinates, performance appraisal, and staff development.  Unfortunately, the process of supervision receives very little attention in the student affairs literature.  Professional conferences may offer a few programs dealing with specific supervisory techniques, and preparation programs may offer course work about administration or management that has a cursory discussion of supervision.  The research that does exist is often available by searching such topics as personnel (Ginsburg, 1997) or Total Quality Management (Holmes, 1996; Rudolph & Howard, 1995).  Even with cursory attention in the literature and limited formal training about supervision, most student affairs practitioners spend a substantial part of each day managing their professional, support, and paraprofessional staff members (Dalton, 1996).

Even though personnel management occupies a substantial proportion of the time of student affairs administrators, the perception of supervision is frequently narrow and remedial.  Winston and Creamer (1997) concluded that supervision is often seen as important only when working with employees who have problems or who are new to the organization.  "This view holds that staff do not need supervision if they are well qualified for the demands of the position, understand their responsibilities and the workings of the institution, and fulfill expectations and meet deadlines.  To provide someone who is doing a good job with supervision could be interpreted as an insult or putdown, or at best a waste of valuable time" (p. 180). 

A recent study by Saunders, Cooper, Winston, and Chernow (2000) indicated that the majority of those being supervised perceived themselves has having relatively few formal supervisory sessions (fewer than one per month).  In addition, results of this study and a previous one by Winston and Creamer (1997) showed that the topics discussed in supervisory sessions emphasized day-to-day operations issues, rather than long-term professional development, achievement of institutional goals, or comprehensive discussions of employee strengths and weaknesses.  "It appears that supervision is seen simply as an instrument to get the job done not as an important way to foster the betterment of staff A (Saunders et al., 2000, p. 8).

There are a few indications in the business management literature that a comprehensive conception of supervision is more effective because it focuses on the employee's development and the congruence between the goals of employee and institutional aims.  In one study published in an organizational behavior journal, for example, researchers found that matching the employee's personal needs and values with organizational goals promotes intrinsic motivation and job satisfaction (Cropanzano, James, & Citera, 1992).  Even though there are few empirical studies about supervision in the literature of higher education and student affairs, there exist some indications that supervision will become an increasingly important area of future inquiry because of the need to retain qualified professionals in the field.  For example, Woodard and Komives (1990) reported that "workers of the future, particularly those in human development fields, will have high expectations for a caring, nurturing supervisor.  Workers are concerned about their quality of life, their personal and professional well-being, and their holistic development, including career and work" (p. 232).

Part of the difficulty in creating a more intentional and intensive approach to supervision is the lack of models for effective supervision (Winston & Creamer, 1997).  The first step in building a workable model is to examine the effectiveness of current supervisory practices.  The purpose of this study was to review the literature commonly read by practitioners in student affairs to ascertain the extent, content, and type of research available to guide improvement of supervision.  Articles were reviewed regarding general supervision issues, staff development, and concerns specific to the supervision of student paraprofessional staff.

Method

The sample included all of the articles published since 1969 in the NASPA Journal and the Journal of College Student Development (excluding editorials and book reviews). Initial plans for data analysis were to conduct a meta-analysis of the articles to summarize the research related to supervision of professional and paraprofessional staff as well as staff development.  However, it became apparent that the vast majority of the articles published on these topics did not report statistics from which an effect size could be calculated.  Because so little information is available in the literature regarding these topics, the researchers decided to continue the review with a modified approach to content analysis.  First, two independent reviews of the journals took place during 1999 to insure that all possible articles on the topics of student affairs supervision, staff development, and supervision of paraprofessional staff would be included.  At this point, two of the researchers reviewed each article to classify: (1) topic area covered, (2) functional area involved, (3) types of staff (professional or support) included in the study, (4) treatment, and where applicable, and (5) instrument or measurement utilized.  These five criteria were selected for review based on Winston and Creamer's (1997) concern that, "if there were a direct relationship between the level of concern for a particular practice in student affairs and the amount of literature on the subject published in the field's journals, it would be clear and convincing evidence that supervision is a neglected domain" (p 183).  When discrepancies occurred in the coding, a third researcher reviewed the material and decided the coding to be used.  This third-party review occurred in only four instances.

Results

Between 1969 and 1999, the NASPA Journal and the Journal of College Student Development together published 25 articles that related to general supervision and personnel management, 50 about staff development, and 70 about supervising paraprofessional staff.  Forty-two of these 145 supervision-related articles were based on controlled research studies and are examined further below.  The remaining 103 articles were primarily a reporting of programs in place or suggestions about implementing various techniques but without any evaluation of program outcomes.  Specific information from the supervision-related categories is discussed separately below.

General Supervision and Management Studies

Of the 25 articles related to supervision and personnel management, only 10 (40%) actually assessed the outcomes of supervision or management practices and thus could be classified by topic, functional area, types of staff, treatment, and instrument or measurement utilized.  The remaining 15 discussed various aspects of supervision and management, but did not focus on outcomes and are therefore not included in the analysis.
The 10 articles that did assess program or training outcomes can be categorized as follows: two surveys of management techniques, two surveys of performance appraisal techniques, two surveys of staffing patterns, a survey of employment (hiring) practices, a study of the relationship between job attitudes and work environments, a study of the relationship between management techniques and work environment, and a qualitative study concerning professional loyalty.
Chief student affairs officers were the most frequently studied group: 5 (50%) of the 10 relevant articles studied the supervisory or management practices of CSAO'S.  Of the remaining five, two studied student affairs professional staff, one studied general residence life staff, one studied residence life paraprofessional staff, and one studied counseling centers.
In only 1 out of 10 instances did researchers use a published assessment instrument.  In one other case the authors used a previously developed (but apparently unpublished) survey; one article used qualitative interviews and no survey; and in seven instances, the researchers developed their own questionnaires or surveys.  The use of rigorous research methodology with validated instruments was conspicuously absent.

Studies Exploring Staff Development

A majority of the studies published in the area of professional development focus on practical implications and applications for student affairs professionals.  Of the 50 studies included in the review, an overwhelming majority (86%) focused on a non-empirical application or practical implication as a primary focus, as compared to the few empirically based studies (4%), or theoretically-derived articles (10%).  Thus, many of the studies included in the analysis relied on summaries of programs currently in place or discussed the implications of contemporary trends in the college student affairs. 

It is not surprising that only a few studies used validated instruments in their assessment (12%), while a majority either used no instrument (50%) or a non-validated questionnaire designed specifically for the purposes of their study.  The end result was that only seven of these articles met the 5-point criteria from Winston and Creamer (1997).

There is a trend in the research in the area of professional development to focus heavily on staff development and staff training (see Table 1).  Twenty-six of the 50 articles (52%) included in this study focused on one of these topics.  Further, 10 (20%) of the remaining articles emphasized general implications of professional development and professional preparation for the college student affairs profession.  Though there were some specific topic areas addressed in this literature base (such as diversity training or organizational commitment), the trend is to address general concerns about professional and staff development issues in a manner that provides tangible implications and interventions that can be applied to various populations (see Table 2).

In terms of the populations that were addressed in the area of professional development, or the functional area covered in these articles, there is a focus on implications for a general student affairs audience.  Twenty-one articles (42%) focused on implications relevant to a generalist student affairs professional audience, while an additional eight (16%) focused on senior student affairs practitioners.  Though some attention was given to specific student populations or functional areas within college student affairs (see Table 3), the majority of articles focused on student affairs professionals as a whole as opposed to specific areas within the college student affairs profession.

There is a trend in the professional development literature to discuss interventions or applications that have been judged to be beneficial or effective by the researchers but have not been empirically evaluated.  Thus, the majority of research about staff development in student affairs is simply descriptive of programs or trends, as opposed to research that empirically evaluates the efficacy of various approaches.

Table 1
Focus of articles regarding staff development

Student Paraprofessional Staff Studies

Of the 70 articles related to supervision and training of student paraprofessional staff, only 25 were studies that addressed the five classification elements mentioned above.  Therefore, only these 25 were analyzed for content and results.  The remaining 45 articles discussed aspects of the five elements reviewed.

The 25 articles reviewed covered helping skills training (48'7o), communication skills (16%), assertiveness training (12%), and general job preparation (12%).  More than 88% of the training/ supervision discussed was housed in a residence life functional area.  The time spent on training ranged from a 3-hour workshop to semester-long courses.  All of the training/ supervision took place as part of a multi-session program.  The paraprofessionals included were resident assistants (80%), orientation leaders (12%), or student activities leaders (8%).

Table 2
Populations addressed in articles regarding staff development

As with the other two content areas reviewed, there was little evidence that these articles included any type of rigorous research methodology to reach conclusions about effectiveness.  A few exceptions included the use of assessment instruments, but the majority only used participant evaluations (32%) or judge's ratings (such as Carkhuff scaling [see Carkhuff, 1969]) (20%) as the source for evaluating the initiative.

Discussion
 
Certainly, student affairs practice has benefitted from opinions about the best approaches to supervision and professional development.  The discussion of strategies that seem to be effective as well as compiling general trends about supervisory or staff development practices (such as frequency of supervision meetings, topics discussed in institutional professional development workshops) are useful first steps to completing rigorous research that provides evidence of what supervision and professional development strategies are most effective.  The results of this study, however, indicate that the student affairs profession has not progressed far beyond the initial step.

Very few of the articles reviewed for this study employed rigorous research methodologies.  The vast majority conducted participant evaluations which undoubtedly amounted to little more than satisfaction surveys.  As noted before, satisfaction surveys do not measure the changes in knowledge, skills, or attitudes that one would hope to accomplish through training sessions.  At best, satisfaction surveys measure the entertainment value of the session, or perhaps the participants' perception of whether the session was "time well spent."

Both quantitative and qualitative methods are appropriate for measuring the changes that take place as a result of training sessions or supervision practices.  Which one the researcher chooses will depend upon the research questions and purpose.  In general, quantitative research is more appropriate for providing information that allows one to empirically measure and then predict changes based upon the intervention, while qualitative research is more appropriate for gaining a fuller understanding of the training experience.  Both methodologies can be employed with rigor.

Table 3
Articles Discussing the Aspects of Training
Student Paraprofessional Staff 1969-1999

When using quantitative methods, one would expect, at a minimum, at least three conditions.  The first is the use of valid and reliable assessment instruments.  This involves the testing and retesting of instruments to ensure that it measures what it attempts to measure and that it does so repeatedly.  As noted earlier, dozens of instruments are available to those attempting to measure the outcomes of their training exercises.  Second, the research design should include pre-test, post-test within group comparisons or between group comparisons using multiple treatments and/or control groups.  Since most training programs serve only a small number of participants, it would be difficult to rely on randomization to control for pre-training levels.  Finally, quantitative studies should report an effect size in order to give other researchers, who may employ different research designs, a basis for comparison.  The effect size also gives an idea of the "practical" significance of the study.

Only five studies came close to meeting these criteria by employing rigorous quantitative methodologies: Newton (1974); Layne, Layne, and Schoch (1977); Schilling (1977); Scroggins and Ivey (1978); and Terrell and Gordon (1978).  Four used tested research instruments (the fifth used Carkhuff Scaling evaluation method), and all used pre-test, post-test designs.  Interestingly, however, none of the five reported an effect size.

Qualitative research, while under-utilized as a methodology, is equally valid as a way of assessing learning outcomes.  Like their quantitative counterparts, qualitative studies should meet certain standards.  First, the researcher should utilize focused research questions.  Second, the researcher should employ meticulous data collection methods, which generally means rigid interview protocols.  Third, data analysis methods should ensure a reasonable degree of reliability (a term that is somewhat different than commonly used in the quantitative sense) by employing triangulation, keeping an audit trail, and using peer examination.  Finally, the study's findings should be written clearly and provide sufficient supporting detail.

Only one study reviewed here used rigorous qualitative methods.  Guido-DeBrito, Chavez, Wallace, and DiBrito (1997) used a constructivist perspective to study the nature of loyalty in student affairs settings.  The researchers interviewed 4 university presidents, I executive vice-president, 4 senior student affairs officers, and 41 student affairs department heads and generated thick description which allowed them to identify several themes and patterns around loyalty.

It was interesting to note that only one article reviewed for this study used a qualitative methodology.  Qualitative approaches may have particular usefulness for topics such as supervision and professional development.  The conceptually broad, sometimes idiosyncratic nature of these topics makes the typically-used quantitative approach all the more difficult.

While the results from the review of the literature are disappointing, they also indicate a recommendation for student affairs professionals to acquaint themselves with research methods in the profession.  Most preparation programs include at least some level of study in this area.  Those engaging in research should learn sound research methodologies and research design techniques, preferably through formal coursework.  Editors and reviewers of professional journals, for their part, should demand the use of rigorous research methods.

One of the interesting findings of this study came in the paucity of validated instrumentation that had been used in those studies included in our analysis.  A few studies noted previously did include well-validated instruments, but most focused on a narrative interpretation of an intervention, survey findings, or an instrument designed "for the purposes of this study." These non-validated approaches to assessing outcome lack the convincing nature of using a psychometrically validated instrument to assess outcome.  The lack of a use of validated instruments may be the result of many factors, such as non-awareness of a specific instrument or the lack of an instrument to assess a specific aspect of a program; nevertheless, this paucity must be addressed.  For instance, even in those areas where validated instruments have not been published, researchers who use instruments designed for their study can assess the validity of their instruments by providing factor analysis findings and alpha levels for their instrument.  Further, research questions can be formulated in a manner that allows psychometrically validated instruments to be used in assessing outcome.

Though well-validated instruments have not been designed in every topic area that will be addressed in the literature, incorporating these instruments into the college student affairs literature base will provide more convincing evidence for the benefits of specific program implementation.  As external accrediting groups increasingly call for evidence to support interventions and programs, providing results through validated means (as opposed to satisfaction survey findings) gives the field a solid foundation from which to base their future interventions.  Further, as more attention is given to these validated approaches, the field will begin to strengthen the 'scientist' aspect of the scientist-practitioner model of practice.

Implications

The results of this study confirm the conclusions of Winston and Creamer (1997).  These authors point out that "since the mid-1970'sYconsiderable literature has been published about staff development in student affairs Y most are not researched based, though some are, and the results typically show mixed reactions to felt need and actual practice.  Rarely do authors offer new ideas, particularly conceptual ideas, but there are exceptions to this generalization and their contributions to the literature are especially useful" (p. 224).

Part of the difficulty of completing rigorous research about supervision and professional development is the challenge of obtaining outcome measures other than perceptions or satisfaction.  Even though there are clear indications that perceptions of positive supervision and professional development practices are associated with staff members' productivity, longevity in the profession, and general organizational effectiveness (Winston & Creamer, 1997; Woodard & Komives, 1990), these associations have not yet been empirically demonstrated.

The rigorous assessment of staff members' improved competence or productivity as a result of certain supervisory or professional development practices requires time and evaluation strategies that are not contaminated by bias.  There are also potential or perceived risks to staff members and supervisors who participate in a rigorous study of the efficacy of staff supervision and professional development.  Even if confidentiality was assured, staff members might fear openly revealing their lack of skill, and supervisors might be threatened by an in-depth analysis of supervision or staff development practices. 

Even though the assessment of outcomes that go beyond perceptions and satisfaction is hard work, it is necessary to be able to justify resources allocated to professional development and staff supervision.  If it cannot be verified that expensive professional development activities and time-intensive supervision strategies make a difference in terms of institutional priorities, such as productivity, competence, retention of students and staff, it may become difficult to continue "doing business as usual."

Another of the trends revealed in the research analyzed for this study was the lack of empirical methodology using comparison groups.  Frequently, only one institution or one functional area at one institution was the focus of the investigation.  While it is indeed difficult to collect data from more than one university, failure to include a broadly based sample severely limits generalizability.  Because supervision and professional development practices are often reflective of an institution's unique culture, any findings that emerge could be extremely difficult to replicate at a university with a different type of organizational culture.

Another trend that emerged from this study was the conspicuous lack of the use of validated instruments within the quantitative studies.  Most studies relied on author-constructed instruments with no attention to validity and reliability concerns.  Without the use of a valid and reliable instrument, practitioners would be ill-advised to implement a strategy shown to be effective in a study, since the positive results described could simply be an artifact of the measurement.

The cost to student affairs of "business as usual" is really an opportunity cost.  If supervision is linked to productivity, then failure to improve supervision will result in failure to enhance productivity.  When institutions are increasingly being asked to improve productivity, student affairs needs to demonstrate how they are contributing to the institutional mission.  In addition, good supervision leads to greater employee satisfaction that could conceivably result in less turnover.  However, until we have adequately researched supervision, we will not really understand the elements that comprise effective performance.

Student affairs researchers and practitioners alike need to be more actively involved in measuring the outcomes that intentional programs and specialized training have on improving the supervision of staff.  We need to assess the effect of staff development programs more rigorously to see if the time and money spent are producing intended results.  Finally, training we provide our paraprofessional staff must also be put to more stringent evaluative standards.  Only through more structured research programs, that include the use of valid and reliable instrumentation, as well as more structured methodologies, can we truly provide others with useful applications to practice.

References

Carkhuff, R. (1969).  Helping and human relations.  New York: Holt, Rinehart, & Winston.

Cropanzano, R., James, K., & Citera, M. (1992).  A goal hierarchy model of personality motivation and leadership.  Research in Organizational Behavior, 15, 267-322.

Dalton, J. C. (1996).  Managing human resources.  In S. R. Komives, D. B. Woodard, Jr. & Associates (Eds.), Student services: A handbookfor the profession (3rd ed., pp. 494-511).  San Francisco: Jossey-Bass.

Ginsburg, M. (1997).  Organization, budgeting, and staffing.  In L. H. Dietz & V. L. Triponey (Eds.), Serving students at metropolitan universities: The unique opportunities and challenges. (New Directions for Student Services, No. 79, pp. 2737).  San Francisco: Jossey-Bass.

Guido-DiBrito, F., Chavez, A. F., Wallace, J. A., & DiBrito, W. F. (1997).  Loyalty between senior student affairs officers and their supervisors and staff members.  NASPA Journal, 38, 244-254.

Holmes, T. A. (1996).  TQM: Finding a place in student affairs.  In W. A. Bryan (Ed.), Total quality management:: Applying its principles to student affairs. (New Directions for Student Services, No. 76, pp. 33-43).  San Francisco: Jossey-Bass.

Layne, R. G., Layne, B. H., & Schoch, E. W. (1977).  Group assertiveness training for resident assistants.  Journal of College Student Personnel, 18, 393-398.

Newton, F. B. (1974).  The effect of systematic communications skill training on residence hall paraprofessionals.  Journal of College Student Personnel, 15, 366-369.

Rudolph, J. T., & Howard, N. L. (1995).  Implementing TQM at Oregon State University: Moving continuous quality improvement practices into higher education.  CUPA Journal, 46, 25-31.

Saunders, S. A., Cooper, D. L., Winston, R. B., Jr., & Chernow, E. (2000).  Supervising staff in student affairs: Exploration of the synergistic approach. Journal of College Student Development, 41(2),181-192.

Schilling, K. L. (1977).  Impact of a training course on personal development of resident assistants.  NASPA journal, 14(4), 33-37.

Scroggins, W. F., & Ivey, A. E. (1978).  Teaching and maintaining microcounseling skills with a residence hall staff.  Journal of College Student Personnel, 19, 158-162.

Terrell, T. C., & Gordon, K. (1978).  Microtraining in paraprofessional orientation training.  Journal of College Student Personnel, 19, 190.

Winston, R. B., Jr., & Creamer, D. G., (1997).  Improving staffing practices in student affairs.  San Francisco: Jossey-Bass.

Woodard, D. B., Jr., & Komives, S. R. (1990).  Ensuring staff competence.  In M. J. Barr & M. L. Upcraft (Eds.), New futures for student affairs (pp. 217-238).  San Francisco: Jossey-Bass.

 

Counseling & Human Development Services • 402 Aderhold • The University of Georgia • Athens, GA 30602
Phone: (706) 542-1812 • Fax: (706) 542-4130 • chds@uga.edu • Web Feedback

About the COE | Academic Programs | Admissions | Departments & Directories | News & Events | Research & External Affairs | Resources & Services
COE Home | UGA Home | Sitemap | Contact | UGAMail | Give to the COE
TEXT-ONLY VERSION