See also: Advising Administration Index

“The Academic Advising Program must conduct regular assessment and evaluations” (Council for the Advancement of Standards in Higher Education, 2005).

Richard Light (2001) noted that good advising may be the single most underestimated characteristic of a successful college experience” (p. 81). But how does a campus community know how their advisors are doing? In this era when “assessment” is at the forefront of the higher education community, attempts to develop campus-appropriate instruments to evaluate advising quality are strongly encouraged. A variety of factors affect the development of such an instrument and include such things as: current campus advising model and method of evaluation; level of constituent support for the development of a new instrument; diversity and overall demographics of the campus; and methods available for distribution/collection of the instrument. These factors lead to a bevy of questions, including “Who is responsible for the development of the instrument?”, “What questions should be asked?”, “What format should be used?”, and “How will data be collected and analyzed?”

Faculty and staff at North Central College found that we were dealing with many of these same issues when we determined that the current advisor evaluation was antiquated, ineffective, and not meeting the needs of the college.  The task of writing a new, original, online advisor evaluation was charged to the staff of the Advising Center.  After two years and multiple meetings, drafts, feedback sessions, and focus groups, the new evaluation was introduced during priority registration in spring, 2006. The following article details the more pertinent stages of the process, along with five specific recommendations that might be helpful to advisors and administrators wishing to undertake a similar challenge on their respective campuses.

North Central College is an independent, comprehensive college of liberal arts and sciences with approximately 2200 undergraduate students. There is a faculty-based advising system with a satellite Advising Center. Over a period of years, there was growing discontent with the advisor evaluation form that had been in place for decades. As part of a more general review of the advising system, a faculty committee issued a report in 2004 suggesting that a new advisor evaluation be developed. This recommendation was supported by the Dean of Faculty, Registrar, and faculty governance. Without the support of these appropriate campus power brokers, the development of a new evaluation would not have taken place.

Recommendation #1: Become familiar with specific campus processes and engage appropriate campus power brokers in a discussion of the need for a new advisor evaluation. Additionally, include an individual widely respected for his or her academic advising as a major player on the development team.

In early 2005,Advising Center staff began work on the new evaluation. Initially, a broad-based survey of information was conducted, using a variety of sources, but primarily using the NACADA Clearinghouse of Academic Advising Resources. This survey of information referenced content, style, delivery methods, and anything else deemed relevant with regards to existing advisor evaluations. As summarized by Cuseo (2002), research has repeatedly demonstrated that students value advisors who are 1) available/accessible; 2) knowledgeable/helpful; 3) approachable; and 4) able to serve as a mentor. As this information was conveyed back to the faculty committee providing oversight to the development of the new advisor evaluation, it was accepted that the content of the College’s new advisor evaluation should address these same four areas. Additionally, commercially available advisor evaluations, such as the Academic Advising Inventory (Winston and Sandor, 1984) and ACT’s Survey of Academic Advising, were offered as options. While considered, all parties were in agreement that the new evaluation should be something original that reflected the unique nature of North Central College but still addressed the four core areas mentioned.

Recommendation #2: Review the literature and determine what is already available while maintaining primary focus on specific campus needs and culture when developing the content of the evaluation. Use knowledge and expertise as advising professionals to inform decisions.

Upon sharing these findings with the faculty committee and as development of the evaluation began, several key logistical points needed to be addressed and agreed upon by the staff of the Center and the faculty committee. First, the length of the evaluation was important – consensus was that it not take more than 5-8 minutes to complete. Second, there should be both quantitative and qualitative portions. Third, the instrument should be used not just for evaluative purposes, but also to educate students and faculty advisors about the kinds of topics that could be discussed as part of the advising relationship. The specific wording used in the instrument, particularly in Part I, reflects this concern. Fourth, the intent was to place the evaluation online and tie its completion to students’ priority registration during the spring term although there needed to be an “opt-out” option for students who wished not to complete the evaluation. Finally, the evaluation should be anonymous in that individual student responses would not be conveyed to the advisor, but rather an overall summary of the results would be shared. Over the course of three months, six drafts were presented to the faculty committee before the final version was eventually accepted and approved by 89% of the full faculty.

Recommendation #3: Keep others informed and solicit input from appropriate parties as much as possible throughout the process.

Information Technology Services (ITS) was a major player in this process. Advising Center staff maintained regular contact with ITS and continually updated them on the status of the evaluation discussion. The plan all along was to have the evaluation completed online, anonymously, by students, yet have enough data collected “behind the scenes” to forward results to specific advisors as well as generate campus-wide advising reports. It is beyond the scope of this article to delve into the technical details of the process, but, needless to say, this project was a major undertaking by ITS staff and required a significant commitment from that particular office. After full faculty approval of the new evaluation, it was turned over to ITS in November, 2005, with a request to have it online by April, 2006, in time for priority registration. ITS developed several different versions (consisting of mostly cosmetic differences) before completing the final version in February, 2006. This version was then presented to a student focus group for a test run. After a few minor changes, the evaluation was complete and ready to be implemented. [Click to view North Central College’s Advisor Evaluation.]

Recommendation #4: Become friends with ITS and maintain regular contact with them throughout the process. Set hard deadlines for various stages of the process.

The evaluation was placed online in time for priority registration in the spring. Students accessed the evaluation by logging in to their campus computer accounts. In order to achieve as high a response rate as possible, and with the support of all levels of administration, a hold was placed on students’ accounts that prevented them from registering unless they had completed the advisor evaluation or elected to opt out. Thus, students were not required to complete the evaluation, but had to at least visit the evaluation and consciously elect not to complete it (only 6% of the students opted out). Upon either submitting the evaluation or opting out, the hold was automatically released and the student was free to register. Alternative measures for capturing graduating seniors, who were not registering for the upcoming academic year and thus not impacted by the registration hold, were only partially successful and need to be revisited.

During the several week time span when students were completing the evaluation and registering, regular contact was maintained with ITS to determine if any problems had developed or if any changes were needed. Very few issues arose and all parties were quite satisfied with how the initial roll-out of the new evaluation took place. However, the one area that did not get enough attention during the development stage was the reporting of data. ITS collected and maintained the data in a very raw form, but little attention was paid to converting that data into user-friendly results until considerably later in the process. Over a period of several months during the following academic year, ITS and Advising Center staff developed an appropriate format for the results to be distributed to each advisor; ideally, this task would have been a more significant part of the evaluation development.

Recommendation #5: Consider the data reporting format early in the development process.

In conclusion, all parties involved in the process are pleased with the new evaluation, which is a vast improvement of the paper evaluation previously used. However, by no means is it considered an unchangeable document; feedback will routinely be gathered and evaluated to determine if revisions are necessary. Hopefully, the recommendations included here and summarized below will prove to be beneficial to other institutions and expedite the development process should a similar task be undertaken. Please see the resources listed in the annotated bibliography for opportunities to read more about advisor assessment and evaluation.

Summary of recommendations for developing an original advising evaluation instrument:

1.   Know campus processes and keep power brokers informed.
2.   Consider existing instruments but remain cognizant of the campus’s culture.
3.   Solicit input throughout development.
4.   Be friends with ITS, but set deadlines.
5.   Include discussion of the format for reporting data early in the process.

Authored by:
Scott Gabbert and Mary Jean Lynch
North Central College

References

Council for the Advancement of Standards (CAS). (2005). Academic Advising:CAS Standards and Guidelines. Retrieved from http://www.cas.edu/getpdf.cfm?PDF=E864D2C4-D655-8F74-2E647CDECD29B7D0
 
Cuseo, J. (2002, March 19). Advisor evaluation. Policy Center on the First-Year of College.
 
Light, R. (2001). Making the most of college: Students speak their minds. Cambridge,MA:Harvard University Press.
 
Winston, R. B., and Sandor, J. A. (1984). The Academic Advising Inventory. Athens,GA: Student Development Associates.

Annotated Bibliography

Astin, A.W. (2002 ). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. Westport,CT: American Council of Education and The Oryx Press.

• Detailed overview of all aspects of assessment in higher education. Not specific to academic advising but provides a good overview.

 
Banta, T. R., Jansen, M. J., Black, K. E., & Jackson, J. E. (2002). Assessing Advising Outcomes. NACADA Journal, 22 (1), 5-14.

•   Specific guidelines for planning, implementing, and improving advising assessment. Does include some points regarding the evaluation of individual advisors.

Gordon, V. N., & Habley, W. R. (2000). Academic advising: A comprehensive handbook. San Francisco: Jossey-Bass

• Chapters 23 and 24 specifically deal with assessing the effectiveness of advising programs and individual advisors.

Habley, W. R. (ed.). (2004). The status of academic advising: Findings from the ACT sixth national survey. (NACADA Monograph Series No. 10). Manhattan, KS: NACADA.

• Chapter 5: Goal Achievement and Advisor Effectiveness. Provides statistics on national norms regarding advisor effectiveness based on type of institution (four-year institutions, two-year institutions, etc.).

Higa, L. & Kirk-Kuwaye, M. (2002, September). Assessment of advising: A call for wider participation. The Academic Advising News, 25 (3). Retrieved from the NACADA Clearinghouse of Academic Advising Resources Web site.  

• Provides an overview of why assessment is important and ways to get started thinking about it.

Knapper, C., & Cranton, P. (Eds.). (2001). Fresh approaches to the evaluation of teaching. New Directions for Teaching and Learning, No. 88. San Francisco: Jossey-Bass.

• Discusses the evaluation of teaching, but some points carry over to advising, particularly when one considers the 'Advising as Teaching' theme.

Kramer, H. C. (1982). Evaluation of academic advisors: Administrator and faculty perspectives. NACACA Journal, 2 (1): 30-36.

• Historical overview of the importance of evaluating advisors from the perspectives of administration and faculty.

Legutko, R. S. (2006). Students grade their professors: An evaluation of a college's faculty advising by its graduating seniors. The Mentor: An Academic Advising Journal, (8) 4. Retrieved from http://www.psu.edu/dus/mentor/060901rl.htm

• Discussion of research specifically designed to measure student satisfaction with advising. Good background.

McGillin, V. A. (2003). Research versus assessment: What’s the difference? Academic Advising Today (26) 4.

•   The similarities and differences between research and assessment are discussed including goals, methods, results, and audience. Beneficial to determine what is needed before embarking on a large scale project such as the development of an evaluation instrument.

Multari, R. J. (2004). Integrating technology into advisement services. The Mentor : An Academic Advising Journal, (6) 2. Retrieved January 24, 2007, from http://www.psu.edu/dus/mentor/040526rm.htm

• Supports the use of technology in general in advising. Provides areas to consider should a technological upgrade or change (such as developing an online advisor evaluation) be implemented on a campus.

Cite this resource using APA style as:

Gabbert, S. & Lynch, M.J. (2007).  Developing a New Evaluation Tool for Advisors: From Conception to Implementation. Retrieved from NACADA Clearinghouse of Academic Advising Resources Web Site: http://www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/Evaluation-of-Academic-Advisors.aspx