Victoria A. McGillin, Advising Assessment Commission Member; NACADA Research Committee Past Chair
One frequent question heard from NACADA members is, "What's the difference between research and assessment?" The following is an effort to articulate both the overlap, and the distinctions, between these two.
In our workshop on advising research and grant proposal development, the NACADA Research Committee discusses the similarities and differences between research and assessment. The following is a synopsis.
Goals
The goals of experimental research and program assessment differ significantly. While research focuses on the creation of new knowledge, testing an experimental hypothesis, or documenting new knowledge, assessment and evaluation focus on program accountability, program management, or decision-making and budgeting.
That is, while research is designed to document or measure a phenomenon not formerly recorded, e.g. applying a new theory to an advising encounter and documenting how well a model 'explains' what is going on between advisor and advisee, program assessment provides information to your campus about whether you are achieving prescribed goals, expending resources wisely or meeting a documented campus need.
Methods
While the methods employed in good program assessment and evaluation may be similar to those used in good research, they need not be. The range of methods employed in both may range from subjective field observations through objective questionnaires. If your key assessment question is how your campus advising compares to national data on advising (such as the ACT survey), the use of a nationally-standardized, reliable and valid instrument would be crucial to answering that question. However, nationally-standardized instruments may not always 'fit' your campus as they may utilize differently-named services or institutional structures not present. When an existing measure just won't do, good research AND good assessment practices call for the development of a reliable and valid new measure. We must be wary of developing a 'quick and dirty' measure in an effort to just 'get a quick answer' to our questions, without taking the time to ensure our measures are reliable or valid for our own campuses.
One major methodological difference between research and assessment is that researchers will 'experimentally manipulate a variable' (for example, randomly assigning students to one model of orientation or another), while program evaluation tends to be non-random (we rarely have the luxury of such random manipulation of our students). At best, assessment just looks at 'natural' differences that emerge, such as comparing students who chose one orientation event over another).
Results
Just as experimental research and program assessment differ in their goals, they also differ in the use of their results. Research results are expected to be generalizable beyond one's own campus, with implications for similar institutions or similar populations. Program assessment results are applicable only to one's own campus. While both are of great value, research should contribute new knowledge to the field. When opening the NACADA Journal, you expect documentation of research that began as an advising question and culminated with statistically significant research of an advising method, theory or programmatic intervention that you can apply with some assurances of success.
Conversely, program assessments are designed to be site-specific and crucial for campus decision-making. Good program assessment ensures that you are responsive to the changing (or unchanging) needs of your populations. You may want to reuse the same measure each year to document the high level of program success over time. Your results may be particularly appropriate for the NACADA Journal's Tool Box section that highlights examples of best advising practices that link to current research in the field.
Finally, while research should provide possible answers to identified questions, it should also generate new research questions from the results. For example, if one's data showed that both male and female students were more critical of male advisors than female advisors, the researcher would want to explore this research question further. Assessment, however, looks for answers. Viewed from an assessment standpoint, such results might lead to interventions, such as additional training for male advisors and the desire to assess the effectiveness of that intervention on one's campus.
Audience
Given these differences, it is not surprising that there are vastly different audiences intended for program assessments, as compared to research. Assessment results are targeted for the key decision-makers on your campus. When budgets are cut, new programs proposed or accreditation rolls around, assessment/evaluation reports help you make a case for your program. As I am fond of saying, 'Whoever gets to the table with numbers first, wins.' The ability to produce an executive summary of key assessment findings (no more than 2 pages) documents the effectiveness of your work and moves your programs to the top of the funding lists, ahead of those supported only by anecdotal information.
In contrast, research is intended for the field of advising and higher education as a whole. Your results will be read by many, debated and critiqued, copied and expanded upon to generate even newer knowledge. While a one-page executive summary submitted to your dean may get you funding for a new advising initiative, your colleagues outside your institution look for full documentation of the research that led you to this question, the literature review of the theory that guided your process, details on the methods you used, the results (strengths and weaknesses) of this study, and the conclusions you drew based upon your research. The 15-20 pages, with bibliography, necessary for a published article, would only gather dust if submitted as part of a funding request to most deans or VPs.
Connecting It All
Let me conclude by emphasizing the most crucial point of connection between assessment and research. Good assessment/evaluation can be expanded into good research. Good research should lead to even better assessment procedures. Good assessment makes use of the best conceptual and theoretical models and the best research measures or methods. With valid and reliable measures, campus-specific questions may have national implications. A phenomenon identified on your own campus may be the cutting edge for an issue of significant importance.
Finally, find significant resources on advising assessment on the Assessment of Advising Commission Web Page.
We urge you to consult with the NACADA Research Committee. They seek cutting-edge proposals. Your assessments may lead to a critical (and fundable) piece of research!
Victoria A. McGillin Dean Wheaton College
Ruth Darling, NACADA President
Dear Colleagues,
After every NACADA National Conference, I return to my campus with a true sense of belonging to a profession that has student learning and development as its core value. I am reassured that I associate with a diverse group of advising colleagues who approach their life's work with this point of view or perspective. I am also reassured that there is a professional association that has as its focus the promotion of academic advising within higher education along with the professional development of its members. These beliefs shape how I think about NACADA, my colleagues and my work.
The ideas of perspective/point of view and shared beliefs are an important piece of a graduate seminar I teach titled 'College Student Development Theory and Practice.' Throughout the course, the students and I explore the ideas and concepts underlying various paradigms and the impact these ideas have on theory, research and practice. To guide our discussions, I use Guba's (1990) definition of paradigm as - 'an interpretive framework, a basic set of beliefs that guides action' (p. 17). Together, through the examination of psycho-social/identity development, cognitive-structural and typology theories, we hope to arrive at a basic set of beliefs that will guide their personal theories and practice as entry- level professionals.
NACADA is developing a strategic plan that promotes a distinct mission and reflects a shared set of beliefs. The plan will be an 'interpretative framework' that 'guides action' as the leaders and membership address the work of the association and ultimately the work we all do in our educational contexts. Within higher education, NACADA's role is to:
During this next year, the leadership of the various NACADA Divisions (regions, commissions/interest groups and committees) as well as the Board of Directors will be calling on you, the membership, to be involved in shaping our work, in shaping our goals and, ultimately, in shaping our shared vision of NACADA. As President, I encourage you to get in involved with the association. The Executive Office staff, your regional chairs or the chairs of various commissions, interest groups and committees can serve as your contacts and information sources as you seek a meaningful way to connect with your colleagues and with NACADA. We need your involvement, your knowledge and your skill as we work to promote academic advising and support the learning and development of our students.
In closing, I hope to see many of you at the regional meetings this coming spring where we can, once again, connect with colleagues and explore our 'advising paradigm!' Thank you for the important work you do each day!
Best wishes,
Ruth Darling, President NACADA
Guba, E.G. (1990). The alternative paradigm dialog, In E.G. Guba (Ed.), The paradigm dialogue (pp. 17-30). Newbury Park, CA: Sage.
Roberta 'Bobbie' Flaherty, NACADA Executive Director
CERTIFICATION - the much used word within NACADA these days and the most often confused. As the Association strives to bring greater professional recognition to advisors, it is exploring a number of ways to recognize the knowledge and skills that advisors attain and utilize in providing effective academic advising.
Currently, there are four distinctly different initiatives being considered that in some way involve a form of the word certificate. They include a 'participation certificate', an 'advising certificate program', a 'graduate certificate program in advising', and 'advisor certification'. Now do we see why folks might be confused?
The participation certificate is simply a NACADA certificate given for completion of a specific NACADA professional development event such as the Academic Advising Summer Institute and/or the Advising Administrators' Institute. It simply denotes that a person participated in that event and hopefully that they gained some advising knowledge from that participation. That knowledge, however, is not assessed in any way.
An advising certificate program is being explored by the Professional Development Committee as an opportunity for members to obtain recognition for having participated in a series of professional development activities that would cover a broad spectrum of advising information. A 'certificate' might then be awarded to verify exposure to this broad spectrum of knowledge. Again, this knowledge would not be assessed or verified in any way.
The Graduate Certificate Program in Academic Advising is a totally independent Graduate Program offered by Kansas State University. Participants in this program earn academic credit and upon completion of the five courses, receive a Graduate Certificate from Kansas State University verifying completion of the program. Of course, each course includes knowledge assessment.
Advisor Certification is the subject of the work of a NACADA Task Force charged with exploring the potential for a 'professional' certification program and designation for academic advisors. This Task Force is identifying the knowledge and skills that effective academic advisors should possess and how this knowledge and these skills can be assessed to earn the designation as a 'Certified Advisor'. Any or all of the above programs might serve as leading toward the professional certification designation through the attainment and assessment of the knowledge and skills presented by each along with experience and other learning opportunities. The Executive Office is seeking estimates from 'certifying' entities as to the costs that would be incurred in assessing the advising competencies as identified by the Task Force and other expenses to anticipate in the administration of such a program. That information will be utilized to determine if such a program would be cost effective and viable for academic advisors.
I hope this helps everyone understand the many uses of the word 'certificate' and how your association is working to enhance the recognition you deserve for your continued education and your expertise. We welcome any suggestions for synonyms!
Roberta 'Bobbie' Flaherty NACADA Executive Director
A. Purpose and Overview The NACADA Journal, the journal of the National Academic Advising Association, seeks to enrich the knowledge, skills, and professional development of people concerned with academic advising and student success in higher education. Through its journal and other activities, NACADA is dedicated to the enhancement of student development by supporting the professional growth of academic advisors and the advising profession.
B. Qualifications Publications in peer-reviewed journals (preference for NACADA Journal publication) Experience in academic advising Interest and strength in quantitative and/or qualitative research methodology Active membership in NACADA Terminal degree (preferred)
C. Expectations Review manuscripts for significance, appropriateness, research design, analysis, and quality of writing within 30 days of receipt. Provide constructive feedback to authors in order to improve manuscripts. Ability to use Microsoft Word and email with attachments. Annual attendance at the Editorial Board meeting held during the NACADA National Conference each year. (Preferred)
D. Appointment Editorial Board members serve three-year terms that begin and end at the national conference. An Editorial Board member may serve non-consecutive terms.
E. Application Applicants familiar with the field of academic advising who are interested in seeking membership on the Editorial Board should submit an email message to [email protected] stating interest in and rationale for serving as a member of the NACADA Journal Editorial Board. A professional resume prepared in Microsoft Word '.doc' format should be attached.
Applications will be considered on a rolling basis until all positions are filled.
NACADA President Ruth Darling has appointed the following members to leadership positions beginning in October 2003. Congratulations to you all and a BIG thank you for agreeing to serve your organization!
Administrative Division Representative: John Mortensen Regional Division Representative: Brian Glankler Journal Co-Editors: Terry Kuhn and Gary Padak
Administrators' Institute Advisory Board: Susan Campbell (Chair), Alice Reinarz, Rich Robbins, Gene Calderon, Lynn Freeman, Vicki McGillin, Carolyn Collins, Tom Grites, Albert Matheny
Summer Institute Advisory Board: Wes Habley (Chair), Nancy King, Susan Campbell, John Burton, Tom Kerr, Dorothy Turk, Wanda Martin, Peggy King, Casey Self, Betsy McCalla-Wriggins
In addition, Barbara Bucey is Program Chair for the 2004 National Conference.
The Council for the Advancement of Standards in Higher Education (CAS) announces the release of a major revision of its landmark publication, the Book of Professional Standards for Higher Education, and an updated version of the CAS Self-Assessment Guides (SAGs). Available August 29, the new 2003 book of standards and guidelines incorporates significant updates, including new general standards with greater detail about desired outcomes in student learning and development. The standards for Academic Advising are presently under going a major revision as well. Newly revised self-assessment guides (SAGs) feature an effective means for measuring how these standards are being met in all 30 functional areas. SAGs are available in both print and interactive CD-Rom formats (both PC and Mac). The CD-Rom also includes a new PowerPoint presentation and an E-learning course to assist institutional staff and faculty members in completing the SAGs.
Further information and online orders are available through the CAS web site (http://www.cas.edu). The book, which replaces the 2001 edition, is available separately or in a special package with the interactive CD-Rom including the full set of SAGs. The SAGs are also available for individual purchase and immediate download. The book and SAGs can be ordered from CAS, One Dupont Circle, NW, Suite 200, Washington, D.C. 20036-1188. Telephone: (202) 862-1400, Fax: (202) 296-3286.
The National Academic Advising Association is a member of CAS and strongly endorses these standards to its members. Founded in 1979, CAS is a consortium of educational associations that promotes quality educational practices through the promulgation of standards and guidelines for 30 programs and services in higher education. Individuals and institutions from the 32 CAS member organizations comprise a professional constituency of well over 100,000 professionals. Excellence in educational practice is a central goal of CAS that is achieved through the implementation of standards in all areas of practice in higher education. This vision for excellence is consistent with contemporary goals for accountability and bring an effective approach-the CAS approach-to program assessment. The CAS approach is based on concepts of self-regulation and self-assessment, and all CAS materials are geared to this approach to quality assurance in higher education.
CAS serves higher education programs and services by providing:
Visit http://www.cas.edu for orders for all CAS materials, a full account of the work of CAS, links to each member association Internet sites and to the leadership of CAS, and a brief PowerPoint presentation providing an overview of the CAS approach.
Heidi Koring, Chair, Advisor Training and Development Commission
Training and development of advisors becomes ever more central to the effectiveness of the advising process with the increasing diversity and complexity of our students' environments. While there is no 'one-size-fits-all' method for advisor training and development, case studies are among the most useful items in the trainer's tool box.
Case studies are an effective part of the training process whether advisor training takes place as a single workshop, or as series of continuing in-service meetings, or in formal presentations or informal discussions. The use of case studies was pioneered by the Harvard School of Business faculty in the late 1960's. Currently used to enhance skills development of a variety of populations, case studies add richness and complexity to advisor training, reflecting the complex environment of contemporary college students. Case studies not only help advisors come to grips with the ambiguities and complexities of student development, but aid them in improving human relations and problem solving skills. Case studies can be used as exemplars of carefully defined problems, providing advisor with opportunities to practice analysis of an advising situation. Presenting a platform for addressing differences in advising styles, case studies stimulate personal and professional growth and reflection. Cases can be used with advisors at all levels of experience, engaging them in discussion and simulating problem solving in real life situations.
Good cases are realistic and personalized to the advisors' milieu. They are dramatic enough to engage the participants and ambiguous enough to allow for multiple interpretations. To prepare effective cases, collect anecdotes from advisors throughout the academic year. Asking advisors to reflect in writing on difficult situations can yield rich material for case studies. When turning the raw material of experience into cases for training, it is wise to assemble a team of stakeholders to read the anecdotes and discuss the issues addressed in each. Simultaneously, develop a list of resources that could help advisors address each case. Work from the issues creating composite cases that address one main question and at least one subordinate issue. If you are working from real experiences, make sure that details are changed so the persons involved in the original anecdote are not recognizable. Divide cases into categories by issue for use when planning training events.
Since the traditional case study approach uses small or large group discussion, begin with advisors enumerating the issues presented in the case. Discourage any tendency to find easy closure by encouraging participants to consider the case from different characters' points of view. Ask 'what if' questions. Consider locus of control and responsibility issues. What aspects of the case are within the advisor's locus of control? What aspects are not? Ask probing questions about each character's motivation. Look for hidden agendas. Use a team approach to problem solving by encouraging the exploration of resource and referral possibilities. Discuss how college confidentiality policies would affect each case. If appropriate, ask if the gender or ethnicity of characters affect the outcome of the case. Explore several related cases to develop the best practices or procedures for dealing with a particular advising challenge, at your institution.
Less traditional delivery methods can be used when approaching cases. Enlist the cooperation of theater or broadcast majors to make videos acting out specific cases. At some colleges and universities, faculty and student organizations are eager to produce case study vignettes as projects. The NACADA faculty advisor training video contains eight brief vignettes, six of which show a developing relationship between a first year student and a new faculty advisor and two scenarios exploring the needs of adult students. If the training event includes trainees who are comfortable with each other, have participants role play cases. Begin an advisor training electronic list which features one case a month for discussion.
To stimulate your use of case studies with advisors, here's a scenario that can be adapted to your institution for advisor training:
Lisa is a first year student from a neighboring state. She attended a competitive 'magnet' school with a 90.3% average and 1200 SAT's (27 Composite ACT). During orientation she tells you she's considering a pre-veterinary science track because she loves animals. Her midterm grades are B's and C's in calculus and biology. When she meets with you at midterm, she slumps in her chair and doesn't make eye contact. She's lost a lot of weight. Through her hesitant replies, you learn that math and science are tougher than she expected. She says she's dumber than she thought. She has a lot of headaches and sleeps a lot. She's missing classes because she says it doesn't matter if she goes. She tells you she's thought about going home, but is sure her family would just say she's a failure. Besides, her parents are getting a divorce and she's not sure where she would live. She says she knows you can't help, so maybe she'll just 'give up.'
Want to know how others are using case studies? The new monograph, Advisor Training: Exemplary Practices in the Development of Advisor Skills, features several Exemplary Practices utilizing case studies.
Heidi Koring Lynchburg College
Johanna Pionke, Probation, Dismissal & Reinstatement Issues Interest Group Chair
Why do some students fail to succeed in college? What interventions are most successful with these students? There is great demand for research revolving around these questions. As chair of the Probation, Dismissal & Reinstatement (PDR) Issues Interest Group, I challenge you to approach your PDR students from a research perspective.
Students typically do not come to college expecting to fail. Instead, most enter college with the expectation that they will have the opportunity gain knowledge that can help them earn a better living for themselves and their families. Research verifies that students often believe that there are few reasons why they will not succeed. They view academic probation or dismissal as something that will not happen to them.
Bartlett (2002) cites the 2001 Cooperative Institutional Research Program (CIRP) Annual Survey of Freshman Students showing that 44.1% of freshmen reported earning 'A' averages in high school. 57.5% of freshmen estimated their chances of making at least a B average in college as being very good. 76.5% of students expected to earn a bachelor's degree while 20.8% thought they had a very good chance of graduating from college with honors. Only 0.9% felt there was a good chance they might drop out of college temporarily, while 0.7% felt chances were very good that they would drop out of college permanently. Among this same cohort of students, 67.9% rated themselves above average in academic ability, though only 45% ranked themselves above average in writing ability and 44.2% ranked themselves above average in mathematical ability.
Despite students' positive attitude regarding their academic abilities, many end up on the academic probation, suspension or dismissal rolls. At the conclusion of each academic term, advisors, faculty and administrators review these students' academic progress and wonder why they were not successful. We look for ways to identify, or predict, those who are at greatest academic risk so that we may prevent their downward spiral. We question whether we should intervene with students who are struggling academically, or if it is better to invest time and resources on more successful students.
In journals, books and other publications, we search research for information relating to academic recovery issues, yet find little available. At conferences, it's often standing room only' in sessions discussing intervention programs at other institutions. Yet different student and program variables affect an institution's intervention program. Programs vary widely in terms of their requirements, structure, and level of intrusiveness. Some require a weekly class while others rely upon regular contact with advisors or mentors. Some intervention programs utilize group activities and tutorial support services, while still others require counseling services. Some programs are organized at the departmental level; others are college wide.
With so many different variables, it is difficult to attribute student academic success or failure directly to participation in an intervention program. What roles do student characteristics play in a student's ability to succeed? Do students' academic preparation, job and family responsibilities, study skills, or locus of control affect success? Is there a way to account for these variables?
At the conclusion of each academic term, I challenge you to look for research questions within the components of your institution's probation, dismissal and reinstatement procedures. Turn these into research projects and share your results with the Probation, Dismissal & Reinstatement (PDR) Issues Interest Group.
Johanna Pionke Kent State University
Reference
Bartlett, Thomas. (2002, February 1). Evaluating Student Attitudes is More Difficulty This Year. The Chronicle of Higher Education, Volume 29, Issue 21, p. A35, 4p.
At the NACADA Clearinghouse -
Higgins, Beth (2004, December 1) 'Advising Students on Probation' http://www.nacada.ksu.edu/Clearinghouse/AdvisingIssues/Probation.htm
Jennifer L. Bloom, Member Career Services Committee Chair
Dear Career Corner: I just found out that I have been invited to participate in a video conference interview-do you have any advice on how to approach this interview?-Signed, Video Neophyte
Dear Video Neophyte:Congratulations on making it to this preliminary interview stage-it means that the written materials you submitted caught the attention of the selection committee. The key now is to prepare like you would for any interview-do your homework on the institution, the position, and your potential new boss and colleagues. Please request the full job description from the search chair as well as other materials that will prepare you for the interview-strategic plans for the unit and/or the institution, written materials that the institution distributes to prospective students, organizational charts, mission statements, etc. You will also want to contact people in your network of colleagues that are or have been affiliated with this institution. Find out as much information about the position, the person who held this position previously and why they left, and the culture of the unit.
Here are some specific tips concerning the video conference itself.
This new NACADA Newsletter feature will be a regular column. Submit questions on-line at http://www.nacada.ksu.edu/AdministrativeDivision/career.htm. Questions will be answered anonymously.
Want to read more about it?
Kennedy, J. L. (1996). Job interviews for dummies. Foster City: IDG Books Worldwide Inc.
Krannich, C. R.,& Krannich, R. L. (1999). 101 dynamite answers to interview questions. Mannassas Park: Impact Publications.
Martin, C. (2001). Interview fitness training: A workout with Carole Martin the interview coach. San Ramon, California: Interview Publishing.
Martin, N. A. & Bloom, J. L. (2003). Career Aspirations & Expeditions: Advancing Your Career in Higher Education Administration. Champaign, IL: Stipes Publishing.
