home

Rejoinder to the ACC Ad Hoc Evaluation Committee on FIGs

September 2005

This minority report is in response to the ACC ad hoc evaluation committee
report issued in spring 2005. Its purpose is to delineate some of the methodological
shortcomings of the ad hoc committee report, which therefore calls into question the
report’s final recommendations regarding the FIG program’s failure to meet its original
objectives.

Second, additional information about the demographic background of FIG
students is examined, which further underscores the efficacy of the program. Third, the
ad hoc report recommendations are reviewed and considered in light of existing resources
and organizational structures. This report is prepared by the Office of the Vice Provost
for Undergraduate Education, with assistance from the Office of Institutional
Assessment, Research and Testing (OIART).

Review of the Literature

Programs to help students in their transition to college –termed of late the “first
year experience” –are not new. They have, in fact, been around since the mid-forties, fell
out of favor in the sixties, and re-emerged slowly in the late seventies. By the late
eighties, more and more campuses turned to some type of academic program for first year
students that would address issues of retention, changing student populations, and fiscal
pressures.i  Today, it’s estimated that nearly three-fourths of all undergraduate campuses
in the U.S. offer some sort of freshman seminar. Of that number, half require it of all
their first-year students. Most are letter-graded and address outcomes pertaining to the
transition to college and study skills.ii

In the approximately three decades since first year programs have taken hold, a
strong body of research on their efficacy has emerged, not just on retention, but on a
variety of other outcomes deemed important for students’ effective transition to college
and level of engagement with the college experience. In general, As Boyer points out:

A successful freshman-year program will convince students that they are part of
an intellectually vital, caring community, and the spirit of community will be
sustained by a climate on the campus where personal relationships are prized,
where integrity is the hallmark of discourse, and where people speak and listen
carefully to each other. ( Boyer, 1987, p. 57)iii

New research also indicates that a seminar that is linked to another course with an
explicit academic theme, and taught with “engaging” pedagogical strategies (including
active learning, group work, and civil discourse) have a strong positive correlation to
students’ acquisition of cognitive reasoning skills, motivation, and intellectual
development. The success of this “learning community” model is compelling its adoption
by a growing number of institutions. ivv Western’s FIG Program is based on such a linked
“learning community” model, and commensurate with national research, is demonstrating
positive outcomes for its students.

Methodological Critique

The report submitted by the Ad Hoc Fig Evaluation Committee shows clear
evidence of bias and misrepresentation of evaluation outcomes regarding the FIGs. In
addition, it privileges data obtained from a single non-representative focus group
conducted by a person without training in focus group research, or qualitative
methodology, over both quantitative and qualitative data obtained from yearly samples of
the entering FIGs cohort of students. To substantiate these claims and elaborate on the
poor quality of the committee’s research, the following points are offered

First, the report begins with a typical unsubstantiated statement that “The central
issue we have uncovered is a nagging concern with the unevenness of the academic
content and evaluation procedures in the FIG seminars.” Nowhere in the report is
evidence presented that such a “nagging” concern exists and nowhere is there a report on
a detailed analysis of the course materials used in the FIGs seminars. Over the years
many seminars had integrating course projects. Where is the information on these
projects and an evaluation of their academic soundness? If the committee is going to be
critical of the academic content of the seminars, then sound evaluation practices would
dictate a careful analysis of these materials rather the reliance on hearsay from a few
students who made themselves known to the committee. Such broad unsubstantiated

statements can be found throughout the report (e.g., see page 10 under Seminars, where
no reliable evidence is provided to support claims of failure to achieve program goals or
concern among students regarding academic content of seminars).

Second, the report fails to mention any consultation with Gary McKinney in the
OIART who has been responsible for conducting some of the most thorough program
evaluations on campus for the FIGs program. More consultation may have avoided some
of the misinterpretations of data found in the committee’s report.

Third, there are numerous instances of data non-specificity and misrepresentation
in the report. For example, on page 6 the report states that FIG students in the 1999-2000
year were more likely to receive a D or less in their large lecture classes whereas non-FIG
students were more likely to receive a B+ or above. In fact, the data show that 23% of
FIG students received a B+ or above compared to only 21.3% of non-FIGs and the
slightly greater percent of FIGs students with grades of D or below could have been due
to higher completion rates in the course by the FIGs students who were marginal. As a
result, 1999-2000 FIGs students tended to accumulate more credits in their first year as
well as achieve higher overall GPAs (even with the greater attrition of poorly performing
students from the non-FIG group). Further, the report fails to note an obvious preexisting
difference between the FIGs and non-FIGs students that is presented in the same table
they reference. Namely, that FIGs students reported a much lower proportion with HS
GPAs of B+ and above (67.8% vs. 76.6%) and a greater proportion with HS GPAs of B-
or below (14.5% vs. 4.9%), which suggests initial disadvantage that was overcome.

Fourth, the report criticizes FIG evaluation surveys for not being reliable because
they don’t compare FIGs to non-FIGs. This is disingenuous since these surveys were
created primarily for the purpose of process evaluations designed to modify and improve
the program in its early years. Even so, the methodology the committee relies upon (a
focus group of 7 students, with 5 additional students providing “comments”) is far
inferior to the well planned and implemented surveys of nearly the entire FIG student
cohorts done each year by the OIART. In fact, the use of focus groups to do outcome
evaluations of programs is unheard of, because qualitative data is easily manipulated and
subject to researcher bias. The committee should only use these data to give them ideas
for measures or analyses using quantitative data. If they are able to substantiate claims
made in the qualitative data, it would be fine to then use the students’ and faculty’s
comments to give greater meaning to their results, but instead they use the qualitative
data to try to counteract the positive effects found in the more reliable quantitative
surveys. For example, on page 24, they note that more than half of students agreed that
the GUR courses were related to each other (and only 12-24% disagreed), but then say
“The students in the focus group felt that the link between the seminar and the GURs was
strongly lacking.” Are we to believe comments of a focus group of 7 (+5) taken from an
unknown number of different years of FIGs and solicited under an unknown context of
discussion over nearly the entire group of students who took FIGs seminars and
completed an anonymous survey? I would hope not.

Fifth, the report states that the FIG program goals are not clear (page 7). Contrary
to this statement, there are extensive documents spelling out the FIGs goals and there
have been detailed evaluation outcome criteria developed in accordance with those goals.

Sixth, there may be legitimate problems with the grading in the seminars, but it
would be wrong to discount any success that the FIGs is having by saying that the
resulting good grades for students in the seminars are proof that there is no academic
substance to them. There may be a confounding of issues in this report (with a tendency
to stray) into consideration of grade inflation. This is a much broader campus (and, in
fact, societal) issue…one that cannot be solved in these analyses.
 

Seventh, the report makes no mention of several survey responses that would suggest a
broader success of the seminars and a greater academic component than is inferred by the
report. For example, the following are the percent that agreed vs. disagreed (neutrals
being removed) with each of the following in 2002 and 2003:

2002 2003

--The seminar challenged me 39% 55%

--The seminar created an environment

in which I learned a great deal 51% 68%

--The seminar was worthwhile 51% 75%

--Skills I learned in the seminar will

help me with other courses 69% 81%

Even in 2004 (an arguably “failed” experiment), the percentage who agreed somewhat or
a great deal with the idea that the FIGs helped make connections with peers, assisted with
learning in the linked course, and gave me skills and strategies for learning in other
courses were 90.4, 55.7 and 73.7, respectively.

Eighth, in general the report fails to make distinctions between years of the
program. The reader is left with no idea as to what respective years faculty or students
were referring to when they made comments in the qualitative interviews.

Ninth, and perhaps most importantly, the statement that the FIGs students are not
different from or are possibly more qualified and motivated than the non-FIGs students
when they enter is patently false. The report takes this information from the 2000 report
which limited its comparisons to gender, race, first generation status and GPA. In fact,
reports since the 2000 report show that later, more expanded, cohorts of FIGs students
look more disadvantaged than their non-FIG counterparts. The attached comparison for
the 2003 cohort clearly shows a more disadvantaged group both academically (in terms
of SAT scores, HS preparation, running start, motivations and honors) and socio-
emotionally (in terms of expected difficulties and concerns). It is also highly likely that
science students are greatly underrepresented in the FIGs group due to advising that
encourages these students to get started in their math and science courses earlier on and
not get involved in a FIG which requires two GUR courses in the first term. Although the
report notes that science courses have lower GPAs, science majors are likely to have
higher GPAs and skew the non-FIGs group upward (although further analysis is needed
to account for this factor).

Tenth, there are points where data is manipulated to show a result. This is evident
on page 23 where the percent of respondents with a neutral or negative response to the
question of whether FIGs is what was expected is reported. Unfortunately, most of the
responses were neutral and if the negatives are compared to the positives, there was a
much greater percentage of positive responses to this question than negative ones. A
similar report is found on page 25 with respect to the perception that FIGs was
worthwhile or that it was a positive experience. In fact, the data clearly show that in 2003
73% felt it was a positive experience and 68% said it was worthwhile with only 11 %
saying it wasn’t worthwhile. In addition, another 73% said that they would recommend
the FIGs to incoming students. These are the important data. The whole discussion about
declining positives toward FIGs within cohorts over a single year is really tangential to
the question of effectiveness and satisfaction. Such declines should be expected in any
course over a quarter and these declines are very small given the positive expectations
that most students had to begin with. In all likelihood this is a statistical artifact of
regression toward the mean.
 

In conclusion, the Senate is encouraged to look at the FIGs evaluation data for
themselves before forming opinions about it and making decisions. The OIART is also
conducting further evaluation studies of the 2003 cohort using WELS survey data and a
two year follow-up.

Preliminary Report on FIGs Student Comparisons Based on AY2003-2004 Data
from WELS

FIGs students entering in 2003 were compared to Non-FIGs students entering the
same year on a wide range of characteristics to assess the likelihood that outcome
assessments could biased by self-selection into the FIGs group. All possible variables for
which we had data that could distinguish incoming students and possibly affect their first-
year outcomes were compared across the two groups. The results are summarized below:

Sex:  FIGs more likely to be female
HS GPA: No statistical difference (3.52 vs. 3.52)
First Generation: No statistical difference (32% vs. 34%)
SAT Math: FIGs significantly lower SAT scores in Math (537 vs. 561)
SAT Verbal: FIGs significantly lower SAT scores in Verbal (543 vs. 562)

Running Start: FIGs significantly less likely to have been in running start (5.9%
vs. 15%)
Admissions Index: No statistical difference (High weighting of GPA makes this
redundant)
Honors Enrollment: FIGs significantly less likely to be enrolled in honors (0% vs. 4%)

There were no statistically significant differences between FIGs and Non-FIGs
students on the following measures assessed in the 2003 WELS Baseline Survey: College
Importance, WWU as First Choice, HS College Prep, College Motivation, Expected
Academic Engagement, Expected Difficulty Balancing Demands, Expected Community
Engagement, Expected Issues with Deviant Behavior, Expected Financial Difficulties,
Expectation of Seeking Help, Utilitarian Approach to HS Courses, Intrinsic Value of HS
Courses, Nature of Motivation to Attend College (Utilitarian, Independence, Career,
Other Non-Academic), HS Engagement (Academic, Social, Technology, Television),
Academic Self-Concept (Foreclosed, Expect Success), HS Academic Prep (Group Work,
Cultural Understanding).

There were significant differences on the following:
 

Transfer Expectations: FIGs less likely to expect to transfer.

Expected Academic Uncertainty: FIGs more likely to expect uncertainty re:
academics.

Expected Emotional/Spiritual: FIGs more likely to expect difficulties

HS Challenge – Personal: FIGs more likely to select HS courses for personal
challenge

College Motivation – Personal FIGs more likely to attend college for personal
improvement

HS Academic Preparation - Science FIGs lower scores on HS academic preparation in
science

HS Academic Preparation – Comm FIGs lower score on HS academic preparation in
communication

These data leave little question that the FIGs group is more disadvantaged when
they enter and any positive outcomes they experience from FIGs are likely to be muted
by these initial differences. Further analyses will attempt to make statistical adjustments
for these initial differences.

Review of Recommendations

While a number of criticisms were aimed at the FIG program, there appear to be
only two central recommendations to come from the ad hoc evaluation committee with
regard to future efforts in the program. These are 1) faculty oversight of the program,
and 2) allocation of resources to insure quality of the program.

With respect to the first recommendation, there has been extensive and ongoing
faculty input in the design, implementation and oversight of Western’s FIG program. In
January 1998, former Provost DeLorme formed a task force to study the first-year
experience. This task force was chaired by Marie Eaton and included Kris Bulcroft,
Anna Carey, Dennis Catrell, Catherine Craine-Thorsen, Laura Crary, Thor Hansen, Milt
Krieger, Donna Qualley, Carmen Werder, Gary McKinney, and Ted Pratt. Of these
twelve task force members, 8 were tenure-track faculty members. Based on the task
force recommendations and identification of 6 specific learning outcomes, an
implementation task force was formed in October 1998. Members of this group included
Gerry Prody, Catherine Craine-Thorseon, Milt Krieger, Ken Hoover, Kathy Young, Carol
Jansen, and Amanda Eurich, all tenure-track members of the Western community. The
implementation group also included student support professionals such as Joe St. Hilaire,
Karen Copetas, Deb Frost, Tony Gulig, Anna Carey, Gary McKinney and Karen Casto.
Kris Bulcroft was asked by interim Provost Murphy to chair this implementation task
force. Based on the task force recommendations, and working within the budgetary
parameter of $20,000 allocated for the pilot project of FIGs, the cluster model (as
described in previous reports) was recommended and FIGs were open to students in fall
1999.

Faculty and staff were further involved in refining the registration processes of
FIGs when problems regarding linked course registration surfaced during Summerstart
1999. Provost Andrew Bodman asked that a FIG debriefing task force consider ways of
improving the registration process. This group met in winter 2000 and consisted of
faculty and administrative support staff including representation from the registrar’s
office, ADMCS, and University Planning and Analysis.

Following the first set of FIG offerings in fall 1999, the Office of Institutional
Assessment, Research and Testing worked closely with the Vice Provost for
Undergraduate Education to conduct annual program evaluations that included feedback
from FIG students, seminar facilitators and faculty involved in the GUR courses. From
these, program improvements were continuously made. Since its inception, reports about
the FIG program have been given to the Academic Coordinating Commission (3/30/04),
the Board of Trustees (October 2001), and the General Education Task Force (Winter
2002).

In conclusion, the FIGs program has garnered considerable faculty and campus-
wide oversight. (In fact, it may be one of the most thoroughly assessed and
“committeed” programs on Western’s campus.) It is not a “top down” designed program,
but represents countless hours of faculty and student support professionals’ time and due
consideration. The missing link in the organizational structure, it would seem, is a
feedback loop that brings these campus-wide oversight efforts back into the shared
governance system. We would agree that the ACC ad hoc committee report
recommendation calling for a better method of reporting program outcomes is long
overdue; numerous other interdisciplinary programs and efforts across the campus have
also lamented the absence of such a body.  As is evidence by examining the historical
underpinnings of the FIG program, in the absence of such a committee, numerous task
forces and ad hoc committees have filled the void in an attempt to improve our first-year
offerings for students and respond to the state’s demand to improve freshmen retention
rates. (This may also have contributed to what the ad hoc committee referred to as the
lack of transparency in the FIG Program). With renewed emphasis on international
offerings, coupled with proposed changes in general studies and first-year programming,
the time has come to work in concert to develop such an oversight group.

The second fundamental recommendation of the ACC ad hoc evaluation
committee is to provide better financial support for FIGs and first-year programs, in
general. Based on the recommendation of the General Education Task Force, and
endorsed by ACC and the Faculty Senate regarding the establishment of a first-year
seminar in the revised general education program, the time is right to increase support for
first-year curricular and co-curricular offerings. However, given the fiscal scenario of
higher education funding, it seems unlikely that substantial amounts of new dollars will
be forthcoming for FIGs or other first-year offerings. Knowing the fiscal limitations,
ACC recommended that the first-year seminar be an “option” rather than a “requirement”
of the revised general education program. In keeping with ACC’s recommendation, four
first-year seminars have been added to the 2005-‘06 offerings of first-year courses. FIGs
has slightly fewer seats offered this year (225 students). Student demand, however,
continues to increase with each subsequent cohort of new students, and it remains to be
seen if additional free-standing first-year seminars will be created and implemented in the
absence of additional financial incentives to academic departments. The commitment to
better support first-year programming is a decision that seems to have met with resistance
by some academic units, in keeping with their concern to maintain quality and current
levels of funding in academic major fields of study.

In conclusion, despite the methodological shortcomings of the ACC ad hoc
evaluation task force, bringing attention to the campus community and the shared
governance bodies the need for sustained, collective vision and commitment to better
funding the first-year experience is welcomed.

i Betsy O. Barefoot, Fidler, P. (1996). An Historical and Theoretical Framework For the Freshman
Seminar. In The 1994 National Survey of Freshman Seminar Program: Continuing Innovations in the
Collegiate Curriculum The Freshman Year Experience, Monograph Series Number 20: National Resource
Center for The Freshman Year Experience & Students in Transition: University of South Carolina

ii The 2000 national survey of first-year seminar programs: Continuing innovations in the collegiate
curriculum (Monograph No. 35). National Resource Center for The First-Year Experience and Students in
Transition. (2002). Columbia, SC: University of South Carolina,

iii Boyer, Ernest. (1987). College: The Undergraduate Experience in America. New York: Harper & Row.

iv Henscheid, J.M.(2004). First-year Seminars in Learning Communities: Two Reforms Intersect. In J.M.
Henscheid (Ed.) Integrating the First-Year Experience: The Role of First-Year Seminars in Learning
Communities. University of South Carolina: National Resource Center for the First-Year Experience and
Students in Transition.

v Swing, R.L. (2004). The improved learning outcomes of linked versus stand-alone first-year-seminars. In
J.M. Henscheid (Ed.) Integrating the First-Year Experience: The Role of First-Year Seminars in Learning
Communities. University of South Carolina: National Resource Center for the First-Year Experience and
Students in Transition.
 

home