Global Security and Intelligence Studies Volume 4, Number 1, Spring/Summer 2019 | Page 26

Forging Consensus? Approaches to Assessment in Intelligence Studies Programs
ligence and National Security Capstone course. The expectation is that student
scores on this assessment instrument should improve between the pre and posttest
due to their exposure to the curriculum.
While collecting quantitative data to measure these outcomes is the easiest
to process, many outcomes do not easily lend themselves to quantitative measurement.
Objectives related to critical thinking and communication fall squarely into
this realm. Many programs utilize qualitative approaches to assessment, but these
come with their own problems. For instance, the program director at Notre Dame
College noted the subjectivity associated with trying to assess a complex skill such
as communication (Gregory Moore, Telephone interview with author, December
20, 2017). So, how a given objective is defined and operationalized can vary widely.
Even with a common standard, is the measure applied without subjectivity? This is
where the utilization of rubrics is intended to assist.
The other issue with qualitative assessment practices is the time required in
the effort. For instance, while processing multiple-choice tests can be done in minutes,
the assessment of terms papers is likely to involve many hours of effort. This
issue is compounded by the fact that faculty often have to implement the assessment
tool as an extracurricular function to their teaching. Some faculty address
this problem by incorporating the assessment tool as a part of a course grade, but
this can lead to potential confusion over grades, which are a measure of a student’s
progress in a given course, and assessment, which is about assessing the effectiveness
of the program that the student is in.
Indeed, given that assessment tools can involve substantial time outside of
the classroom, some programs do not assess all of their SLOs on an annual basis.
Instead, some select a particular outcome for that year and only assess that area.
Another option is to assess only a sample of the student work. However, both of
these approaches exacerbate concerns about validity in the process since they raise
concerns that the particular outcome being studied in a given year or the sample
being reviewed may not be representative.
Capstone courses are a common venue for assessment mechanisms since
they are scheduled to be one of the last courses in a student’s curriculum. For
instance, of the 13 undergraduate programs that are listed in the Coulthart and
Crosston study, 7 have a capstone requirement listed as a required course. In the
Bachelors program at Angelo State University, all of the summative SLOs for the
program are assessed within their capstone course, ISSA 4403 Capstone: Case
Studies in Intelligence (Tony Mullis, E-Mail Message to Author, January 25, 2018).
At the graduate level, thesis projects often serve a comparable function—indeed,
in the graduate program at Angelo State University, students can either take the
capstone course or complete a thesis. The system uses either venue as the summative
assessment for its discipline-specific knowledge and communication learning
objectives.
15