Track Analysis

Ronald H. Stevens is designing testing materials that employ pattern recognition to help students profit from success and learn from their mistakes.

The Interactive MultiMedia Exercises (IMMEX) software system, now used for problem solving in science curriculums from K-12 to medical school, grew out of the frustration experienced more than a decade ago by the UCLA School of Medicine’s Ronald H. Stevens while administering multiple choice tests to his students. Having concluded that the traditional testing methods were inadequate in assessing students’ medical diagnostic skills, Stevens began to experiment with other formats. “In doing so, I discovered the extent to which the paper and pencil approach limits the depths to which you can go,” recalls the professor of microbiology and immunology. So he started learning about computer programming.

Stevens created the first version of IMMEX specifically for the UCLA medical school’s microbiology and immunology students. The program involved patient simulations: Students were given a case history and, after choosing and obtaining results from any of dozens of laboratory tests (with each, of course, presenting both benefits and costs), they were asked to come up with a range of possible diagnoses. Then, what began as an attempt to move away from multiple choice testing blossomed into a community wide effort in education and assessment. Today, IMMEX is a Windows based software system containing an “authoring shell” that can be adapted to fit any one of the more than 200 problem-solving scenarios currently available, each presenting a starting condition,a goal and numerous options on how to achieve that goal.

For example, students of the environmental sciences may confront a situation in which dead fish are washing up on the shores of a river and be asked to pinpoint the cause as efficiently as possible. In doing so, the students can venture down several paths, studying the river’s geography and surveying the surrounding industries, sampling various sites and, after deciding where to sample, choosing from among many possible tests. “With each selection students make, they get another piece of information — data they must interpret— to help them make their next decision,” Stevens notes. When they get to a test they don’t understand, they can always log on to the library for additional information.

When students work on IMMEX, a database is initialized and their movements are recorded. “It’s important to know how students are going about solving problems,” says Stevens. An IMMEX program called Analysis delves into the database of student performances — either individually, or as a partial or entire group — and allows instructors to view the problem-solving processes students undertook. Lines are connected from one concept domain to another, showing the path a particular student followed. When looking at an aggregate group, heavier lines represent more commonly traveled routes. Major concept areas are represented by colors. So, for example, when attempting to understand why students aren’t able to solve a problem, instructors can determine when students failed to obtain the relevant data, or when they obtained the data but failed to understand its significance. This process also enables instructors to gauge the relative difficulty and educational value of the problems themselves.

The next step is to automate the process of evaluating students’ problem solving strategies. Artificial “neural networks” may be quite useful for this purpose. The maps produced by Analysis require teacher interpretation, are not quantitative and don’t provide real time feedback to students. The neural networks, on the other hand, employ pattern recognition software programmed to detect the subtle patterns that translate to success, distinguishing among strategies in a quantifiable way.

“We’re trying to quantify the problem-solving process in a way that is simple and informative enough that teachers will want to use it, but also research oriented enough that teachers interested in pursuing advanced degrees will be able to explore the subject further,” Stevens says.

Efforts are also focused on teaching instructors how to program their own content. Supported by a $2-million National Science Foundation grant, as well as by grants from GTE and the Drown Foundation, Stevens is training Los Angeles K-12 teachers in the use of IMMEX. In Stevens’ program, the teachers attend a month-long workshop in which they learn how to author IMMEX programs, with UCLA faculty helping to ensure the quality of the new programs’ content. “We have K-12 teachers who are now writing grants, giving talks and returning to school for advanced degrees in this material,” Stevens reports. In addition, approximately 80 high school and middle school teachers have attended three day UCLA workshops, where they learn how to integrate the programs into the classroom setting.

“We’re looking at what students need to know prior to using the program, what types of follow ups are desirable and what supplementary materials are needed if students are to record their observations and results,” says Stevens, who is actively pursuing such questions with researchers in UCLA’s Graduate School of Education and Information Studies.

In the long term, Stevens would like to explore how to best use IMMEX to personalize the learning process for students. “Over the 12,000 times people have run the immunology problems, essentially no two students have followed exactly the same sequence of tests,” he says. “The learning of complex skills is very individualistic, and yet we continue to stand up and lecture to large groups of students as if they were all the same. This technology can help change that.”