|
The National Board of Medical Examiners (NBME) Computer-based Case Simulations
(CCS) have been designed to simulate patient-management exercises. There are
over 2300 possible choices that a student can select without any prompting
other than the initial patient presentation. The strategy employed in solving
the case is recorded as a series of the test items selected. This provides a
very rich source of data from which models of student understanding of complex
patient management can be built.
The first step in this process is the generation of a logical grouping of the
test item selections into groups of similarity or "domains." This grouping
results in the definition of a "problem space" that organizes all the elements
of the case. A student’s progress through the various domains can be
monitored.
By tracking performance strategies, we are able to generate a map of the steps
used to work through a case, a process we refer to as search path mapping.
This process has been useful with specific course material we have developed
for courses in allergy and immunology at the UCLA School of Medicine using
IMMEX (Interactive MultiMedia Exercises) software. However, attempting to
dissect strategies on CCS-based data can lead to difficulty due to the
complexity of these cases. An example of a CCS case involving an emergency
infectious disease case displays the complex nature of the strategies employed
in clinical case management. (Click Here to view search path map example)
We have utilized a constructive data modeling approach that utilizes the
pattern recognition capabilities of artificial neural networks to build
performance models from existing complex data sets. Artificial neural networks
are non-parametric techniques which build rich models of complex phenomena
through a training and pattern recognition process and are capable of
categorizing behavior based on actual performance sequences. Neural networks
have had practical utility in solving classification problems with ill-defined
categories, where the patterns are often deeply hidden within the data or
where there are poorly defined models of behavior.
By training our ANN’s with the perfromance data from our example case we have
been able to correlate NBME ratings to specific areas of output on the ANN. We
have also been able to understand why certain strategies result in
classification at varoius nodes. The following examples illustrate some of our
findings:
 ANN
outputs for excellent ratings (greater than 6 out of 8). This is the
ANN output for highly rated performances note that nodes 43 and 44 are the
most significant clusters.
 ANN
outputs for poor ratings (less than 4 out of 8). This is the ANN
output for the poorly rated performances. Note that clustering, in general, is
not a common feature of poor performance strategies. Also note that node 29 is
represented as it was in the highly rated performances. An explanation for
this will appear here soon and will also be part of a manuscript, which is
currently in press. Check periodically for updates.
|
