Development and Testing of an Innovative Listening Comprehension (LC) Item Type Interface (CT-01-05)
by Kimberly Swygert and Michael Contreras

Executive Summary

The research and development plan for a computerized Law School Admission Test (C-LSAT) includes the development of prototypes for innovative item types, one of which is Listening Comprehension (LC). This item type requires test takers to listen to a selection once and then answer questions related to the selection, without the option of playing the selection again. The selections are designed to be analogous to monologues and dialogues that would be encountered in legal education situations. This item type is being considered for the C-LSAT for several reasons—listening skills have been judged to be critical for success in law school, these skills may be fundamental to counseling and negotiation processes in the practice of law, and the computerized format of a C-LSAT lends itself well to the administration of individual listening selections. Two versions of a computerized prototype for the LC item type were developed by Law School Admission Council (LSAC) in 2000 and fully usability tested that same year at the Law School Recruitment Forums.

These LC prototypes differ in several major ways from all C-LSAT prototypes that precede them. One way is that the selection is spoken, not written, so the user must have some way to access sound files. Another difference is that two versions of the interface were produced that presented the item stems differently—in one version the item stem was printed on the screen, and in the other version the item stem was a sound file that had to be played, much like the selection. The same selection and items were used in both versions. Finally, because this item type is innovative, the user was provided with an extensive set of directions to explain this item type and impart crucial information, such as the fact that the selection may be played only once, but the item (if played instead of read) may be played multiple times.

The LC prototype was put through formal usability testing on users who were self-selected from the candidates at the 2000 Los Angeles Forum. These users put on headphones, read the directions, played the selection, took notes if they chose, answered six questions about the selection, and viewed their score. Users were given no help in using the interface nor in answering the items. While the users were performing the usability tests, an LSAC staff member was recording their responses and movements through the prototype on a data entry sheet. A survey was administered in conjunction with the usability tests that collected demographic information, computer usage and comfort levels, and attitudes towards computerized tests in general and a potential computerized LSAT specifically.

The demographic results show that the forum attendee sex and ethnic group breakdowns are fairly close to the LSAT test taker breakdowns, but the forums tend to oversample female, African American, and Asian American attendees. Although no data was collected on the first language of the respondents, the Asian American and Hispanic participants are the most likely participants to have a language other than English as their first language. It is encouraging that this sample had a relatively high percentage of those ethnic groups, as the LC item type will ultimately be used to measure comprehension of monologues and dialogues spoken in English.

The computer usage and comfort levels of the users were very similar to what has been seen during other usability testing at the forums—the user group was very comfortable with computers. No users said they used a computer Seldom, Never, or only a few times per month, 16.0% used a computer weekly, and 84.0% used a computer daily. Almost all users rated themselves as being Somewhat or Very Comfortable using a computer, and Internet use was almost as popular as general computer use.

The reported comfort level in taking a computerized LSAT was polarized; 56.0% reported being Very or Somewhat Comfortable with the idea, 32.0% reported being Very or Somewhat Uncomfortable with the idea, and only 12.0% were neutral about it. Males seemed more positive about the computerized test than females. The same polarization appeared when users were asked which test they would take, if both a paper-and-pencil (P&P) LSAT and a computerized version of the LSAT were available; the females greatly preferred the P&P test, and only a quarter of them would prefer to take the computerized test, while almost half of the males would choose to take the computerized test, and less than a quarter would take the P&P test. Overall, however, the users were very positive about the aspects of a computerized test that were listed in the survey.

During the usability tests, the vast majority of the users did not have any trouble reading the text or understanding the directions. However, when queried on the specifics of the directions, almost a third of the users incorrectly said that the selection could be played more than once. Listening to the selection was easy for almost all of the users, but “following” the selection was not as easy. All of the users found it easy to move among items, and over three quarters were able to locate useful on-screen information such as the number of items and the timer. When the results were broken down by sex, one main difference seemed to be in the tasks involving understanding the rules and directions. All of the male users correctly said that both the items and selection would be spoken in the play-item version, while only 62.5% of the female users answered this correctly. Female users were also less likely to say (correctly) that the selection could be played only once.

Based on these results, it appears that the prototype interfaces presented in this paper may provide a viable method for further evaluation of the LC item type. Once revisions have been made to address the concerns with the directions and a final set of usability tests are conducted to assess these revisions, this interface should be suitable for field testing the LC items and gathering user performance statistics. Larger samples should be obtained so that any sex and ethnic group differences can be assessed. Finally, continuing efforts should be made to survey the users about the perceived difficulty and fairness of the item type, and English-as-a-second-language users should be utilized to insure, as much as possible, that there is no differential item functioning (DIF) or differential impact on such users.

Development and Testing of an Innovative Listening Comprehension (LC) Item Type Interface (CT-01-05)

Research Report Index