WORKSHOP ON DEVELOPING A RESEARCH AGENDA FOR ELECTRONIC
VOTING TECHNOLOGIES
September 17-18, 2004
Convened by the American Association for the Advancement of Science
Main | Program | Participants | Synopses
Sharon Laskowski
A SYNOPSIS OF PRESSING RESEARCH QUESTIONS PERTAINING TO ELECTRONIC VOTING TECHNOLOGIES
The research issues described here are based on the findings and recommendations in NIST Special Publication 500-256 "Improving the Usability and Accessibility of Voting Systems and Products" which was written to address the mandate in the Help America Vote Act (HAVA) of 2002, Public Law 107-252 for the Election Assistance Commission in consultation with the National Institute of Standards and Technology (NIST) to submit a human factors report to Congress. This report "assesses the areas of human factor research, including usability engineering and human-computer and human-machine interaction, which feasibly could be applied to voting products and systems design to ensure the usability and accuracy of voting products and systems, including methods to improve access for individuals with disabilities (including blindness) and individuals with limited proficiency in the English language and to reduce voter error and the number of spoiled ballots in elections." The report was delivered to Congress in May 2004 and can be found at http://vote.nist.gov and http://www.eac.gov .
The report examines the key needs for developing a set of voting system standards that will ensure a high level of usability and accessibility. Because voting equipment will be tested for conformance against these standards and qualified/certified to the standard, the standards must be testable and the test protocols must assure repeatability of the results. There are ten recommendations in the report covering not just voting equipment standards, but also recommending the development of guidelines for ballot design, layout, usability of documentation and training, and vendor user centered design. I also refer readers to Chapter 5 of that report which summarizes current research in human factors, user interaction, usability, and accessibility and discusses what types of research in these areas can be applied to voting systems. In this short synopsis, we focus on research issues for usability testing.
In general, the single most critical need identified in the report is a set of usability standards for voting systems that are performance-based and that support objective measures and associated conformance test procedures that can be used for the certification and qualification of voting products and systems. These measures and conformance test procedures primarily depend upon having objective usability testing processes. (Note that when we are referring to usability we include all users, with and without disabilities, at different levels of reading proficiency and from different cultural and economic backgrounds.) A set of design requirements cannot properly address the issues of usability for voting system products. Also, no document can contain a sufficient set of design requirements to ensure voting product usability unless the document completely specifies a design already shown to be usable. Further, formative or diagnostic tests are valuable tools in the design process; they do not guarantee that the final product is usable as measured by the standard usability metrics (efficiency, effectiveness, and satisfaction) since they are used during the design process, not on a final product. Even tests that are conducted on a final product design are generally not conducted in a way that would allow the results to be generalized to the intended populations (i.e., the participants of the study may or may not be appropriately extrapolated to a majority of all actual users). This is particularly true for voting system products since the range of users required for such a test would make this type of testing cost prohibitive to most vendors. In addition, there are currently no defined standards for usability metrics that vendors could use as benchmarks for their testing.
To ensure usability of a voting product, it is imperative that the product be tested with actual users performing realistic tasks in a realistic environment, in sufficient numbers, and using a broad enough cross-section of users to be truly representative of the voting population. Further, to ensure good usability of the system, we must test not only the interaction of voter with the product but also the interaction of the voters, election administrators, and poll workers with the entire voting system.
We recommend the development of a valid, reliable, repeatable, and reproducible process for usability testing of voting products against agreed-upon usability pass/fail requirements. In particular, there must be a careful definition of the metrics, such as what counts as an error, how to measure error rate, time on task, etc., by which systems are to be measured. The pass/fail criteria should be restricted to usability problems leading to partial failure, and usability problems leading to total failure. Since we are dealing with outcomes, usability problems prior to success need not be specifically included, but would be represented in the time on task measure from testing. Note that while excessive time required does not lead to failure, it is still unacceptable.
Since human users are involved in the process, it is unlikely that the error rates will be zero for any criteria established, so a specific acceptable error rate and margin of error will likely be required. For example, it may be possible to enforce a requirement that no user be allowed to consciously cast a ballot with an over vote for one or more contests since this error represents the action of the voter. However, a voter still might inadvertently cast a vote for an unintended candidate in any product but this error cannot be detected without knowing the intent of the voter. Yet, both of these conditions must be tested. This test process must be defined at a high enough level of generality that the same procedure could be applied to any product (i.e., we do not want to define product-specific tests). Otherwise, the results for various products would not be comparable. Fortunately, the task requirements for voting are specific enough that this should not be difficult to do. It might be necessary, however, to have technology-specific variants of the test procedure and protocol (e.g. DRE vs. paper-based), although we believe the differences can and should be kept minimal.
Research needs to be conducted to determine: (1) the nature of errors possible during a voting process (this includes voter errors and poll worker errors), and (2) the level (rate) of these errors (both the current levels for existing products and recommendations for "acceptable" levels of each error type). Once this information is available, we recommend that a set of repeatable and reproducible processes be defined and that each voting product be tested using these test processes and usability test pass/fail criteria. This would include the definition of all test procedures, the data collection required, the data analysis approach, participant screening and selection procedures, and reporting requirements.