AuditEngine

This Page: copswiki.org/Common/AuditEngine

AuditEngine is an independent, comprehensive ballot-image auditing system developed by Citizens' Oversight. It examines digital images of paper ballots, independently determines the votes recorded on each ballot, and compares those findings with the official cast vote records (CVRs) produced by the voting system.

AuditEngine can analyze both hand-marked paper ballots and ballots produced by ballot marking devices (BMDs). For BMD ballots, it reads the human-readable selections rather than relying on encoded barcodes or QR codes.

The resulting reports identify discrepancies, support reconciliation of election records, and help election officials and the public evaluate the accuracy of reported election results.

How AuditEngine Works

AuditEngine overview: cast vote records and original ballot images pass through Index, Map, Extract and Compare

AuditEngine can be understood as a series of four major phases.

Input Data

AuditEngine requires two primary data sources, both exported from the voting system:

  1. Cast Vote Records (CVRs): Machine-readable records containing the votes interpreted by the voting system, along with associated metadata such as precinct, ballot style, and voting group. CVRs are typically provided in CSV, spreadsheet-compatible, or JSON formats.
  2. Ballot Image Archives (BIAs): Digital images of ballot sheets produced by the voting system, typically including both front and back images, organized into one or more ZIP archives.

Phase 1: Indexing

AuditEngine examines the CVR metadata and ballot image archives to create a unified index of ballot sheets. The votes reported in the CVR are kept separate from the independent vote-extraction process until Phase 4.

  1. CVR Metadata: Extracts ballot identifiers, precincts, ballot styles, and other metadata without using the CVR vote selections.
  2. Image Inspection: Examines every ballot image to identify its dimensions, encoded ballot-style information, and cryptographic hash.
  3. Ballot Classification: Distinguishes full-face ballots from barcoded ballots.
  4. Detect Exact Repeats: Identifies repeated ballot identifiers and identical ballot images.
  5. Indexing Report and Election Information File (EIF): Summarizes the input data and creates an EIF describing the election's contests and candidate or response options.

Phase 2: Mapping

  1. Ballot Layout Analysis: Identifies ballot layouts and the locations of voting targets, such as ovals.
  2. Optical Character Recognition (OCR) Prep: References for comparing printed text to identify voter selections without using barcodes.
  3. Independent Mapping: Associates voting targets with contests and options without relying on the voting system's proprietary target maps or encoded vote selections.
  4. Templates: Creates a template for each ballot style, combining many images of that style.
  5. Automated and Manual Mapping: Uses automated analysis and, when necessary, the browser-based TargetMapper application to establish accurate mappings.
  6. Proof Review: Generates visual proofs that reviewers can inspect to verify the accuracy of ballot mappings.

Phase 3: Extraction

  1. Hand-Marked Ballots: Examines the original ballot images and evaluates voter marks at the mapped target locations.
  2. BMD Ballots: Uses OCR to interpret the human-readable vote selections printed by ballot marking devices, without relying on barcodes or QR codes.
  3. Parallel Processing: Processes ballot images concurrently using scalable computing resources, allowing large elections to be analyzed efficiently.

Phase 4: Comparison

  1. Independent Comparison: Compares AuditEngine's independently extracted vote selections with the corresponding official CVR selections.
  2. CVR Vote Isolation: The official CVR votes enter the audit process at this stage, after independent extraction is complete.
  3. Discrepancy Classification: Identifies and categorizes differences between the two interpretations, including missing records, unmatched ballots, and vote-selection discrepancies.
  4. Repeated Scanned Ballots: Identifies batches scanned twice and adjusts reports.
  5. Reconciliation and Contest Reports: Produces detailed reports of discrepancies, ballot-level findings, and contest-level results, including differences that could affect election outcomes.

Optional Input Data

  • File Hash Values: We prefer to have a file of hash values we can check with the official ballot images and cast vote records.
  • Ballot Style Masters: Mapping can use these files to assist our own automated approach, particularly if the images have invisible ovals or an unusual layout, such as landscape grid layouts.
  • Results Summary Files: We can parse a variety of files providing the aggregated results, which may differ from the CVR.
  • Digital Poll Tapes: Polling place reports generated by the Election Management System, as a cross-check.
  • Verification Scans: Batches of ballots can be scanned manually and provided as verification scans, for comparison with the voting system's batches.

More on each phase: How AuditEngine Works, in the AuditEngine documentation.

How the Voting System and AuditEngine Work Together

The diagram illustrates two separate processes.

Voting System (upper portion): The voting system scans paper ballots, produces digital ballot images, interprets vote selections, and generates cast vote records (CVRs). The CVRs are used to tabulate the official election results.

AuditEngine (lower portion): AuditEngine receives the ballot images and CVR files, analyzes ballot layouts, and independently extracts vote selections from the images. During this independent processing, the official CVR vote selections remain separate.

Only after independent extraction is complete does AuditEngine compare its findings with the official CVRs. Differences are classified and reported for further examination, including human review when appropriate.

AuditEngine High level flow chart.png

Audit Projects and Results

AuditEngine Resources and Documentation

The AuditEngine.org front page

AuditEngine Website

Technical Documentation

AuditEngine Application

How you can help

AuditEngine depends on collaboration among election officials, technical contributors, researchers, and members of the public. Volunteers can help obtain election records, process ballot images, review audit results, and investigate discrepancies. The roles below describe ways to participate and the experience required. To help, complete the form: SignUp

Role Responsibilities Experience Needed
Public Advocate Encourage election officials to preserve ballot images and make them available for independent review. None, but may need attorneys to assist when officials claim that ballot images and cast-vote record files are not public records or cannot be made available for review.
Requestor Submit public-records requests for ballot image archives and CVRs. Limited technical expertise, but requires knowledge of public records requests and the desired file types, flash drives, and zip files. Assistance from attorneys may be required if officials resist.
Uploader Organize and upload election data into AuditEngine. Limited technical expertise, no legal background required.
Observer Monitor audit progress and review publicly available findings. Common sense
Auditor Configure audit jobs and supervise processing. Moderate technical expertise, knowledge of ballot formats and program settings.
Mapper Identify ballot targets and associate them with contests and options using TargetMapper. Detail-oriented and able to use a mouse.
Reviewer Verify mapping proofs and examine reported discrepancies. Must be detail-oriented.
Adjudicator Evaluate disputed ballot interpretations according to applicable election rules. Common sense required.
Subscriber Follow audit projects and review published reports. None required.

Historical Reports and Announcements

2020 General Election Audit Reports

AuditEngine's 2022 reports on the 2020 General Election in Bartow County, GA, Fulton County, GA, and Dane County, WI.

AuditEngine Case Study Report

This 2021 case study examines AuditEngine's analysis of elections in Collier, Volusia, and St. Lucie Counties, Florida. The report compares AuditEngine's interpretations with official voting-system records and examines cases in which the two systems disagreed.

Historical introduction: 2021 Florida case study and videos

The first video is a good introduction to the case study:

AuditEngine Frequently Asked Questions

We have started to compile the questions that tend to asked on this page: AuditEngineFAQs

AuditEngine Press and Announcements

Summary of articles submitted for this project:

The following archive contains AuditEngine reports, correspondence, research, announcements, and supporting documents, listed from newest to oldest.

Discussion List

See ListServe for all email discussion lists and to add more.

View MD Edit MD Attach