Audit Engine

Direct link to this page:
Share Button

Latest News


Audit Engine is a cloud-based service which will perform a full-ballot-image audit for any election that produces ballot images.

We hope that this service will become the go-to standard for fully reviewing any election conducted on hand-marked paper ballots and then scanned by popular equipment.

Initialy, Audit Engine will support ES&S and Dominion ballots and election data.

How It Works

Getting the data together

First, the data must be assembled for use by Audit Engine. Most of this is already produced and available by Election Officials.
  • Equipment: Election officials will use equipment that can produce full ballot images, and configure that equipment to produce and preserve the full ballot images.
    • We are working to establish a full-ballot image securty standard which is based on the concept of "Trusted Systems" to ensure that we can trust that the ballot images are an accurate representation of the paper ballots.
  • Full Ballot Images: Election officials will then upload these full-ballot images to a posting service that provides trusted timestamps, like
  • Cast Vote Record File: Also, officials will produce a cast-vote-record file CVRF which preferrably lists the results of their interpretation of the vote on each ballot. We support the ES&S format and the Dominion format. (The system can also be run without the CVRF if you want independent results.)
  • Election Information File: The EIF lists of all contests, contests options, and the full-text descriptions found on the ballots of yes/no question type contests in all languages supported. This information is commonly provided to voters in the form of the Sample Ballot. This file will also provide the contest names as found on the ballots, and also the names used for each contest in the CVRF. This is a spreadsheet file in .xlsx format with the format we specify.
  • Results Summary: The system must also be provided with the official (or unofficial) results that should match the totals from the cast-vote-records file, but may not if any ballots have been suppressed for privacy concerns. The total number of ballots suppressed must also be provided. For some systems, we can scrape the data from the official election website report.
  • Poll List: We are planning also to support comparison with the "Poll List" which provides the totals of all the voters who voted so we can check that all the ballots are represented in the ballot images set. This is an area that needs further standardization.
  • Hash Codes File: This file provides the SHA256 hash for each file uploaded to the posting service.
  • Computer Code Verification File: This is at present a proposal to make sure we can trust the scanner hardware. See Securing Digital Ballot Images to Enable Auditing
  • Please note that we do not require that the styles of the ballots be provided, which normally is used internally by election equipment vendors to determine the location of the ovals or "targets" on the ballot. Instead, Audit Engine reads the ballots and learns the styles as it goes.

Creating an Account

  • Registration: You must register on the website to create a user identity
  • Payment Method: You must create a payment method that we can use to access funds for the cost of the runs. We accept credit cards or we can invoice you after credit approval.

Creating a Job

  • Name the Job: Create a job with a name for future reference. You must include the district, election date, and election type to distinguish it from other jobs you (and others) might submit.
  • Image Files:
    • Direct Uploading: If the image files are not too large and numerous, you can upload them directly to our site. We recommend this only if you have one or two ZIP files of ballot images with fewer than about 50,000 ballots.
    • Posting Service Uploading: This is our preferred option and we recommend that you use the file posting service Please split the images files into zip files that are definitely less than 10GB in size and preferrably about 5GB or less. We have utilities that can help you split folders with many ballots to separate zipped folders for uploading that will also respect the underlying structure of the ballot archives. has a very convenient uploading capability that rarely times-out for large files. Nevertheless, it is best to keep them small (maybe around 2GB) as many things can happen on your local LAN and internet link to interrupt the process. Smaller chunks are easier to incrementally complete. Once you get them uploaded, you need only share the link to the Sharefile folder in the job and Audit Engine will access all the ballot image zip files automatically. If you specify a folder, you will click upload, and we will transfer the files to our site or access them from the account as needed.
  • Cast Vote Records File(s) (CVR Files): The cast-vote-record (CVR) files should be in either CSV format (character separated data) or .xlsx format. Please compress using ZIP. Please limit the size of any single .xlsx file to no more than 99,999 records (100,000) lines by splitting into several files. Each of these should be separately zipped. If you are using uploading, then they can be provided to the same folder containing the image archive ZIP files.
  • Election Information File (EIF): We also require an Election Information File which provides the contest names, options, and the full-text descriptions found in each language as an .xlsx file. The format of this file is found below.
  • Ballot Options File (BOF): It is sometimes required that you also provide the "Ballot Options File" which provides two columns, "official_option'
  • Results Summary: The system requires the official or unofficial results eiher in CSV, .xlsx format, or as a link to a web-page report, if we support that format.
  • Adjudications File: After you have completed your first run, you may want to also submit and adjudications files, which essentially amends the CVRF to reflect the review of voter intent from the ballot images on any ballots that are a concern.
  • Job Controls: You can limit the review to given ranges of ballot image numbers or only to certain precincts.

Election Information File

This file deals with the inconsistencies between various presentations of the same contests and options that are on the ballot. This file is not a standard file from election systems today, but it can be constructed from other data sources, such as the sample ballot, CVR File, Summary report, etc. This is specific to ES&S files.

Column Name Data
official_contest_name Str A brief yet correct and distinctive contest name. These must be an exact replacement for the Cast Vote Record file header.
original_cvr_header Str This column provides the exact header from the CVR files for reference. This column can be created by copying the header line and pasting it into the spreadsheet using "paste-special" and "transpose". This column will also include the first identifying columns, such as "Cast Vote Record", "Precinct", and "Ballot Style"
ballot_contest_name Str The contest title as actually found on the ballot including all embellishments that are actually written, such as "Vote for 1". If it is exactly the same as the Official_Contest_Name, then it need not be listed.
vote_for Int The maximum number of votes in this contest. 1 is assumed if not provided.
writein_num Int The number of "Write-in:" options provided on the ballot for this contest. If left blank, 1 is assumed. 0 means no write-in lines are provided.
official_options Str List of official option names, separated by commas. The order of the options provided here may not match the order on the ballot.
description Str This is the exact description of question-type (Yes/No) contests as found on the ballot. These should be entered with embedded newlines to reflect how it appears on the ballot.

Ballot Options File (BOF)

A BOF is required only if the options as printed on the ballot differ substantially from the official_options as provided in the EIF. This file includes three columns, as follows. This file need only provide records for those options that need to be substituted.

Column Name Data
official_contest_name Str The same contest name as provided in the CVR as official_contest_name.
official_option Str A ballot option as listd in official_options in the EIF
ballot_option Str The option as listed on the ballot, exactly as provided, including newlines. The ballot options are sometimes very different, particularly if they list a candidate pair, such as for Governor and Lieutenant Governor, or President and Vice President, listed on separate lines.

Launching a Job

  • Once your job has been defined, the job can be launched. There are several phases:
  • Phase 1 -- Uploading files:
    • The first phase of the process includes uploading ballot images, CVR files and EIF files.
    • The system will inspect the job specification and the input files for format and consistency.
    • Consistency checks include comparing the number of ballot images with the number of cast-vote records in the CVR files and that the contests as provided in the EIF can be found in the CVR.
  • Phase 2 -- Generate Style Templates:
    • After the files have been uploaded as provided in Phase 1, Style Templates can be generated.
    • Audit Engine will visually inspect the ballots and learn the format of ballots that have the same style. Each style has either all or a subset of the contests in the election, and in each style, the location of targets for a given contest option will be in the same location.
    • Style Template generation provides the location of each contest option target and associates it with the correct CVR option.
    • This is a critical phase of processing. At the end of this process, a set of ballot images which are marked up to show the location of each target and the associated official_contest_name and the official_ballot_option, and these images must be carefully inspected to determine that the correct contest and option is associated with the correct target.
    • After you have inspected these images, then you can proceed to vote extraction.
  • Phase 3 -- Vote Extraction and reporting:
    • After generation of Style Templates and approval of the contest mapping, you can launch the extraction phase.
    • Audit Engine will detect marks on the ballots and create an comprehensive database containing information for each style and each ballot.
    • During this phase, the work will be split up among many separate virtual machines in the cloud-based computing services, which can provision 10s of thousands of virtual machines that run in parallel in massive data centers. This will allow us to finish your job in minutes instead of many hours or days if run on a single machine.
  • Phase 4 -- Reporting and Adjudication:
    • After the processing is complete, Audit Engine provides a comprehensive report, and allows interactive inspection of individual ballots and contest to resolve voter-intent in difficult cases.
    • The Adjudications File can then be prepared for future runs.

Summary of articles submitted (Add | All):

Number of topics: 9

Discussion List

See List Serve for all email discussion lists and to add more.

Project Form edit

Project Name Audit Engine
Project Description Full-ballot image auditing service
Project Founder Ray Lutz
Project Curator Ray Lutz
Project Type Issue Oversight
Project Parents Election Integrity
Related Keywords Open Ballot Initiative
Project Status Hot
Publish Status Published
Thumbnail Link
Forum Link
List Serve Topic
Topic revision: r7 - 2020-01-03, RaymondLutz
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Cops? Send feedback