You are here: Cops>Common Web>AuditEngine (05 Mar 2025, RaymondLutz)Edit Attach
AuditEngineLogo.png

Audit Engine

Direct link to this page: https://copswiki.org/Common/AuditEngine
Share Button

Hide Table of Contents 

Latest News

Audit Engine Case Study Report

Audit Engine was used to audit three counties in Florida: Collier, Volusia, and St. Lucie Counties. Learn about audit engine and its unbelievable accuracy, which is better than the official voting systems by 93% on contests where we have located a disagreement (excluding corrupted ballot images). Read the report and watch four videos on this page.
START HERE IF YOU ARE NEW TO AUDIT ENGINE

JUMP DIRECTLY TO THE FIRST VIDEO regarding the Case Study, which is a great introduction:

Audit Engine Frequently Asked Questions

We have started to compile the questions that tend to asked on this page: Audit Engine FAQs

Audit Engine Press and Announcements

Simplified Block Diagram

The following block diagram should help your understanding of how Audit Engine works.

The upper block shows the operation of the voting system, which scans hand-marked paper ballots and creates ballot images, which are then analyzed to extract the vote and create cast-vote records. Those can be totalled up to create the election results.

The lower block shows how Audit Engine accepts ballot images, and then performs style analysis, to provide a map of the contests and options to where the bubbles are. The ballots are then fully processed to extract the vote and then these cast vote records are compared with the official records. Discrepancies can then be reviewed by hand if the election is close enough to require that.

AuditEngine High level flow chart.png

How you can help

We need help to apply Audit Engine so as to help improve our confidence in our elections. Can you help? Please complete the form at this link: Sign Up

Role What you will do Technical Expertise Required
Public Advocate Contact public officials and insist that election officials should PRESERVE ballot images, and make them available for public review on a routine basis. None, but may need attorneys to assist when officials claim that ballot images and cast-vote record files are not public records or cannot be made available for review.
Requestor Contact election officials and request that they post ballot image archives and cast-vote record files (See details below for format and various options) Limited technical expertise, but requires knowledge of public records requests and the desired file types, flash drives, and zip files. Assistance from attorneys may be required if officials resist.
Uploader After receiving ballot image archives and cast-vote records, create an "Election" and upload these files to Audit Engine Limited technical expertise, no legal background required.
Observer Stakeholders and those with specific expertise can participate on the observers panel. Common sense
Auditor Once the files have been uploaded, any additional files may need to be created, and settings established for successful style analysis and vote extraction. Moderate technical expertise, knowledge of ballot formats and program settings.
Mapper One phase of the audit process involves a manual process we call "Target Mapping" and it uses a browser-based tool called the Target Mapper. Once the styles are understood, there is an opportunity to reduce the number of styles if some are equivalent. Mapping each style means locating the "targets", i.e. the ovals that are marked by voters, and pairing them up with the contest and option of that target. Details oriented and able to use a mouse.
Reviewer As audit files are created, members of the public should review redline proofs to insure that the style analysis was completed correctly, and review the results of the comparison Must be detail-oriented.
Adjudicator If there are discrepancies, an adjudication team will review them and enter voter intent or indicated votes as appropriate for the state of interest. Common sense required.
Subscriber Any member of the public can watch public audits as subscribers, and review the data files produced, redline proofs, discrepancies, adjudications, and results. None required.

How It Works

Getting the data together

First, the data must be assembled for use by Audit Engine. Most of this is already produced and available by Election Officials.

  • Ballot Image Creation:
    • Modern voting equipment used in most districts in the U.S. create ballot images as they process the ballots.
    • It is essential that this equipment be set so the ballot images are not deleted after they are used to extract the vote.
    • These images are then transferred to the Election Management System (EMS)
    • The EMS can export the ballot images for use by Audit Engine. These should be placed into ZIP archives with about 30,000 to 50,000 ballots per archive.

  • Election Information File (EIF):
    • Our system requires additional information regarding the information actually printed on the ballots.
    • The EIF lists
      • all contest names, as used in the CVR, as printed on hand-marked paper ballots, and as printed on BMD ballots
      • the options in each contests, the full-text descriptions of yes/no contests.
      • the number of write-ins, and the vote-for number.
    • This information is available from sample ballots and can be assembled prior to the election.
    • This is a spreadsheet file in .xlsx or .csv format.
    • The exact format of this file is provided below.
    • In rare case, we also need the Ballot Options File (BOF) which lists the actual text of each option if they differ from the official CVR text.

  • Cast Vote Record (CVR):
    • Officials can produce cast-vote-record files (CVR) which lists the results of their interpretation of the ballots.
    • Audit Engine currently supports the ES&S format and the Dominion format.
    • If any ballots are suppressed for privacy reasons, the total number of ballots suppressed and their vote totals should be provided.
    • Audit Engine can also be run without the CVR but some additional information is required:
      • Results Summary: The official (or unofficial) results that should match the totals from the cast-vote-records file, but may not if any ballots have been suppressed for privacy concerns. For some systems, we can scrape the data from the official election website report.
      • Style to Contests File: This is a spreadsheet which can be generated later in the process that lists the contests included in each style. This can be generated by inspecting the ballots or it can be provided by elections officials.

EXPORTING: Step-by-step instructions

To view googledoc for comments, please click here

UPLOADING: Step-by-step instructions

To view googledoc for comments, please click here

Creating an Account

  • Registration: You must register on the https://AuditEngine.org website to create a user identity
  • Payment Method: You must create a payment method that we can use to access funds for the cost of the runs. We accept credit cards or we can invoice you after credit approval.

Creating a Job

  • Name the Job: Create a job with a name for future reference. You must include the district, election date, and election type to distinguish it from other jobs you (and others) might submit.
  • Image Files:
    • Direct Uploading: We recommend that you upload your archives directly to our site using our browser interface, which has no limit to the size of each file uploaded.
    • Posting Service Uploading: In the future, we plan to provide transfers from your file posting service, such as Dropbox, Sharefile, Drive, etc.

  • Cast Vote Records File(s) (CVR Files):
    • A cast-vote record file (CVR) provides the result of the election as determined by the voting system.
      • A CVR can be broken down to the individual ballot (which is preferred) or it can provide results based on higher level groups, such as precincts or batches.
      • Audit Engine does not require the CVR file, and it does not rely on the CVR file in its own processing of the ballot images.
      • However, Audit Engine will make use of the CVR to a limited extent, if it is available, as follows:
        • To reduce the processing required in the initial pre-scan of the ballots.
          • If no CVR is provided, all ballot images are pre-scanned to extract style information from barcodes on hand-marked paper ballots, i.e. which contests are included on each ballot style.
          • If the CVR is provided with style for each ballot, then a full pre-scan of the ballots is not required. Use of this style information, if incorrect, will be discovered in the style templating and mapping process. Reliance on the style from the CVR does not change the vote extraction process (which is performed without reference to the CVR).
        • To locate discrepancies between the voting-system tabulation and the audit system.
          • Only after Audit Engine has fully extracted the vote, the voting system CVR is compared with that result to provide the discrepancy report.

    • Audit Engine supports voting-system cast-vote-record (CVR) files in vendor-defined formats and in the new "Common Data Format" as defined by NIST or variants.
      • ES&S Legacy Format:
        • This is a table-oriented format which results in a reasonable size and number of files.
        • Use CSV (character separated data) or .xlsx format.
        • Please limit the size of any single .xlsx file to no more than 99,999 records (100,000) lines by splitting into several files if needed.
        • Please compress each file using ZIP.
      • Dominion Legacy Format:
        • This is a CSV "tidy" format which has only a few columns and many records.
        • Each record contains fields that identify the data value, and one data value.
        • Please compress each file using ZIP.
        • This format has been generally used for batch or precinct level reports and not broken down by ballot.
      • Dominion JSON CVR format:
        • This format is similar to the NIST standard but is not an exact implementation.
        • It results in typically one JSON file per batch on a given tabulator.
        • For example, in the 2020 Primary in San Francisco, more than 13,000 files resulted.
        • All the files should be combined into a single ZIP file.
      • Other formats:
        • We can add other formats as needed.

  • Election Information File (EIF): Audit Engine generally also requires an Election Information File which provides the contest names, options, and the full-text descriptions found in each language as an .xlsx or CSV file. The format of this file is found below.
    • Ballot Options File (BOF): This is actually logically part of the EIF dataset but is only rarely needed when the text for ballot options differs from the official option. This might occur in gubernatorial or presidential contests when the option includes two names, one for governor and leiutenant governor, for example. But it may not be required even if that is the case so you may want to try it first without.

  • Results Summary: The system requires the official or unofficial results eiher in CSV, .xlsx format, or as a link to a web-page report, if we support that format. Even if the full CVR is provided, Audit Engine will compare with the final totals in the contests.

  • Adjudications File: After you have completed your first run, you may want to also submit and adjudications files, which essentially amends the Audit Engine result to reflect the review of voter intent from the ballot images on any ballots that are a concern.

  • Job Controls: You can limit the review to given ranges of ballot image numbers or only to certain precincts.

Election Information File

This file deals with the inconsistencies between various presentations of the same contests and options that are on the ballot. This file is not a standard file from election systems today, but it can be constructed from other data sources, such as the sample ballot, CVR File, Summary report, etc. NOTE: Since we are now using Target Mapper to perform computer-aided (rather than fully automated) mapping, many of the fields mentioned below are not required, and are noted in the comments as TM-NR, as not required when using Target Mapper.

Some elements of this data file are specifically to allow Audit Engine to process legacy CVR files.
  • In legacy ES&S CVR files:
    • the header may include non-unique names, and are processed based only on their order.
    • in this case, the original_cvr_header is provided as a starting point, and then the official_contest_name is derived typically by an analyst.

Column Name Data
type 		Description
official_contest_name str (Required) A brief yet correct and distinctive contest name. These must be an exact replacement for the Cast Vote Record file header. These can contain spaces and punctuation, but it is smart to limit it. Avoid newlines.
original_cvr_header str (Optional column) This column provides the exact header from the CVR files for reference. This column can be created by copying the header line and pasting it into the spreadsheet using "paste-special" and "transpose". This column will also include the first identifying columns, such as "Cast Vote Record", "Precinct", and "Ballot Style". If the initial columns differ from those, they should be set in the JOB settings file using the 'initial_cvr_cols' directive. (This column is not needed for non-ES&S systems.)
contest_id int (Optional) The numeric id of this contest
ballot_styles csv (Optional, TM-NR) comma separated values of ballot style_num values, for those styles where this contest is found. Leave blank if the contest is on all styles. Alternatively, the manual_styles_to_contests_table format can be used to map contests to styles.
sheet int (Optional) The sheet number where this contest is found, if a multi-sheet ballot is used. Here, sheet numbers start at 1.
ballot_contest_name str (Required, TM-NR) The contest title as actually found on the ballot including all embellishments that are actually written, such as "(Vote for 1)". If it is exactly the same as the Official_Contest_Name, then it need not be listed. In the case of bilingual ballots, it may work best to include only the English, but it must be the first portion of the text.
contest_name_num int (Optional, TM-NR) The number of lines of text used by the contest name. If not provided, the lines are counted in the 'ballot_contest_name' field. Used in cases when the ballot_contest_name has more lines than are used for comparison, such as in multi-lingual ballots. It is recommended that this fields be used. This is a critical value.
bmd_contest_name str (Optional) If BMDs are used, the this column should provide the exact string used in by the BMD for this contest in the vote summary
vote_for int (Required) The maximum number of votes in this contest. 1 is assumed if not provided.
writein_num int (Required, TM-NR) The number of "Write-in:" options provided on the ballot for this contest. If left blank, 1 is assumed. 0 means no write-in lines are provided.
official_options str (Required) List of official option names, separated by commas. The order of the options provided here is not required to match the order on the ballot. If these option names differ from what is actually printed on the ballot, then use a BOF file to define the exact text for each option (See details below).
description str (TM-NR) This is the exact description of referendum or question-type (Yes/No) contests as found on the ballot. These can be entered with embedded newlines to reflect how it appears on the ballot.
descr_num str (optional, TM-NR) The number of lines of text used by the description. If not provided, the lines are counted in the 'description' field. Used in cases when the description has more lines than are used for comparison, such as in multi-lingual ballots. The use of this field has been deprecated because it is better to use the 'auto_detect_descr_lines_reserve' directive, which will simply include all lines except for those reserved at the end in the OCR lines used in the comparison.
qualified_writeins str (Optional) List of official option names of qualified writeins. These are provided to BMDs to allow voters to vote for the official writeins and avoid random writeins that are not qualified.

Additional columns can be used as desired by the analyst and are ignored if not supported. Here is a blank spreadsheet with the columns initialized: Blank EIF Google Spreadsheet If a detailed CVR is provided, then most of the above fields are automatically generated when the starter EIF is generated.

Additional Notes about the EIF

FOR FULLY AUTOMATED MAPPING, the following applies. This method is now no longer being used and computer-aided mapping is used instead. However, these statements still do apply when processing BMD ballots as the exact strings of the contest names and option names must be provided.

The data in the EIF is extremely critical to get exactly right so that Audit Engine will be able to successfully map the contests to the OCR'd strings from the ballot. If the information is not exactly correctly, it can be found out and corrected during the mapping process, but it will take a lot more time and effort to do it then. Here are some areas which should be carefully checked.

  1. official_contest_name is the internal name for that contest. It is best if these are short to conserve space and yet easy for humans to distinguish from each other.

  2. ballot_contest_name and contest_name_num are critical to get correct. This is because Audit Engine uses these values to match the contest and then look for the options in "fully_joined" layout, i.e. where the contest title is in the same box on the ballot as the options. The actual ballot image should be reviewed to determine the number of lines used by the contest header. When counting lines, do not count blank lines.

    In the case of yes/no contests that have a description, it may be somewhat arbitrary where to split the contest name from the rest of the description. For example, if there is a title and a subtitle, it may be best to include both of these in the ballot_contest_name, particularly if the contest names are otherwise somewhat similar. "Constitutional Amendment I" does not differ much from "Constitutional Amendement II" and thus adding the subtitle can be a smart move. Another way to deal with this is to set 'question_contests_have_no_contest_name' to TRUE and then just put all the text in the descr.

  3. descr is the other component of the issue described above, and must not include additional lines that are also in the ballot_contest_name. descr_num has been deprecated, and instead you should use 'auto_detect_descr_lines_reserve' to set how many lines should be set aside at the bottom of the region for Yes and No options, usually 2. If you include the 'auto_detect_descr_lines_reserve' directive, then all the lines in the ocr'd text are matched with the descr except for those reserved at the end for the Yes/No options. It is not necessary to include all the lines of the description in the descr field, just enough to thoroughly distringuish it from all other descr fields, and the fuzzy matching algorithm attempts to match left-justified.

Ballot Options File (BOF)

A BOF is required only if the options as printed on the ballot differ substantially from the official_options as provided in the EIF, and now only for BMD ballots. This file includes (at least) three columns, as follows. This file need only provide records for those options that need to be substituted.

Column Name Data
type 		Description
official_contest_name Str The same contest name as provided in the CVR as official_contest_name.
official_option Str A ballot option as listed in official_options in the EIF
ballot_option Str (These now only needed for BMD options. The option as listed on the ballot, exactly as provided, including newlines. The ballot options are sometimes very different, particularly if they list a candidate pair, such as for Governor and Lieutenant Governor, or President and Vice President, listed on separate lines. If the ballot has separate lines, then they must be entered as separate lines in the BOF spreadsheet. This can usually be done by using CONTROL-ENTER. Please note: Audit Engine cannot support options with differing number of option lines for the same contest. If the line on the ballot has the party designation, it is good practice to include that in the BOF to enhance success of matching options. For example "Joe Blow (DEM)" can be entered as the ballot option while "Joe Blow" might be the official option.
ballot_option_num int (optional TM-NR) The number of lines of text used by the ballot_option. If not provided, the lines are counted in the 'ballot_option' field. Used in cases when the ballot_option has more lines than are used for comparison, such as in multi-lingual ballots. For a single contest, all ballot options must have the same number of lines.

Additional columns can be used as desired by the analyst and are ignored if not supported.t Here is a blank spreadsheet with the columns initialized: Blank BOF Google Spreadsheet

Processing an Election on Audit Engine

  • Register with the site and create a user name.
  • Once your job has been defined, the job can be launched. There are several phases:
  • Phase 1 -- Uploading files:
    • The first phase of the process includes uploading ballot images, CVR files and EIF files.
    • The system will inspect the job specification and the input files for format and consistency.
    • Precheck: Consistency checks include comparing the number of ballot images with the number of cast-vote records in the CVR files and that the contests as provided in the EIF can be found in the CVR.
  • Phase 2 -- Generate Style Templates:
    • After the files have been uploaded as provided in Phase 1, Style Templates can be generated.
    • Audit Engine will visually inspect the ballots and learn the format of ballots that have the same style. Each style has either all or a subset of the contests in the election, and in each style, the location of targets for a given contest option will be in the same location.
  • Phase 3 -- Target Mapping
    • We now use the helper browser app called Target Mapper. The user will select each style and using a mouse, locate the target for that option.
    • Similar styles can be copied and pasted, and the system will drop any contests not in the new style, and then the user can add any new contests.
    • Multiple users will be able to help perform the mapping by splitting up the range of styles.
    • At the end, the result is exported from Target Mapper.
    • The map is imported into Audit Engine, and it performs an additional consistency check.
    • Redline proofs are created in a styles report. These proofs must be inspected manually to insure accuracy.
  • Phase 4 -- Vote Extraction and reporting:
    • After generation of Style Templates and approval of the contest mapping, you can launch the extraction phase.
    • Audit Engine will detect marks on the ballots and create an comprehensive database containing information for each style and each ballot.
    • During this phase, the work will be split up among many separate virtual machines in the cloud-based computing services, which can provision 10s of thousands of virtual machines that run in parallel in massive data centers. This will allow us to finish your job in minutes instead of many hours or days if run on a single machine.
  • Phase 5 -- Comparison and Discrepancy Report
    • If a detailed CVR is available detailed to the individual ballot, then the compare-cvr (CMPCVR) process is started to compare all the records and generate a discrepancy report.
  • Phase 6 -- Reporting and Adjudication:
    • After the processing is complete, Audit Engine provides a comprehensive report, and allows interactive inspection of individual ballots and contest to resolve voter-intent in difficult cases.
    • The Adjudications File can then be prepared for future runs.

  • We are working to establish a full-ballot image securty standard which is based on the concept of "Trusted Systems" to ensure that we can trust that the ballot images are an accurate representation of the paper ballots.
  • In addition,
    • Digital Poll Tape Report: Audit Engine can perform a digital poll tape audit, which compares the "poll tape" from each of the polling place machines (in digital form) with the official report of election day and early voting use of those machines.
    • The "Poll List" which provides the totals of all the voters who voted so we can check that all the ballots are represented in the ballot images set. This is an area that needs further standardization.
  • Hash Codes File: This file provides the SHA256 or similar hash for each file uploaded to the posting service. When the files are uploaded to our cloud-based provider (AWS) they use a slightly different method and it is important to generate the hash values the same way. They generate the hash value by taking the MD5 hash value of smaller chunks, of say 8MB each, and then hashing that list.
  • Computer Code Verification File: This is at present a proposal to make sure we can trust the scanner hardware. See Securing Digital Ballot Images to Enable Auditing

Summary of articles submitted (Add | All):

Number of topics: 56

Discussion List

See List Serve for all email discussion lists and to add more.

Project Form edit

Project Name Audit Engine
Project Description Full-ballot image auditing service
Project Founder Ray Lutz
Project Curator Ray Lutz
Project Type Issue Oversight
Project Parents Election Integrity
Related Keywords Open Ballot Initiative
Project Status Hot
Publish Status Published
Thumbnail Link
Forum Link
List Serve Topic
I Attachment Action Size Date Who Comment
A-Logo.PNGPNG A-Logo.PNG manage 10 K 03 Oct 2020 - 01:48 Raymond Lutz  
AuditEngine High level flow chart.pngpng AuditEngine High level flow chart.png manage 191 K 05 Nov 2022 - 18:33 Raymond Lutz Audit Engine high-level flow chart
AuditEngine-CreateAccount.jpgjpg AuditEngine-CreateAccount.jpg manage 42 K 29 Jun 2020 - 03:29 Raymond Lutz Sign up with Audit Engine
AuditEngineLogo.pngpng AuditEngineLogo.png manage 70 K 04 Oct 2020 - 18:54 Raymond Lutz  
ESS_ Exporting Guide for <a class="foswikiCurrentTopicLink" href="/Common/AuditEngine">Audit Engine</a>.pdfpdf ESS_ Exporting Guide for AuditEngine.pdf manage 1 MB 14 Feb 2023 - 17:20 Raymond Lutz  
LogInDialog.jpgjpg LogInDialog.jpg manage 25 K 29 Jun 2020 - 03:31 Raymond Lutz Log in to Audit Engine website
Press Release <a class="foswikiCurrentTopicLink" href="/Common/AuditEngine">Audit Engine</a> Available for 2020 General Election -- 2020-10-02.pdfpdf Press Release AuditEngine Available for 2020 General Election -- 2020-10-02.pdf manage 288 K 04 Oct 2020 - 23:23 Raymond Lutz 2020-10-04 Press Release Announcing Audit Engine for 2020 General Election
Sending Election Data to <a class="foswikiCurrentTopicLink" href="/Common/AuditEngine">Audit Engine</a>.pdfpdf Sending Election Data to AuditEngine.pdf manage 233 K 14 Feb 2023 - 17:26 Raymond Lutz  
Edit  | Attach | Print version | History: r28 < r27 < r26 < r25 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r28 - 05 Mar 2025, RaymondLutz
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Cops? Send feedback