Direct link to this page: https://copswiki.org/Common/AuditEngine
- Audit Engine was first announced and demonstrated at the National Election Integrity Conference in Berkeley, CA on Oct 6, 2019, which was sponsored by the National Voter Rights Task Force. You can watch a video of the presentation in the links below.
is a cloud-based service which will perform a full-ballot-image audit for any election that produces ballot images.
We hope that this service will become the go-to standard for fully reviewing any election conducted on hand-marked paper ballots and then scanned by popular equipment.
Initialy, Audit Engine
will support ES&S and Dominion ballots and election data.
How It Works
Getting the data together
First, the data must be assembled for use by Audit Engine
. Most of this is already produced and available by Election Officials.
- Equipment: Election officials will use equipment that can produce full ballot images, and configure that equipment to produce and preserve the full ballot images.
- We are working to establish a full-ballot image securty standard which is based on the concept of "Trusted Systems" to ensure that we can trust that the ballot images are an accurate representation of the paper ballots.
- Full Ballot Images: Election officials will then upload these full-ballot images to a posting service that provides trusted timestamps, like Sharefile.com
- Cast Vote Record File: Also, officials will produce a cast-vote-record file CVRF which preferrably lists the results of their interpretation of the vote on each ballot. We support the ES&S format and the Dominion format. (The system can also be run without the CVRF if you want independent results.)
- Election Information File: The EIF lists of all contests, contests options, and the full-text descriptions found on the ballots of yes/no question type contests in all languages supported. This information is commonly provided to voters in the form of the Sample Ballot. This file will also provide the contest names as found on the ballots, and also the names used for each contest in the CVRF. This is a spreadsheet file in .xlsx format with the format we specify.
- Results Summary: The system must also be provided with the official (or unofficial) results that should match the totals from the cast-vote-records file, but may not if any ballots have been suppressed for privacy concerns. The total number of ballots suppressed must also be provided. For some systems, we can scrape the data from the official election website report.
- Poll List: We are planning also to support comparison with the "Poll List" which provides the totals of all the voters who voted so we can check that all the ballots are represented in the ballot images set. This is an area that needs further standardization.
- Hash Codes File: This file provides the SHA256 hash for each file uploaded to the posting service.
- Computer Code Verification File: This is at present a proposal to make sure we can trust the scanner hardware. See Securing Digital Ballot Images to Enable Auditing
- Please note that we do not require that the styles of the ballots be provided, which normally is used internally by election equipment vendors to determine the location of the ovals or "targets" on the ballot. Instead, Audit Engine reads the ballots and learns the styles as it goes.
Creating an Account
- Registration: You must register on the https://AuditEngine.org website to create a user identity
- Payment Method: You must create a payment method that we can use to access funds for the cost of the runs. We accept credit cards or we can invoice you after credit approval.
Creating a Job
- Name the Job: Create a job with a name for future reference. You must include the district, election date, and election type to distinguish it from other jobs you (and others) might submit.
- Image Files:
- Direct Uploading: If the image files are not too large and numerous, you can upload them directly to our site. We recommend this only if you have one or two ZIP files of ballot images with fewer than about 50,000 ballots.
- Posting Service Uploading: This is our preferred option and we recommend that you use the file posting service Sharefile.com. Please split the images files into zip files that are definitely less than 10GB in size and preferrably about 5GB or less. We have utilities that can help you split folders with many ballots to separate zipped folders for uploading that will also respect the underlying structure of the ballot archives. Sharefile.com has a very convenient uploading capability that rarely times-out for large files. Nevertheless, it is best to keep them small (maybe around 2GB) as many things can happen on your local LAN and internet link to interrupt the process. Smaller chunks are easier to incrementally complete. Once you get them uploaded, you need only share the link to the Sharefile folder in the job and Audit Engine will access all the ballot image zip files automatically. If you specify a Sharefile.com folder, you will click upload, and we will transfer the files to our site or access them from the Sharefile.com account as needed.
- Cast Vote Records File(s) (CVRF): The cast-vote-record files (CVRF) should be either CSV format (character separated data) or .xlsx format. Please compress using ZIP. Please limit the size of any single .xlsx file to no more than 99,999 records (100,000) lines by splitting into several files. Each of these should be separately zipped. If you are using Sharefile.com uploading, then they can be provided to the same folder containing the image archive ZIP files.
- Election Information File: We also require an Election Information File which provides the contest names, options, and the full-text descriptions found in each language as an .xlsx file.
- Results Summary: The system requires the official or unofficial results eiher in CSV, .xlsx format, or as a link to a web-page report, if we support that format.
- Adjudications File: After you have completed your first run, you may want to also submit and adjudications files, which essentially amends the CVRF to reflect the review of voter intent from the ballot images on any ballots that are a concern.
- Job Controls: You can limit the review to given ranges of ballot image numbers or only to certain precincts.
Election Information File
This file deals with the inconsistencies between various presentations of the same contests and options that are on the ballot. This file is not a standard file from election systems today, but it can be constructed from other data sources, such as the sample ballot, CVR File, Summary report, etc.
| Column Name
|| A brief yet correct and distinctive contest name. These are in the order they are presented in the Election Summary Report and the CVR file, typically.
|| The maximum number of votes in this contest. 1 is assumed if not provided.
|| The core option names without embellishments of candidate numbers or party designations. Each option is placed on a separate line with repeated Official_Contest_Name. If the Official_Contest_Name is not provided, the last Official_Contest_Name provided on a previous line is used.
|| Typically three-letter party designations such as REP, DEM, LBT, GRN, etc or NON or NP for non partisan that relate to this option name.
|| The contest name as found in the Cast Vote Record (CVR) file. These contest names SHOULD be distinct and not repeated within the CVR file. If they are the same as the Official_Contest_Name, it need not be provided.
|| The option (i.e. candidate) names as found in the CVR file, on the same lines as the options as provided in the Official_Options column. If the option name is the same as in the Official_Options, then it need not be listed.
|| The contest title as actually found on the ballot including all embellishments that are actually written, such as "Vote for 1". If it is exactly the same as the Official_Contest_Name, then it need not be listed.
|| The core option name as found on the ballot, on the same line as the same option in the Official_Options column. If it is exactly the same as the Official_Options, then it need not be listed. (The options on the ballot may be in a different order than shown here.)
|| This is the exact description of question-type (Yes/No) contests as found on the ballot. These should be entered with embedded newlines to reflect how it appears on the ballot.
|| The contest name as found on the ExpressVote ballot summary (for ES&S districts only) or any other BMD that creates a ballot summary card. These are limited to 51 characters (ES&S). If it is exactly the same as the Official Contest Name, then it need not be listed. (optional column)
|| The option names as printed on the ES&S ExpressVote ballot summary lined up with the options names in the other columns. If the option name is the same as in the Official Options, then they need not be listed. (optional column)
Launching a Job
- Once your job has been defined, the job can be launched. There are two phases:
- Phase 1:
- The first phase of the process will inspect the job specification and the input files for format and consistency.
- Consistency checks include comparing the number of ballot images with the number of cast-vote records in the CVRF files and that the contests as provided in the EIF can be found in the CVRF.
- Phase 2:
- After the initial inspections of Phase 1, you can launch the job.
- Audit Engine will visually inspect the ballots and learn the format of ballots that have the same style. Each style has either all or a subset of the contests in the election, and in each style, the location of targets for a given contest option will be in the same location.
- Audit Engine will detect marks on the ballots and create an comprehensive database containing information for each style and each ballot.
- During this phase, the work will be split up among many separate virtual machines in the cloud-based computing services, which can provision 10s of thousands of virtual machines that run in parallel in massive data centers. This will allow us to finish your job in minutes instead of many hours or days if run on a single machine.
- Phase 3:
- After the processing is complete, Audit Engine provides a comprehensive report, and allows interactive inspection of individual ballots and contest to resolve voter-intent in difficult cases.
- The Adjudications File can then be prepared for future runs.
Summary of articles submitted (Add | All):
Number of topics: 8
- (M1940) 2019-12-06 Uniform Audit Report -- Ray Lutz, Citizens Oversight,
- (M1938) 2019-11-30 The Four Fatal Flaws of RLA Audits -- Ray Lutz, Citizens Oversight,
- (M1936) 2019-11-11 Securing Digital Ballot Images to Enable Auditing -- Ray Lutz, Citizens Oversight,
- (M1931) 2019-11-04 Earth Summit Presentation: Elections: Stop Scams with Oversight and Audits -- Ray Lutz, Citizens Oversight,
- (M1928) 2019-10-05 AuditEngine Announcement at 3rd National Election Integrity Conference, Berkeley -- Ray Lutz, Citizens Oversight,
- (M1917) 2019-08-19 Ballot Image Samples -- Ray Lutz, Citizens Oversight,
- (M1913) 2019-08-12 Comprehensive Risk Estimation of Election Audits -- Ray Lutz, Citizens Oversight,
- (M1895) 2019-03-17 Letter to OC ROV requesting Ballot Images -- Ray Lutz, Citizens Oversight,
See List Serve
for all email discussion lists and to add more.