Cops Canvass Report Audit Log Discrepancies

This is a section of the Cops Canvass Report. See also San Diego County Canvass Procedure

Audit Log Discrepancies

COPs requested and received the central tabulator audit log. This is perhaps the most important means for oversight groups to track the election and check on the reliability of the central tabulator, elections officials, and staff. However, it is not sufficient for these purposes as there are limited data in the log. However, the log could be easily enhanced (perhaps by just making a simple setting requesting more data) without any additional cost, since the audit log is never reduced to hardcopy form.

We have the following observations regarding the audit log:

• The number of cards cast is noted in the log for precincts scanned on election night. No counts are provided for VBM ballots, Provisionals, and Touch Screen entries.
• The audit log does not provide any vote information, only the count of ballots, and then only for election night scanning.
• The Registrar does not have full documentation for the audit log, such as what all entries mean and any options or configuration settings in terms of what is provided in the audit log. (Their reply to Question C00021-21.1)
• The Registrar has no procedures for review of the log. (Their reply to Question C00021-21.2) They said the logs are reviewed by IT staff to verify that an action has taken place such as a batch has been closed and the correct number of cards cast has been captured. But there was no documentation regarding when these reviews took place, if in fact they ever did.
• There is no outside review of the audit log, except the one you are reading.
• Details of the Audit Log are available in COPs document C00019: Letter to RegistrarOfVoters regarding election processing

DRE Machine counts illegally uploaded.

There are two cases of transactions of the following form:

1225838065	Connection Opened on 172.16.1.20 (Port:1027)
1225838074	Uploaded VCenter 17000 Machine 1 DLVersion 1 Copy 0  on 172.16.1.20 (Port:1027) SN 223582 #Ballots 12 12 Precincts
1225838074	Connection Closed on 172.16.1.20 (Port:1027)

1225838148	Connection Opened on 172.16.1.20 (Port:1029)
1225838160	Uploaded VCenter 17000 Machine 0 DLVersion 1 Copy 0 on 172.16.1.20 (Port:1029) SN 208790 #Ballots 20 18 Precincts
1225838160	Connection Closed on 172.16.1.20 (Port:1029)

Ro V said (C00021-21.5) this represented voted contest counts for the ballot types at vote center 1700 (early voting location at Ro V office), from machine number 1, from the 1st copy of that touch screen card, which is being read from a touch screen connected to the GEMS server over a secured isolated network on IP address 172.16.1.20 (port: 1027) SN 223582., 12 ballots in 12 different precincts, and the next was 20 ballots in 18 precincts. (Total of 32 ballots).

We understood that the Ro V would be converting all touchscreen ballots to durable paper ballots in the election, so we are confused as to why this was occurring, and if the log is correct, this is a violation of their agreement with the So S regarding the method of conducting elections.

Audit Log Incomplete

In numerous places throughout the log, we see the transactions for starting and stopping the server. For example, at the start of the log, we see:

1223283632	Starting AV Server

However, in numerous places (ten), there is no notation that the server was stopped and it is started, or it is started and there is no “Stopping” entry. This makes us worry that the log is incomplete. In addition, during those intervals, there is significant time elapsed in the unaccounted time period. For example, (NOTEs were added):

1224517959	Connection Closed on COM3
1224517962	Waiting on COM3
1224518202	Starting AV Server  (NOTE NO “STOPPING SERVER”)

1224519519	Stopping AV Server
1224571786	Waiting on COM3     (NOTE NO “STARTING SERVER”)
1224571786	Waiting on COM2

1224586245	Waiting on COM8
1224587341	Starting AV Server  (NOTE NO “STOPPING SERVER”)

1225728881	Waiting on COM1
1225729028	Waiting on COM1
1225817009	Starting AV Server  (NOTE NO “STOPPING SERVER”)

1225837932	Waiting on COM2
1225838035	Starting AV Server  (NOTE NO “STOPPING SERVER”)

1223564989	Stopping AV Server
1224490449	Waiting on COM5     (NOTE NO “STARTING SERVER”)
1224490449	Waiting on COM3

1224519519	Stopping AV Server
1224571786	Waiting on COM3     (NOTE NO “STARTING SERVER”)
1224571786	Waiting on COM2

1225708019	Stopping AV Server
1225712288	Waiting on COM1     (NOTE NO “STARTING SERVER”)
1225712411	Connection Opened on COM1

1225817574	Waiting on COM1
1225817579	Stopping AV Server
1225819478	Waiting on COM1     (NOTE NO “STARTING SERVER”)
1225819489	Waiting on COM2

1228227206	Connection Closed on COM8
1228227210	Waiting on COM8
(END OF THE LOG)                (NOTE NO “STOPPING SERVER”)

The Ro V said (C00021-21.7):

This question assumes an order of processing that is erroneous. As stated above, the AV Server Log is a continuous log file that contains events.
Events happen before the AV server is started.
Events happen while the AV server is running.
Events happen after the AV server is stopped.

And also said to the question: Why does the log have this inconsistency? (C00021-21.7.1):

Because it is an Event driven log file. The events themselves can be driven by a myriad of other varying events.

We find these answers unsatisfactory. The audit log should be a complete accounting of events. Also, we find it interesting that the Ro V can even answer this since they have documentation regarding the meaning of the Audit Log (Their reply to Question C00021-21.1)

• We asked (C00021-21.7.2): It is possible that some transactions have been deleted from the log, when the central tabulator was still running, and then shut down later, with the “Stopping AV Server” also deleted? And they said: "We are unable to answer this. Perhaps Premier could answer it." But we were never given a contact at Premier Election Solutions (formerly Diebold Systems).

• We asked about the fact that we did not notice any “Clearing Vcenter” entries prior to the initial accumulation of votes. How do we know that the accumulators were zeroed?

Ro V said:

The initial system is cleared at the beginning of the mail ballot processing, at which time a zero ballots cast report is run. From this point forward, ballots are added to the system. It should be noted that the “clearing Vcenter” entry is for an OS memory card that is used to transfer ballots cast at the polling places (contained in ballot cartons) into GEMS on election night and during the canvass as reruns.

• We asked for the log from the earliest date it was maintained. ("If the log includes such an entry at an earlier date, please forward the complete log, including the clearing entry(ies).") However, we were never provided with the complete audit log. Therefore, we must assume it does not exist. The Ro V does not maintain an audit log for the complete election process. This is a severe and possibly negligent or criminal problem.

Audit Log Documented Software Crashes

There were a number of errors and system crashes noted in the log:

1225838402	Error Internal Error
File: DownloadAVSPort.cpp, Line 665, Date: May 13 2005 on COM8

And:

1225840829	Error Internal Error

And, and invalid Password detected:

1225850242	Connection Opened on COM2
1225850242	Error "INVALID PASSWORD" on COM2
1225850244	Connection Closed on COM2

Ro V officials were unable to explain any of these crashes and error conditions. They referred us to Premier Election Solutions but we were never given a contact (and we did ask for one).

Audit Log Documents Numerous Re-scans

There are a number of times (133 total) where the totals were cleared and a precinct re-run, starting Nov. 20 and occurring through Dec. 2, 2008, as listed below.

1227198273	Clearing VCenter 5890:0
1227198287	Clearing VCenter 14660:0
1227198295	Clearing VCenter 14810:0
1227198304	Clearing VCenter 14840:0
1227198313	Clearing VCenter 14860:0
1227198320	Clearing VCenter 14870:0
      ...
(121 lines omitted for brevity)
      ...
1228220643	Clearing VCenter 4620:0
1228220657	Clearing VCenter 10880:0
1228220671	Clearing VCenter 11660:0
1228220684	Clearing VCenter 350:0
1228227151	Clearing VCenter 7770:0
1228227169	Clearing VCenter 8230:0

Elections officials could not explain these specifically, as they do not maintain any notes of operation of the election. They said that ballots are rerun for several reasons, and that there may have been an operator error that allowed one ballot to be entered into the system twice or one that was missed because the system displayed an error that was not caught during processing. There may have been a mail ballot in the deck of polls ballots, which would also require a rerun to remove the ballots.

Recommendations regarding the Audit Log

Issue A043 - Audit Log Discrepancies

In the November 2008 Election, a number of discrepancies were detected in the Central Tabulator (GEMS) Audit Log.

• The Registrar does not have full documentation for the audit log, such as what all entries mean and any options or configuration settings in terms of what is provided in the audit log. (Their reply to Question C00021-21.1)

• The Registrar has no procedures for review of the log. (Their reply to Question C00021-21.2) They said the logs are reviewed by IT staff to verify that an action has taken place such as a batch has been closed and the correct number of cards cast has been captured. But there was no documentation regarding when these reviews took place, if in fact they ever did, and no report resulting from any review.

• In ten places, the log showed SERVER STOPPED without first seeing SERVER STARTED and vice versa. These should be paired in all cases. If they are not paired, then transactions may have been deleted (C00021-21.7.2). An explanation of this anomaly was requested from the Registrar (C00021-21.7.1) and they did not have an answer to fully explain it other than saying it could be a myriad of reasons. The Registrar suggested that we contact Premier Election Solutions, but no referral was ever provided.

• Crashes. There were a number of crashes and other error messages. The Registrar suggested that we contact Premier Election Solutions but no referral was ever provided in response to our request.

• Re-running precincts. In 133 cases, precincts were re-run but no explanation given as to why they were re-run and no report as to why the initial run was in error. (See Issue A027: No Operator Notes)

• Clearing not shown. The central tabulator should be fully cleared prior to accumulating votes, but this was not in the log. Apparently, the Ro V does not produce a log for the entire election process.

• Direct connection to DREs: Contrary to the directive of the So S, the Audit Log shows that the Ro V downloaded several dozen ballots from directly-connected DRE equipment. The So S ordered, in their Top to Bottom Review that an “air gap” be created to prevent the very unsecured DREs from being connected to the very unsecured central tabulator. If this is done, at least you eliminate DREs and the entry point for malicious software. These brief connections could have allows malicious software to enter the central tabulator and provide for later corruption of vote totals.

• Issue A043: Question to Ro V
  
  • Please obtain full documentation about the audit log, including configuration settings to allow the GEMS system to provide additional detail. (Ref: C00021-21.1)
  • Please generate procedures for review of the log and appropriate reports for any anomalies. (Ref. C00021-21.2).
  • In that review, note time that the log shows SERVER STOPPED without first seeing SERVER STARTED and vice versa, as well as any crashes and error messages, and whether initial clearing is shown.
  • Please contact Premier Election Solutions and obtain an explanation for why START and STOP are not always paired, and obtain an explanation regarding the crashes and error messages.
  • Please provide a contact at Premier Election Solutions so we can ask for additional information.
  • Please utilize written procedures for all operations, including why certain precincts are being re-run. (See Issue A027: No Operator Notes)
  • The central tabulator should be fully cleared prior to accumulating votes, and this should appear in the log.
  • Transactions for the entire election should appear in the log.
  • The directives of the So S should be adhered to regarding the requirement that DRE ballots be remade into paper ballots and that there should be no direct connection of DRE equipment to central tabulator equipment.

• Issue A043: Legislative
  The audit log can be our most important tool to provide oversight to the canvass process. However, the log must be much more complete and detailed to be of the most use to oversight groups.
  • Full documentation should be available to GEMS and the audit log. We request this so we can suggest additional settings that may also be available to increase the level of detail in the log.
  • Procedures should exist for review of the log, and it should be reviewed and a report created for each election. The review should include an explanation for any and all anomalies, such as:
    • Places where the log shows SERVER STOPPED without first seeing SERVER STARTED and vice versa. These should be paired in all cases. If they are not paired, then transactions may have been deleted.
    • Crashes. There were a number of crashes and other error messages. The Registrar suggested that we contact Premier Election Solutions but no referral was ever provided.
    • Clearing shown. The central tabulator should be fully cleared prior to accumulating votes.
    • Inappropriate use of directly-connected DRE equipment.
  • The reconciliation procedure (See Issue A026: "No Reconciliation Procedure") should include an explanation for any and all precincts that are rescanned. In 133 cases, precincts were re-run but no explanation given as to why they were re-run and no report as to why the initial run was in error. (See Issue A027: "No Operator Notes")

Issue A018 - Audit Log Insufficient

The Audit Log available from the GEMS Central Tabulator does not provide sufficient detail to reconstruct the election. The audit log should be improved to provide an exhaustive list of vote counts for every race and every precinct. This would result in a much larger file, but file size should not be cited as a factor limiting the provision of this information. COPs asked for documentation regarding the availability of additional reporting options, but the Ro V said they did not have any documentation.

The following concerns exist:

1. The entire election must be included in the audit log. The log we were given did not include initial clearing and pre-election voting intervals.
2. The initial clearing command(s) must appear in the log if we are to know that the system was correctly initialized.
3. The Ro V should have complete documentation for the audit log and for the GEMS central tabulator.
4. Every crash and error should be investigated and explained.
5. More robust entries are desired to allow an independent reconstruction of the election from the audit log.
   • Ballot counts should be provided for VBM and Provisional processing. (See Issue A020: No Log of changes to the Election Management System)
   • Complete (exhaustive) report of each download to the GEMS central tabulator should appear in the log, including the total of votes for each race.
6. The log file should have a sequential number on each of the lines to eliminate the possibility that the audit log could be altered by election workers. In the log provided to COPs, we noticed anomalies that indicate that the log may have been tampered with, specifically the intervals when the log had STOP or START entries without the other associated entry adjacent to it.
7. The Ro V was apparently in violation of the requirements that they remake all touchscreen ballots into durable paper ballots, since on several occasions, votes were transferred directly from DRE equipment.
8. Oversight groups should have access to the audit log as the election is processed, and not be required to wait for final election certification. Nothing in the audit log can change once it is produced, and therefore, it is a public document that should be disclosed immediately, and not withheld until the election is certified, even though the log will become more extensive as the election is processed.

Providing a complete audit log is the easiest, least expensive way to document the election while providing sufficient detail for oversight groups.

• Issue A018: Question to Ro V
  We asked for information about the availablity of other, more robust settings, for the GEMS central tabulator with regard to creation of the Audit Log. These questions were not answered and we were referred to Premier Election Solutions, however, our request for a contact at Premier was not ever answered. Please provide a contact at Premier to determine if additional settings in the central tabulator exist that will allow an Audit Log that includes the aspects we require for our audit.

• Issue A018: Legislative
  Election law should be modified to mandate the creation of an exhaustive audit log file. The audit log should include not only the number of ballots included in the tabulation, but also the count of votes for each precinct and for each batch of VBM and provisional balltos. In addition, a procedure must exist for the review of the log and to address every discrepancy.

Next Section: Manual Tally Procedure