Cops Canvass Report Five Percent Review

This is a section of the Cops Canvass Report. See also San Diego County Canvass Procedure

COPs Five Percent Review

COPs conducted a review of Ro V procedures by gather all available and relevant documentation on a 5% random sample of the 1697 precincts, i.e. 85 precincts.

Sample Size

The analysis by Kathy Dopp, 'Checking election outcome accuracy Post-election audit sampling' {{http://electionmathematics.org/em-audits/US/PEAuditSamplingMethods.pdf Kathy Dopp, 'Checking election outcome accuracy Post-election audit sampling' - Derives a simple formula to determine the number of precincts that must be sampled to validate election results with known apparent vote counts.}} Derives a simple formula to determine the number of precincts that must be sampled to validate election results with known apparent vote counts.

The following parameters govern the sample size:

• N = Total number of audit units (= number of precincts, or 1697 for San Diego County).
• w = number of votes for the winner with the least number of votes.
• r = number of votes for the runner-up with the most number of votes.
• k = a constant k: 0 < k < 1, the assumed maximum rate of margin error that could occur without raising enough suspicion to be detected without an audit. Choosing a value of 1 implies that the perpetrator could change all the ballots in any single precinct and no one would notice. Since this would likely be noticed, the value of 0.40 is a common value, that if 40% of the ballots in a precinct were changed, it would be apparent without an audit.
• b = the total number of ballots cast
• P = the probability that the sample will detecting one or more corrupt vote counts, usually set to 95% (0.95) or 99% (0.99).

The spreadsheet formulas to calculate the sample size are:

• C = the number of corrupt counts that could cause an incorrect election outcome is ‘‘=(N*(w-r))/(k*(b+w-r))’’.
• S = the audit sample size needed to detect, with probability P, one or more corrupt vote counts if there are C corrupt vote counts is ‘‘=N*(1-(1-P)^(1/C))’’.

For PROP 1A in the November 2008 Election

Parameter	Description	Value
N	auditable precincts or batches	1697
b	number of ballots cast	1245947
K	maximum margin detectable without an audit	40.00%
P	Confidence level	99.00%
w	winner vote count	592692
r	runner-up vote count	560342
C	number of corrupt precincts that could change the election = (N*(w-r))/(k*(b+w-r))	107.37
S	Sample size needed to detect C precincts changed = N*(1-(1-P)^(1/C))	71.25
	Sample size rounded up	72
	Audit sample percent	4.24%

Therefore, we settled on sampling 5% of the precincts, or 85 instead of the minimal 72 precincts as determined by the statistics. (Note that the 1% Post Election Manual Tally sample size of 17 precincts is insufficient to detect such errors to any reasonable confidence level. See Issue A045 - "PEMT Sample Size Insufficient.")

Random Selection of 85 Precincts

The precincts were chosen by selecting four-digit random numbers from a random number table and keeping values between 1 and 1650 inclusive. The random number table was from William H. Beyer, Ph.D. CRC Standard Mathematical Tables, 25th Edition, CRC Press (1978){{https://copswiki.org/Common/M880 William H. Beyer, Ph.D. CRC Standard Mathematical Tables, 25th Edition, CRC Press (1978) - using 'A Table of 14,000 Random Digits' (p 545-548)}} using 'A Table of 14,000 Random Digits' (p 545-548) and using the following procedure:

1. Start at the first digit in the table
2. Consider the next four digits.
3. If the numerical value of the four digits is outside the range from 1 to 1650, inclusive, continue at step 2.
4. If the number has been chosen before, continue at step 2.
5. Accept the number as a precinct number in the random sample.
6. Continue from step 2 until 68 precincts have been chosen.

These 68 randomly chosen precincts are combined with the 1% sample processed in the "One-Percent Manual Tally" procedure (17 precincts) conducted by the Ro V to result in a total of 85 precincts in the sample.

Note:

• After completing our review, we suggest that ten-sided die be used, and the die rolled again if the precinct is from a collection center that was already sampled. In the procedure above, many collection centers were sampled more than once with one sampled five times. (See Issue A035 - "PEMT Random Selection Not Technically Correct" for the recommended procedure.)
• The actual number of precincts was 1697, according to Collection Center Logs.

Data Collection

%IMAGE{"example_of_overprinting.jpg" type="frame" align="right" caption="An example of overprinting, effectively making the tape unreadable for this race and there is no other record for oversight review." size="300x300"}%

• We were only able to gather documentation on ballots processed on election night, including the 1% sample in the Ro V Post Election Manual Tally ("PEMT") and another 4% randomly selected. It was not possible to gather data on VBM ballots or provisional ballots because these are directly scanned into the central tabulator and do not provide a paper trail.
• The Scanner Tapes were scanned by COPs volunteers for all precincts in the sample and posted on our web site. Volunteers entered the data into web-based forms from the scanned tape files so that we could determine how many ballots were scanned for each precinct and could determine the voted result of the election if only those ballots are considered.
• Scanned the Ballot Statements for all 85 precincts in our random sample with signatures of precinct workers redacted. This was performed by the Ro V, and then we posted these statements for the public to review.
• Scanned the Street Indexes for about half of the precincts in our sample. These were not useful from a paper-trail standpoint because they were not always completed by the precinct workers in the 6pm to 8pm time period, thereby leaving off about 30% of the voters. Some of the ballot counts from the Street Indexes were added to our database, but we did not process all of them when we realized that they were incomplete.
• Other related documents were also scanned.
• We manually inspected sign-in rosters in about half the sample precincts. We were unable to scan these because the Ro V said it was a violation of privacy of the voters signatures. We found almost no discrepancies in the count of signatures on the sign-in rosters and what was reported in the Ballot Statements.

Precinct Data

The web page All Precincts provides results of our data collection activity for all the precincts in our 5% sample. Details and supporting documents can be viewed for each precinct by clicking on the precinct number.

Races

We also tabulated the results of races in the election, based on the vote from our 5% sample. The web page All Races provides a link to a separate page for each race in the election and lists the vote count from each precinct in the 5% sample. At the date of this writing, additional review of the data used in the races is required before we are confident of the reliability of these data. In many cases, the exact count was not available from some precincts because the tape was unreadable or the scan was defective. Readers who may be interested in specific races are encouraged to join our review effort and work to validate the count based on the scans of the scanner tapes.

Because of the other issues we detected in our review, we did not attempt to check the exact count to see if it matched the report from the Ro V, although that is our logical next step in the review. Such a comparison could detect some types of election fraud such as count rigging by the central tabulator operator.

Issues of Concern specific to this process

• Issue A002: Public Posting of Audit Trail -- The paper audit trail should be posted on the Internet on election night by elections officials so anyone can review it.

• Issue A024: Scanner Tapes Not Accessible
  Scanner report tapes are no longer available early in the canvass.

• Issue A025: Scanner Tapes Difficult to Process
  The audit trail provided by the current Ro V procedures include cash-register style paper tape which is difficult to process and sometimes impossible to read.

• Issue A009: Ballot Statements Not Scannable
  Ballot Statements must be scannable in their entirety, including signatures of precinct workers who sign them.

• Issue A018: Audit Log Insufficient
  The audit record available from the GEMS Central Tabulator does not provide sufficient detail to reconstruct the election.

• Issue A023: Documents Automatically Processable
  Documents that are considered public, such as Ballot Statements, Street Indexes, and others should be automatically processable to minimize oversight costs.

• Issue A029: Audit Devices Should be Allowed
  If no improvement to the audit log is possible, then audit devices should be placed on the serial interface cables that supply data to the central tabulator to allow a log of all communication from all scanner and DRE equipment to the central tabulator.

---

This section included only to complete relative links to documents.

Documents Referenced

The following documents were provided by the Ro V as a response to our request or obtained from other sources. These documents will be referenced in later sections by name. "RoV Procedure Documents" specify procedures used in each election while "RoV Election Documents" are specific to this election.

The casual reader of this report may move directly to Procedure Summary while keeping in mind the fact that these documents and abbreviations are defined here.

• Voluntary Voting Systems Guidelines{{https://copswiki.org/Common/VoluntaryVotingSystemGuidelines Voluntary Voting Systems Guidelines - were authored by the Technical Guidelines Development Committee (TGDC), a committee authorized under the HELP America Vote Act (HAVA) of 2002, and researchers at the National Institute of Standards and Technology (NIST) for the Election Assistance Commission (EAC). These guidelines were authored primarily to guide voting machine manufacturers in their design of voting systems.}} were authored by the Technical Guidelines Development Committee (TGDC), a committee authorized under the HELP America Vote Act (HAVA) of 2002, and researchers at the National Institute of Standards and Technology (NIST) for the Election Assistance Commission (EAC). These guidelines were authored primarily to guide voting machine manufacturers in their design of voting systems.
• Logic & Accuracy Testing Procedure{{https://copswiki.org/Common/M829 Logic & Accuracy Testing Procedure - An RoV Procedure Document that specifies how to confirm basic operation of the electronic scanning equipment.}} An RoV Procedure Document that specifies how to confirm basic operation of the electronic scanning equipment.
• Ballot Order Procedure{{https://copswiki.org/w/pub/Common/M717/COP_PRA_Response_Attach_2.pdf Ballot Order Procedure - An RoV procedure document that specifies how the number of ballots is determined.}} An RoV procedure document that specifies how the number of ballots is determined.
• Ballot Order Spreadsheet{{https://copswiki.org/w/pub/CommonM821/C00019_Response_20.3.3_Ballot_order_and_supplemental_order_for_11-08.xls Ballot Order Spreadsheet - An RoV Election Document that lists the quantity of ballots ordered for each precinct and supplemental orders. Includes Sequence Number to Precinct Number conversion.}} An RoV Election Document that lists the quantity of ballots ordered for each precinct and supplemental orders. Includes Sequence Number to Precinct Number conversion.
• Ballot Inventory Sheet{{https://copswiki.org/w/pub/Common/M717/COP_PRA_Response_Attach_1.pdf Ballot Inventory Sheet - An RoV Election Document. This is a sample of the document sent to each precinct leader to verify the count of blank ballots received.}} An RoV Election Document. This is a sample of the document sent to each precinct leader to verify the count of blank ballots received.
• Street Indexes{{https://copswiki.org/Cops/StreetRosters Street Indexes - An RoV Election Document that lists voters by address and each is checked off as they vote. About half of our 5% sample were scanned but we discovered they were incomplete as most polling places did not continue to complete these after 6 p.m. on election day.}} An RoV Election Document that lists voters by address and each is checked off as they vote. About half of our 5% sample were scanned but we discovered they were incomplete as most polling places did not continue to complete these after 6 p.m. on election day.
• Ballot Statements{{https://copswiki.org/Cops/BallotStatements Ballot Statements - An RoV Election Document. These were initially withheld by the RoV due to legal concerns of showing signatures. (Requested in C00022). These have been scanned by the Registrar with the signatures concealed.}} An RoV Election Document. These were initially withheld by the RoV due to legal concerns of showing signatures. (Requested in C00022). These have been scanned by the Registrar with the signatures concealed.
• Collection Center Logs{{https://copswiki.org/Cops/CollectionCenterLogs Collection Center Logs - Reports from 80 collection centers were scanned from RoV on 2009-03-23. Each report handles 21 precincts, on the average. No logging of seal conditions or serial numbers.}} Reports from 80 collection centers were scanned from RoV on 2009-03-23. Each report handles 21 precincts, on the average. No logging of seal conditions or serial numbers.
• Collection Center Seal Report{{https://copswiki.org/w/pub/Common/M717/COP_PRA_Response_Attach_4.pdf Collection Center Seal Report - Describes 6 seals that are missing or wrong}} Describes 6 seals that are missing or wrong
• Tally Center Log{{https://copswiki.org/Common/M828 Tally Center Log - Use by incoming inspectors at the Tally Center. Describes 99 seals broken or wrong, 6% error rate.}} Use by incoming inspectors at the Tally Center. Describes 99 seals broken or wrong, 6% error rate.
• Security Seal Standards{{https://copswiki.org/Common/SecuritySealStandards Security Seal Standards - includes ISO/PAS 17712 - Freight container seal standards.}} includes ISO/PAS 17712 - Freight container seal standards.
• Batch box label (Used as tracking device when in facility).
• Scanner Tapes for our 5% Sample{{https://copswiki.org/Cops/ScannerTapes Scanner Tapes for our 5% Sample - These pdf files contain three scanner tapes each, and were manually scanned by COPs volunteers.}} These pdf files contain three scanner tapes each, and were manually scanned by COPs volunteers.
• Manual Tally Procedure{{https://copswiki.org/Common/M830 Manual Tally Procedure - An RoV Procedure Document}} An RoV Procedure Document
• San Diego County Manual Tally Report for 2008-11-04 election{{https://copswiki.org/Common/M691 San Diego County Manual Tally Report for 2008-11-04 election - This five-page report is extremely deficient compared with the recommendation of the Secretary of State, which would result in about 800 pages. This report includes NO vote totals and only a count of the ballots.}} This five-page report is extremely deficient compared with the recommendation of the Secretary of State, which would result in about 800 pages. This report includes NO vote totals and only a count of the ballots.
• Audit Log (of central tabulator){{https://copswiki.org/Common/M831 Audit Log (of central tabulator) - An RoV Election Document produced by the GEMS Central Tabulator.}} An RoV Election Document produced by the GEMS Central Tabulator.
• Canvass Reports{{https://copswiki.org/Cops/CanvassReports Canvass Reports - Certified results of the RoV for our 5% Sample.}} Certified results of the RoV for our 5% Sample.
• Precinct Sequence 1563{{https://copswiki.org/Cops/PrecinctSeq1563 Precinct Sequence 1563 - which is one of the few Ballot Statements that reconciled completely. The precinct had no touchscreen ballots. The ballot statement discloses 565 ballots counted and that matches the number of non-provisional signatures (565). However, the scanner tape from the Tally Center says there were only 550 cards cast.}} which is one of the few Ballot Statements that reconciled completely. The precinct had no touchscreen ballots. The ballot statement discloses 565 ballots counted and that matches the number of non-provisional signatures (565). However, the scanner tape from the Tally Center says there were only 550 cards cast.

Other Documents and media

• Open Canvass method{{https://copswiki.org/Common/OpenCanvass Open Canvass method - A method to reduce election processing vulnerability based on scanning all ballots and comparing vote extraction from several competing actors}} A method to reduce election processing vulnerability based on scanning all ballots and comparing vote extraction from several competing actors
• Video of the February 5, 2008 Presidential Primary Canvass{{https://copswiki.org/Common/M434 Video of the February 5, 2008 Presidential Primary Canvass - }}
• Nov. 21, 2008 Meeting with Deborah Seiler (video){{https://copswiki.org/Common/M688 Nov. 21, 2008 Meeting with Deborah Seiler (video) - After this initial meeting, the Registrar refused to allow us to record our meetings, and so we discontinued that approach and instead used written requests for information only.}} After this initial meeting, the Registrar refused to allow us to record our meetings, and so we discontinued that approach and instead used written requests for information only.
• Secretary of State Top-to-Bottom Review{{https://copswiki.org/Common/M860 Secretary of State Top-to-Bottom Review - }}
• All Precincts{{https://copswiki.org/Cops/AllPrecincts All Precincts - provides results of our data collection activity for all the precincts in our 5% sample.}} provides results of our data collection activity for all the precincts in our 5% sample.
• All Races{{https://copswiki.org/Cops/AllRaces All Races - provides a link to a separate page for each race in the election and lists the vote count from each precinct in the 5% sample.}} provides a link to a separate page for each race in the election and lists the vote count from each precinct in the 5% sample.
• Kathy Dopp, 'Checking election outcome accuracy Post-election audit sampling' {{http://electionmathematics.org/em-audits/US/PEAuditSamplingMethods.pdf Kathy Dopp, 'Checking election outcome accuracy Post-election audit sampling' - Derives a simple formula to determine the number of precincts that must be sampled to validate election results with known apparent vote counts.}} Derives a simple formula to determine the number of precincts that must be sampled to validate election results with known apparent vote counts.
• Potrero Planning Group Recall Vote Count Video, 2007-12-11{{https://copswiki.org/Common/M351 Potrero Planning Group Recall Vote Count Video, 2007-12-11 - This is the entire initial vote count performed on December 11, 2007 demonstrating the method of manual tallying used by the San Diego County Registrar of Voters.}} This is the entire initial vote count performed on December 11, 2007 demonstrating the method of manual tallying used by the San Diego County Registrar of Voters.

Interaction Documents

The following documents record our requests and responses by the Ro V:

• C00014: Letter to Registrar of Voters requesting Statement of Votes Cast, 2008-10-03{{https://copswiki.org/Common/M667 C00014: Letter to Registrar of Voters requesting Statement of Votes Cast, 2008-10-03 - We requested that the RoV create two copies of the scanner tapes, as specified by law, so that we could start our review of the election right away. They refused, forcing us to wait until the election was certified before we could start to review the scanner tapes.}} We requested that the RoV create two copies of the scanner tapes, as specified by law, so that we could start our review of the election right away. They refused, forcing us to wait until the election was certified before we could start to review the scanner tapes.
• C00015: Letter to Secretary of State requesting support of our request to Registrar of Voters requesting Statement of Votes Cast, 2008-10-08{{https://copswiki.org/Common/M668 C00015: Letter to Secretary of State requesting support of our request to Registrar of Voters requesting Statement of Votes Cast, 2008-10-08 - We forwarded our letter to the RoV (C00014) to the Secretary of State in an effort to persuade the county to comply with access to the scanner report tapes prior to the end of the election. This request was refused, with the rationale being that state law applies only in the precinct and they are not required to create a second scanner tape prior to election certification.}} We forwarded our letter to the RoV (C00014) to the Secretary of State in an effort to persuade the county to comply with access to the scanner report tapes prior to the end of the election. This request was refused, with the rationale being that state law applies only in the precinct and they are not required to create a second scanner tape prior to election certification.
• C00016: Letter to RoV: Manual Tally Report Questions, 2008-11-20{{https://copswiki.org/Common/M691 C00016: Letter to RoV: Manual Tally Report Questions, 2008-11-20 - This three-page request was submitted prior to the closing day of the canvass period and pointed out that they were not using the appropriate format for their PEMT report.}} This three-page request was submitted prior to the closing day of the canvass period and pointed out that they were not using the appropriate format for their PEMT report.
• C00016 R: Response by County Counsel regarding Nov. 20 Letter to the RoV, 2008-11-25{{https://copswiki.org/Common/M689 C00016 R: Response by County Counsel regarding Nov. 20 Letter to the RoV, 2008-11-25 - This response was the least cooperative and would not answer simple questions regarding why ballots were missing in the PEMT report.}} This response was the least cooperative and would not answer simple questions regarding why ballots were missing in the PEMT report.
• C00017 - Letter to San Diego Registrar of Voters regarding Election Procedures (2008-12-02){{https://copswiki.org/Common/M716 C00017 - Letter to San Diego Registrar of Voters regarding Election Procedures (2008-12-02) - This five-page request was submitted on the closing day of the canvass period.}} This five-page request was submitted on the closing day of the canvass period.
• C00017 R - Response to C00017 by San Diego Registrar of Voters regarding Election Procedures (2008-12-16){{https://copswiki.org/Common/M717 C00017 R - Response to C00017 by San Diego Registrar of Voters regarding Election Procedures (2008-12-16) - Includes Ballot Order Procedure, One Percent Manual Tally Procedure, Collection Center Comments on Carton Seals.}} Includes Ballot Order Procedure, One Percent Manual Tally Procedure, Collection Center Comments on Carton Seals.
• C00019 - Letter to San Diego Registrar of Voters regarding Election Processing (2009-01-08){{https://copswiki.org/Common/M819 C00019 - Letter to San Diego Registrar of Voters regarding Election Processing (2009-01-08) - This 19-page request was a followup to C00017, and was submitted twice, the second time (C00021) in an abbreviated format to aide in processing.}} This 19-page request was a followup to C00017, and was submitted twice, the second time (C00021) in an abbreviated format to aide in processing.
• C00019 R - Response to C00019 and C00021 by San Diego Registrar of Voters regarding Election Procedures (2009-02-24){{https://copswiki.org/Common/M821 C00019 R - Response to C00019 and C00021 by San Diego Registrar of Voters regarding Election Procedures (2009-02-24) - }}
• C00022 - Questions to Registrar of Voters on Nov, 2008 Election (Part 3) (2009-09-01){{https://copswiki.org/Common/M878 C00022 - Questions to Registrar of Voters on Nov, 2008 Election (Part 3) (2009-09-01) - Request for Ballot Statements, Street Indexes, Question about ballot counts, and Premier contact.}} Request for Ballot Statements, Street Indexes, Question about ballot counts, and Premier contact.
• C00022 R: Response to C00022 by Registrar of Voters (2009-09-09){{https://copswiki.org/Common/M879 C00022 R: Response to C00022 by Registrar of Voters (2009-09-09) - Answered some questions. Ballot Statements took several weeks to finally get in full, no contact to Premier Election Solutions provided.}} Answered some questions. Ballot Statements took several weeks to finally get in full, no contact to Premier Election Solutions provided.

Abbreviations and Terminology

If you do not find the definition here, please also see this reference:
http://www.eac.gov/vvsg/g/definitions-of-words-with-special-meanings

• Collection Centers - 80 centers throughout the county where ballots are collected from individual precincts, each averaging 21 precincts.
• CVR - Cast Vote Record - Archival record of all votes produced by a single voter. Cast vote records may be in electronic, paper, or other form. Electronic cast vote records are also called ballot images. In San Diego County, the durable paper ballot is the CVR.
• DRE - Direct Recording Equipment - Voting machines that directly record the vote of the voter without using paper ballots. These machines have superior usability for disabled voters, but are inherently subject to election fraud risks, and therefore are no longer used except for one unit per precinct for use by disabled voters, and their vote is transferred to paper ballots and processed with the rest.
• IVVR - independent voter-verifiable records - all voting systems must include a vote-capture device that uses independent voter-verifiable records (IVVR). For example, durable paper ballots that are scanned represents a ballot that can be easily read by the voter and interpreted by machines.
• PAV - Permanent Absentee Voter - Deprecated. Now "VBM."
• PI - Precinct Inspector - The lead poll worker at each polling place.
• RoV - San Diego County Registrar of Voters
• Seq# - The precinct sequence number, from 0001 to 1650. This number is used internal to the Ro V and relates to one polling place.
• SPUW - Supply Pick-Up Weekend - The weekend prior to election day when supplies are distributed to polling places by having polling place workers pick up their supplies.
• SoS - California Secretary Of State CA - This elected office is responsible for overseeing the elections process.
• Tally Center - The central office of the Ro V where the main task of scanning ballots and compiling the final certified canvass occurs.
• VBM - Vote-by-Mail - Previously called PAV, permanent absentee voters.
• Voter Data File - Information about each registered voter is maintained in a database, including whether the voter has voted, whether they have requested a VBM ballot, their digitized signature, etc.

---

Next section: Issues of Concern