MAKING THE CASE
Investigating Large Scale Human Rights Violations Using Information Systems and Data Analysis

Chapter 11

The Guatemalan Commission for Historical Clarification: Generating Analytic Reports

Inter-Sample Analysis

Patrick Ball

Introduction

This paper reports on an analytical study requested by the Commission for Historical Clarification (CEH) and carried out by the American Association for the Advancement of Science.1 The purpose of the study was to answer the question: How many people were killed in Guatemala during the period of the CEH mandate, 1960-1996? To answer this question, we used the information in three databases of human rights violations, resulting from three projects – one conducted by the CEH, one by the CIIDH, and one by REMHI. These databases reported data from interviews with direct witnesses and victims. As a consequence of having three sources, we must first ask a) how many unduplicated killings are documented by the three projects? and then attempt to answer the second question, b) how many killings were there in all during the internal armed conflict?

Our analysis deals with these two questions. We deal first with the information collected by the three projects in light of the objectives of this analysis. We then explain the scientific methods used to estimate rates and quantities that answer the specific empirical questions derived from these objectives. We then present and interpret the results of applying the scientific methods to the information from the three databases. We subsequently analyze four regions of Guatemala in which genocide may have occurred during the period 1981-1983. Finally, using other statistical methods we show that the three projects lead to similar implications about the patterns of violence in Guatemala.

Note that in some tables, there are numbers that are not counts, but result from arithmetic operations subject to rounding error. Thus, totals shown in the table will not exactly add up to the totals of the related columns or rows. In some graphs we have retained the Spanish labels, as it is our intent to present tables and graphs as they appeared in the CEH report.2

The Information

The three databases were created using information gathered from interviews with witnesses and victims. Each contained a list of named victims who were killed, as well as numbers of people who were killed but who could not be identified by name. The three projects did not define "political killing" in the same ways. Therefore the measure we use in this study is deaths, and not the more juridically precise term "extrajudicial execution" that is used elsewhere in the CEH report. The three projects have unique definitions of murder, and to join them, it was necessary to use the broadest possible definition of the killing as a human rights violation. Thus, the totals of deaths in the AAAS study should be compared with the totals of deaths in the CEH report, and not with the totals of extrajudicial execution.

Table 1 shows the number of documented killings (victims with and without names), by time period, region, and database. Many killings were not reported to any project, and therefore, the quantities presented in Table 1 are less than the total actual number of victims who were killed in political violence in the period 1960 to 1996. Table 1 shows only those victims who were reported to one or more documentation project.

Table 1: Number of documented killings (victims with and without names), by time period, region, and project3

If no victims were reported to more than one project, the total of documented victims would be the sum of the three totals, that is 24,910+8,533+21,200 = 54,643. However, many of the same victims were reported to two or three projects. Thus, we cannot assume that the total number of victims is equal to this simple sum.

The projects were managed independently, and each victim could have been reported to more than one project. For example, assume that a victim Juan Pérez was murdered. His wife may have reported the killing to the CIIDH in 1994; his son may have given testimony to REMHI in 1996; and Peréz’s neighbor might have related the story to the CEH in 1997. If we simply sum the three databases, Peréz’s killing will be counted three times.

Duplicated reporting of deaths in more than one database is called "overlap." To estimate the total number of victims reported by the three databases, the overlap between databases must be estimated to reduce the contribution of each database by its overlap rate.

Two possible conditions demonstrate the limits of the overlap problem. If none of the victims in any database appear in any other database, then the total number of victims of killing and disappearance is equal to the sum of the number of victims in the three databases (54,643). This is the upper limit to the number of such victims. The lower limit can be found in the extreme case that the largest of the three databases (here, the CEH database) contains all the cases reported in the other two (REMHI and CIIDH). In this case, the total number of killings is simply the number of killings reported in the largest database (405+24,505=24,910). The sum of the total number of unique victims in the three databases must fall within these two limits, that is, between 24,910 and 54,643. The purpose of our analyses estimating the total number of documented killings is to narrow this range.

Many killings were not reported to any of the three projects. In the following section, we carry out a scientific analysis to estimate the total number of killings,7 including those not reported to CEH, the CIIDH, nor to REMHI. The estimate from this analysis is that between 119,300 and 145,000 killings were committed, with the most likely figure being around 132,000. Figure 1 shows the probabilities that the real value falls within various ranges around the estimate of 132,000.

Figure 1: Probability that the actual number of deaths falls within the indicated interval

The number of victims based on the three independent databases

Analysis of Overlap

The information in the three databases lists victims identified or estimated by witnesses. Some, but not all, of the victims were identified by name.8 The total number of killings in each database is referred to using the notation described below.

None of the three databases directly estimates the total number of killings in the country during the full period of the CEH mandate. Each database is a list of victims of killings who were reported directly to the project and verified according to the methodology of that particular project. As has been mentioned, many victims were not reported to any project. The total number of victims killed in Guatemala and reported (or not) to different projects can be expressed by eight categories, defined below.

The total number of victims of killing in Guatemala, N, is the sum of these eight values. The total number of victims reported to one, two, or three projects, N_k, is the sum of the seven categories that are calculated directly from the databases, that is, N₁₁₁ to N₀₀₁ as shown in Equation 1, below.

(1)

To get the total number of victims reported to one or more projects, estimates must be made of the number of victims reported to all the projects (N₁₁₁, cases that are found in all three database), and those reported to each pair of projects (N₁₁₀, N₁₀₁, and N₀₁₁), and the complements of the number reported to each project (N₁₀₀, N₀₁₀, and N₀₀₁). With this information, we can determine N_k.

Matching

It is difficult to find the same victim in any two or all three of these databases using a computer program. Victims are reported with varying information. Identical names may be spelled differently, sometimes because they were inconsistently or idiosyncratically translated from Mayan languages. Dates of birth and death can be uncertain or wrong.

Thus, it is neither practical nor accurate to match databases by automated means with computer programs. To find a person from one database (the source) in another of the databases (the target), an analyst must compare all of the data relevant to the killing, including the name, place, and date of the killing from the source with all the records in the target. This process we call "matching."9

Many victims are not identified by name in the databases. Often the original witnesses would mention only a group of people. Different witnesses of the same event often estimate different numbers of victims who suffered the same violations. In our analysis we assume that the match rates between unnamed victims are the same as the rates among the identified victims.

Matching databases is difficult, tedious and time-consuming. Instead of trying to match all the records of each database against all the records in the other databases, stratified random samples of the victims identified by name in each database were selected and matched against the records in the other two. The samples were proportionally stratified by region to assure that all regions were covered. The number of records taken in each sample is denoted by the letter m (m_CEH, m_CIIDH, m_REMHI). Including all the regions, the total number of records sampled and matched was 1,412, 1,351, and 1,122, respectively (see Table 2).10

Each person sampled (from each of the three databases) was compared against the records in the other two databases. When the same person was found in one of the other two databases, it was noted as a double-match; when the record was found in both of the other two databases, it was noted as a triple match.

Four groups of samples were chosen from the three databases. One analyst from the CEH matched one group, and a second analyst matched the other three groups. Many records were deliberately included in both samples. Only in a small number of cases were differences between the analysts' decisions found. The implication from this finding is that the error resulting from non-sampling factors was minimal.11 The numbers of matches are shown in Table 2.

Table 2: Number of matched records in three databases, by outcome

Table 2 shows that of the sample of 1,412 victims selected from the CEH database, 21 were found in the CIIDH database and in the REMHI database; these 21 are triple matches. Forty-eight victims in the CEH database were found in the CIIDH database but not in the REMHI database. In addition, 210 more victims in the CEH database were found in the REMHI but not in the CIIDH database. A total of 1,133 of the victims sampled from the CEH database were not found in either of the other two databases. The interpretation of the other two columns is the same.

We obtained overlap rates shown in Table 3 by dividing each m_xyz in Table 2 by the total number of victims sampled in each database.

Table 3: Overlap rates for three databases

To interpret this table, note that r₁₁₀ on the second line indicates that 3.4% of the victims in the CEH database are also in the CIIDH database. The database of the CIIDH is smaller than the CEH database; the same estimation from the point of view of the CIIDH is that 11.3% of the victims recorded in the CIIDH database are also in the CEH database.

Note that the differences in the estimations of the rates are not exactly in proportion to the differences in size among the databases. The differences occur because of the variability that results in the process of taking a random sample, and from the errors in matching. We treat these issues in the later section on the analysis of error.

Estimation of the total number of reported victims

As discussed in "Analysis of overlap" (above), the total number of victims of killing was estimated by the sum of seven components as defined in Equation 1, repeated below.

Equation 1

With the rates from Table 3 and the last line of Table 1 (the total number of victims 1978-1996), the components of N_k based on information in the three databases can be calculated. The results are in Table 4, below.

Table 4: Number of killings, estimated by category and by project

However, to calculate N_k, the number of victims common to all three databases, the several estimates of the number of matched records (N₁₁₁, N_110, N_101, and N₀₁₁) must be reconciled. We used the average of each of these four components, providing the totals shown in Table 5, below.

Table 5: Estimated number of killings in all three databases (CEH, CIIDH, and REMHI).

Thus, our estimate of the unduplicated number of reported killings in the three databases is 47,803. However, we show below, this number is subject to a number of controllable biases.

Demographic Theory and The Estimation of the Total Number of Undocumented Killings

In the previous section, we explained how we estimated the total number of documented killings. In this section, we discuss how to estimate the total number of undocumented killings, denoted N₀₀₀. Below, in Figures 2a, 2b, and 2c, we show with the use of Venn diagrams, the three possible ways in which the databases might be related.

Figure 2a: Distribution of databases in the universe of all violations
(where there is no relation)

Figure 2b: Distribution of databases in the universe of all violations (total overlap)

Figure 2c: Distribution of databases in the universe of all violations (partial overlap)

In Figure 2a, the databases share no violations. In Figure 2b, all the violations are contained in the largest of the three databases. In Figure 2c, some violations are shared. From the previous section, it is clear that Figure 2c is the correct representation of the three databases.

Assume that the three projects operated independently and consequently, that the probability that a project has testimony about a certain violation has no influence on whether another project has testimony about the same violation. What implication does this have for the universe of violations? In Figure 2a, the implication is that the universe of violations is large because working independently, the databases do not overlap. In Figure 2b, the implication is the inverse, that the one database is contained within the next larger, and the next larger is contained within the largest. In Figure 2c, which corresponds to our situation, the levels of overlap are partial. With the assumption of independence and the reality of overlap, the number of violations in the universe can be inferred.

Consider the case of two projects, P_A and P_B, whose databases have an overlap M in a universe of violations N.12 Note that the probability of any given killing being documented by Project P_A is that is , and the probability of any given killing being documented by Project P_B is . The probability that a killing was documented by both databases, Pr(M), is equal to Pr(M) = M/N, and by the definition of an event composed of two independent events, Pr(M) = Pr(A|B) = Pr(A) * Pr(B).

Interchanging the terms, , which reduces to

Combining the first relation with the previous result gives us , and therefore . In order to estimate only the killings that were excluded from the two projects, , or in the notation of the three-database system,

(2)

With the same logic, it is possible to derive an estimator for n_000: the measure of the number of killings that were not documented by any of the three projects.13 This estimator is presented below in Equation 3.

(3)

Measuring the Sampling Error and the Estimator Error

The estimators of N₀₀₀ and of the total number of killings depend on the estimates of the overlap between the three databases.14 To estimate of the number of killings in the categories N₁₁₁, N₁₁₀,…, N₀₀₁ that sum to N_k, the levels presented in Table 3 above are multiplied by the total number of killings in each database. We then estimated n₀₀₀ using Equation 3, above. Summing these two components gives an estimate of . We used the jackknife method to estimate because that method allows us to control the ratio bias inherent in n₀₀₀ and to estimate the variation in the three estimators necessary for this analysis (N_k, N₀₀₀, and ). In the general explanation of the method below, the estimator represents each of the three estimators. For example, N_k in Table 5, above, (47,803), is for N_k taken at the national level.

The method first randomly divides the sample of matched records (total size n records) into k groups, each of which contains m records such that n = mk. is calculated by the same method as but with the sample reduced m(k-1) obtained by omitting group .

Define

and

Equation 4 gives us k values of calculated from the sub-samples reduced by omitting a group k; the average of these values is (see equation 5), called Quenouille's estimator, and removes various biases that affect . This estimator is presented in Table 7.

The other beneficial result of the jackknife method is that the values of are distributed approximately normally.15 The standard error of the estimator (the square root of the variance) is estimated in Equation 6.

The standard errors given in Table 7 were calculated with equation 6.

Coverage in Space and Time and its Effect on N₀₀₀

In the discussion concerning Table 1 we noted that none of the three projects covered well violations in the period 1960-1977, and thus no estimation of for this period is possible. The most important complication for the estimation of N₀₀₀ is that the projects did not cover all of the regions with the same intensity. If the regions with different coverage rates among the three projects are not handled separately, the estimation could be biased.

For example, consider Region IV, in which the CEH found more than 1,300 killings, while the other two projects reported only some dozens of killings. Clearly the levels of overlap are low, but the overlap rates should not be used for an estimate of N₀₀₀ because the concept of overlap requires that the projects actually collected data in the same areas. Therefore the estimation of n₀₀₀ was based only in the projects that were in fact able to work in each region. The projects that contributed to the estimation of n₀₀₀ are in Table 6.

Table 6: Projects used to estimate N₀₀₀, by region

Results and Some Limits on the Interpretation

We now show the results by component and by region.

Table 7: Total number of killings in Guatemala 1978-1996, by category of estimation and region

In Table 7, it can be seen that in Region 0 there were N_k = 17,679 killings documented in all three projects. Over all regions, there were 47,706 killings documented, being the sum of the regional estimations.16 The standard error SE(N_k) is not the simple sum of the regions, but rather the square root of the sum of the squares of the regional values (i=0, I, …, X):

Similarly, the values for N₀₀₀ and for N are the sum of the regional values and the standard error of N₀₀₀ and is the square root of the sum of the squared regional values. In this way, we estimate that there were approximately 84,468 killings that were not reported to the CEH, to the CIIDH, or to the REMHI project. Summing N_k and n₀₀₀ to , we have as our final estimate, that there were 132,174 killings in Guatemala between 1978-1996, with a standard error of 6,568.

Possible Corrections and Limitations to the Interpretation of Table 7

There are five sources of error that cannot be quantified in this analysis due to lack of time, resources, and adequate data. In preliminary analysis of this type, the global effect of these corrections is conservative, in that the corrections tend to reduce the estimation of . The conclusion of this section is that the accumulated effect of the identified biases is does not significantly change our interpretation of Table 7.

Correlation between sources

The estimation of N₀₀₀ depends on the assumption of independence between sources; that is, that the probability that any given respondent gives her testimony to one project has no effect on the probability that the same respondent will give her testimony to one of the other two projects. It is certain that this correlation is not zero, but is positive, for two reasons.

First, psychological research has shown that survivors of human rights violations who are able to give testimonies under supportive conditions experience improvement in their mental health. Thus, it is likely that people who give testimony in these conditions may seek additional opportunities to give their testimony, thereby increasing the overlap rate.

Second, it is known that several popular movement groups organized their social bases to present testimonies to all three projects. In this way, members of these organizations would have greater probabilities of giving testimonies more than once, thus, reporting the same violations more than once, and increasing the overlap levels. The two effects – both of which are certain – would tend to bias the estimate of n₀₀₀ toward a smaller number.

Matching errors

If the analysts who conducted the matching failed to find victims who were in multiple databases, by accident or because there were inadequate data in the original sources, these omissions would tend to depress the level of measured overlap and in consequence bias the estimation of n₀₀₀ upwards. In preliminary investigations, (all that are possible given the partial state of many cases) only minimal effects of this kind were found. At most, they amounted to about 12% of the final estimate of n_000, implying about 8% of . Considering the other sources of error listed in this section, and recognizing that the data for this analysis were limited, we decided not to quantify this error (or a correction for it) in the final analysis.

Internal duplication

All of the projects that receive information from primary sources may have problems with internal duplication that results from multiple reports of the same events.17 Internal duplication tends to artificially increase the number of killings that are reported in a single database. All three projects worked hard to clean their data to reduce internal duplications, but some always remain. In a preliminary analysis, insufficient duplication appeared to require a correction for this source of error.

Rates of overlap between kinds of victims

The amount of overlap between the three databases was based on an analysis of victims identified by name and surname. However, many victims are not identified by name as a result of large-scale violence that overwhelmed the capacity of the witnesses to remember all the victims’ names. It is possible that the level of overlap between victims not identified by name could be either higher or lower than the levels measured among named victims. Given the difficulty of matching unnamed victims, it is not possible to quantify the potentially different overlap levels among victims with different amounts of identifying information. As mentioned earlier, we assumed that the match rates for unnamed and named victims were the same.

Geographic areas excluded from the analysis

In Table 7 we noted that there is not sufficient data to estimate n₀₀₀ in Region IV. Given the experience of other regions, in which the ratio between N_k and n₀₀₀ varies between 0.5 and 2, with a mode and mean close to 0.5, it is likely that the value of n₀₀₀ for Region IV is approximately 2,500. In other regions N_k is composed of data from more than two projects. However, in Region IV information was collected only by the CEH. Thus, it is possible that the ratio between N_k and n₀₀₀ for Region IV could be 0.25 or less, increasing the estimate by a factor of two or more. There are no other methods available to reduce the lack of certainty about this number, and therefore it is not included in the final estimate.

In addition, Region IV exemplifies a more fundamental problem: This methodology works only for areas covered by at least two of the three projects, even if the two projects only partly cover each area. In areas in which only one or none of the three projects conducted interviews, there is no basis for an estimate of the total number of excluded victims (n₀₀₀). Instead, in these situations only N_k (the total documented number of killings) enters the estimation process. As all three projects focussed on particular areas in Guatemala in which large scale violations were known to have occurred on the basis of journalistic and NGO accounts, it is unlikely that the excluded areas had high levels of human rights violations. However, if our analysis included any excluded area in which killings could have occurred, that inclusion would tend to increase the final estimate.

Estimation of Killing Rates, by Ethnic Group and Region

The rate of killing for a defined group is the proportion of people in that group who were killed. Quite simply, it is the number of people in the group who were killed divided by the total number in the group prior to the killings. The CEH was interested in comparing the rates of killing for defined ethnic groups during the period 1981-1983.

Six geographical regions were identified as those in which -- according to secondary sources and anecdotal evidence -- state violence was concentrated against indigenous peoples. These six regions are listed in Table 8, with the ethnic group populations according to the census of 1981.

Table 8: Populations in six regions, by ethnic group, 1981

To calculate the killing rate, the number of victims is estimated. We did this estimation twice, first to get the total number of documented victims, and then to get the estimated number of victims using the methods outlined above.

The following are the steps in this estimation process:

6. Taking from Table 9a the victims with known ethnicity, the victims without known ethnicity were apportioned to the categories "indigenous" or "not-indigenous" according to the proportions shown below in Table 10, creating the figures shown in Table 9b.

Table 9a: Number of documented killings (N_k) in three databases, 1981-1983, by region and ethnicity

Table 9b: The number of documented killings (N_k) in three databases, by ethnic group, including victims without known ethnicity, 1981-1983

Table 10: Percentage of indigenous of victims with known ethnicity of all victims in Table 9a

Figure 3: Documented proportion of the population killed by State forces in Guatemala 1981-1983, by region and ethnic group, with the 95% confidence interval19

Table 11a: Number of projected killings () in three databases, by ethnic group and region, 1981-1983

Table 11b: Number of projected killings () in three database, by ethnic group and region, including victims without identified ethnicity, 1981-1983

Figure 4: Projected proportions of ethnic groups killed by state forces in Guatemala 1981-1983, by region and ethnic group, with 95% confidence interval

Note that Figures 3 and 4 have two interpretations.20 First, regions I and III were the most affected by state violence (based on rates). In these two regions there are clear quantitative signs that the killing was so massive that it could have been genocide. Second: in all the regions the victims were disproportionately indigenous. Note, for example that as shown in Figure 4, in Region I more than 40% of the indigenous population was killed while approximately 8% of the non-indigenous population were killed. The difference between the killing rates is a factor of five. In the structure of violence committed by the Guatemalan state, these are revealing differences.

Comparisons among Databases

In the analysis of multiple databases, the databases can be compared in order to determine the levels of temporal or geographic agreement among them, and in terms of the relative levels to which they attribute responsibility to the state or insurgent forces. In this section we compare the tendencies and statistics among the three databases.

It is clear from Table 1, that there is only a moderate level of agreement about where violence occurred among the databases. In some regions all three projects found many violations, (Regions 0, I, and III), while in other regions only two projects investigated deeply (for example, Regions V, VI, IX), while in Region IX only the CEH carried out an intensive investigation.

Although the projects covered distinct areas, the second section in this part shows that the databases agree on which months saw the peaks of the violence. The third section considers the relative proportions of responsibility attributed to the two parties to the conflict.

Coincidence in Time

If the months are ordered in terms of how many killings are reported according to each of the three databases, a relatively high level of agreement is found. In Table 12, the top ten months are shown ordered as described, presenting the percentages of the total number of killings during the entire period 1979-1984.

Table 12: The ten most violent months in the three databases, 1979-198421

The shaded five months are those for which the three databases show concordance. Within the ten worst months in each database, the three systems agree on five months: January, February, April, May, and July of 1982; other months of the same year (March and June) coincide in two of the three databases.

The databases are in agreement that approximately half of the killings occurred in the ten worst months (63%, 54%, and 56%). This concentration follows Pareto’s Law, which states that 80% of any given phenomenon will occur in 20% of the categories. However, the closeness of these months to each other in time (all of them occur toward the first half of 1982) is strong evidence that this period is the most intense period of political violence. Furthermore, the level of agreement between the databases implies that although the projects did not investigate exactly the same regions of the country, they found the same trends in time.

Coincidence in the Attribution of Responsibility

The three sources agree in the attribution of responsibility to state forces and the insurgents: together they attribute more than 94% of the killings to the state and less than 6% to the guerrilla forces.

Table 13a: Total number of killings with identified perpetrator, by responsible entity22