Skip to contents

The problem

A researcher studying democratic breakdown has read the sources and coded each observation as supporting a working theory or a single rival. Nine observations support the working theory and three support the rival. The researcher wants to say how much more probable those twelve observations are if the working theory holds than if the rival does, in a way a critic cannot call arithmetic chosen to suit the conclusion.

A Bayes factor is that ratio of probabilities. Getting it takes a probability for the counts under each theory, and a theory stated in words does not supply one. Lopez, Bowers, and Gajardo Cooper (2026) derive both probabilities from one model, which this package implements. This vignette works their running example: nine observations for the working theory, three for the rival.

One model

Suppose each observation the researcher might have made supports the working theory with probability θ\theta. Read θ\theta as the share of the available evidence that supports the working theory. The working theory claims that share is above one half. The rival claims it is at or below one half.

At a given share, the probability of kk supporting observations out of NN is the binomial probability, so every number below is one a reader can recompute.

Why one model gives two Bayes factors

Each theory names a range of shares, and a range is not a probability for the counts. The researcher turns the working theory’s range into a probability by averaging the binomial probability over the shares above one half, giving each share the same weight. That average is the numerator of both Bayes factors.

For the rival there are two answers. The first holds the rival to every share in its range equally and averages the same way. The second gives the rival the share in its range that makes the counts most probable, which is one half when the counts favor the working theory. Both denominators, and the numerator they share, are numbers anyone can compute:

n <- 12
k <- 9

# Averaging over a range half as wide as the interval from 0 to 1 means
# dividing the integral by one half, which is the factor of 2 below.
numerator <- 2 * integrate(function(t) dbinom(k, n, t), 0.5, 1)$value
den_avg   <- 2 * integrate(function(t) dbinom(k, n, t), 0, 0.5)$value
den_half  <- dbinom(k, n, 0.5)

round(c(numerator = numerator, den_avg = den_avg, den_half = den_half), 5)
#> numerator   den_avg  den_half 
#>   0.14675   0.00710   0.05371

Dividing the shared numerator by each denominator gives the two Bayes factors, and the package computes each directly:

c(by_hand = numerator / den_avg, from_package = bf_uniform_weights(9, 3))
#>      by_hand from_package 
#>     20.67196     20.67196
c(by_hand = numerator / den_half, from_package = bf_worst_case(9, 3))
#>      by_hand from_package 
#>     2.732168     2.732168

The twelve observations give 20.67 under uniform weights and 2.73 in the worst case. The numerator cancels from the ratio of the two Bayes factors, so that ratio is the ratio of the denominators:

round(c(ratio_of_bayes_factors = bf_uniform_weights(9, 3) / bf_worst_case(9, 3),
        ratio_of_denominators = den_half / den_avg), 2)
#> ratio_of_bayes_factors  ratio_of_denominators 
#>                   7.57                   7.57

Nine of twelve is 7.57 times as probable at a share of exactly one half as it is on average across the shares from zero to one half. The average includes shares near zero, under which nine of twelve is very improbable, and those shares are what lift 2.73 to 20.67.

The worst-case value is the smallest Bayes factor any weights on the rival’s shares can give, because no share in the rival’s range gives nine of twelve a higher probability than one half does. A critic who says the researcher chose weights for the rival that flattered the working theory can be answered with that number.

The rule for setting the rival aside

The paper proposes a rule that uses both numbers. The researcher sets the rival aside, at least provisionally, when the uniform-weights Bayes factor is 20 or more and the worst-case Bayes factor is above 1.

The first threshold follows Kass and Raftery (1995), who call a Bayes factor of 20 or more strong evidence. With even prior odds a Bayes factor of 20 puts the posterior probability of the rival at no more than 1/21=0.0481/21 = 0.048. The second condition says that no weights on the rival’s shares make the counts more probable under the rival than under the working theory. Nine of twelve meets both:

c(uniform_weights = bf_uniform_weights(9, 3), worst_case = bf_worst_case(9, 3))
#> uniform_weights      worst_case 
#>       20.671958        2.732168

With fewer than 101 observations, a uniform-weights value of 20 or more always comes with a worst-case value above 1, so the second condition changes the decision only in larger bodies of evidence. With 59 observations for the working theory and 42 for the rival it does:

c(uniform_weights = bf_uniform_weights(59, 42), worst_case = bf_worst_case(59, 42))
#> uniform_weights      worst_case 
#>      20.7775540       0.9804411

Observations that carry more weight than others

A researcher who judges one observation, say a secret memo, worth ten routine ones enters it as ten observations by summing whole-number weights:

# Eight routine supporting observations plus the memo at weight ten.
w_W <- c(10, rep(1, 8))
w_R <- rep(1, 3)
c(uniform_weights = bf_uniform_weights(sum(w_W), sum(w_R)),
  worst_case      = bf_worst_case(sum(w_W), sum(w_R)))
#> uniform_weights      worst_case 
#>       2336.9621        143.2847

What a reader would have to grant

A critic can press on the coding and on the search. For each, the package reports how far the critic’s concern would have to go before a researcher following the rule would no longer set the rival aside.

Coding

A peer cannot re-read every document, but can ask how many of the nine supporting observations would have to be re-coded as supporting the rival:

sens_coding(9, 3)
#> $bf_uniform
#> [1] 20.67196
#> 
#> $bf_worst
#> [1] 2.732168
#> 
#> $x_uniform
#> [1] 1
#> 
#> $x_worst
#> [1] 2
#> 
#> $x_star
#> [1] 1

One re-coding takes the uniform-weights value to 6.49, below 20. The worst-case value stays above 1 after one re-coding and falls below it after two. So x_star, the number at which the decision changes, is one. The conclusion tolerates so little re-coding because 20.67 starts so close to 20.

Write ω\omega for the bias factor: a search with bias ω\omega is ω\omega times as likely to find a pro-working-theory observation as a pro-rival one. Such a search turns a share θ\theta of the evidence into a share ωθ/(ωθ+1−θ)\omega\theta / (\omega\theta + 1 - \theta) among the observations found, and both Bayes factors take an omega argument that applies that change. sens_bias() finds the bias at which each condition of the rule fails:

sens_bias(9, 3)
#> $bf_uniform
#> [1] 20.67196
#> 
#> $bf_worst
#> [1] 2.732168
#> 
#> $omega_uniform
#> [1] 1.009794
#> 
#> $omega_worst
#> [1] 1.559262
#> 
#> $omega_star
#> [1] 1.009794

A search about one percent more likely to find pro-working-theory observations would change the decision. With the memo counted as ten observations, the search would have to be more than twice as likely:

sens_bias(18, 3)$omega_star
#> [1] 2.334075

Under strong bias the rival’s most favorable share is no longer one half. The biased search carries the rival’s shares onto shares found of at most ω/(1+ω)\omega / (1 + \omega), while nine of twelve is most probable at a share found of 9/129/12. Once ω/(1+ω)\omega / (1 + \omega) passes 9/129/12, which happens once ω\omega passes 9/3=39/3 = 3, a share below one half fits the counts better, and bf_worst_case() uses it:

tilt <- function(t, om) om * t / (om * t + 1 - t)
c(prob_at_half = dbinom(9, 12, tilt(0.5, 5)),
  prob_at_0.375 = dbinom(9, 12, tilt(0.375, 5)),
  worst_case = bf_worst_case(9, 3, omega = 5))
#>  prob_at_half prob_at_0.375    worst_case 
#>     0.1973957     0.2581036     0.2146125

The tipping points do not depend on this. At the bias where ω/(1+ω)\omega / (1 + \omega) equals 9/129/12, the denominator is the largest probability nine of twelve can have at any share, so the worst-case value is already below 1 there. It therefore falls to 1 at a smaller bias, where one half is still the rival’s most favorable share.

Weights on the shares other than the uniform

Giving every share in a range the same weight is a choice, and a reader may want a different one. bf_rescaled() takes a Beta distribution rescaled onto each theory’s half, one pair of parameters per side; the uniform default is the pair (1,1)(1, 1) on both sides. Below one the weight collects at both ends of each half, including next to one half where the two theories claim nearly the same share. Above one it collects in the middle of each half, so the two theories claim shares far apart:

c(uniform      = bf_rescaled(9, 3),
  next_to_half = bf_rescaled(9, 3, 0.5, 0.5, 0.5, 0.5),
  far_apart    = bf_rescaled(9, 3, 10, 10, 10, 10))
#>      uniform next_to_half    far_apart 
#>    20.671958     9.526137   252.917211

The worst-case Bayes factor takes the same pair for the working theory’s shares only, because the rival stays at its most favorable share whatever weights its proponent might state:

c(uniform      = bf_worst_case(9, 3),
  next_to_half = bf_worst_case(9, 3, a1 = 0.5, b1 = 0.5),
  far_apart    = bf_worst_case(9, 3, a1 = 10, b1 = 10))
#>      uniform next_to_half    far_apart 
#>     2.732168     2.070490     4.384581

The weights on shares are a different thing from the weights on observations above: those act as extra observations, these say what each theory predicts.

An archive of known size

The binomial model treats the observations as independent draws. A researcher who reads documents from an archive of MM documents does not read any document twice. bf_uniform_weights_finite() and bf_worst_case_finite() compute both Bayes factors for that case:

sapply(c(20, 100, 1000), function(M)
  c(M = M,
    uniform_weights = bf_uniform_weights_finite(9, 3, M),
    worst_case = bf_worst_case_finite(9, 3, M)))
#>                      [,1]       [,2]        [,3]
#> M                20.00000 100.000000 1000.000000
#> uniform_weights 162.88462  25.222782   21.048391
#> worst_case       16.84375   3.194593    2.770414

Both values fall toward the binomial ones, 20.67 and 2.73, as the archive grows. In every case the paper’s supplement computed, counts whose binomial Bayes factors meet the rule also meet it from an archive of any size, so the researcher need not state MM.