Skip to contents

Changelog

manytestsr 0.0.4.1014

New functions

  • pStephensonQuad() and stephenson_scores(): the combined Stephenson rank test of the sharp null that the block_test_power paper runs. One linear rank statistic per tuning parameter zeta, each with the score choose(r - 1, zeta - 1) on within-block ranks, combined by the quadratic form in coin::independence_test() and referred to a chi-square with one degree of freedom per zeta. Until now that test lived only in the paper’s replication archive, while the package’s pCombStephenson() wrapped the CMRSS quantile procedure under a similar name. The two are different tests; the documentation of each now says so. Tests in tests/testthat/test_stephenson_quad.R pin the score definition, tie the p-value to a computation done directly with coin, and check that the test depends on the outcome only through ranks and that its signed scores cancel across blocks whose effects reverse sign.

manytestsr 0.0.4.1013

Changes

  • pCombCauchyDist() now includes the maximum distance, so it combines seven p-values: six single-score tests and the six-score quadratic test that pIndepDist() runs. The function had used a five-score set with no recorded reason since the June 2025 refactoring that introduced it. A check on the block_test_power paper’s simulations (Analysis/cauchy_maxdist_check.R there) found that under effects that cancel across blocks the maximum distance is the score that carries the signal: the five-score quadratic form rejected in 13.5 and 5.5 percent of replications in the two canceling scenarios, the six-score form in 44.5 and 49 percent, and the Cauchy combination rose from 7.5 and 2 percent to 30.5 and 34 percent once its quadratic term was the six-score form. The other six scenarios moved by two points or less. The tests in tests/testthat/test_combcauchy_components.R now pin seven components, check that the quadratic term’s p-value equals pIndepDist()’s, and rule out both earlier compositions.

manytestsr 0.0.4.1012

Bug fixes

  • pCombCauchyDist() now combines the six components its documentation lists. The function builds one test formula per score and then adds a quadratic test of all the scores together. The line that added the quadratic formula overwrote the last single-score formula instead of appending, so the tanh score was tested only inside the quadratic and the Cauchy sum held five terms rather than six. It now appends. On the block_test_power paper’s simulations the change moves rejection rates by less than Monte Carlo error except in the sparse-effects scenario, where the rate rises from 0.513 to 0.530. Tests in tests/testthat/test_combcauchy_components.R compute the six components with coin directly and check that the function returns their combination, with and without a block term.

manytestsr 0.0.4.1011

Bug fixes

  • pCombCauchyDist() implements the Cauchy combination rule of Liu and Xie (2020). The previous rule halved each p-value before the tangent transform, which put every angle in (0, pi/2), made every tangent positive, and made the two-sided correction select the same branch every time; fed five uniform independent p-values, its output fell below 0.5 about 80 percent of the time. The new acat_combine() in R/pval_combine.R takes each component on the log scale in both tails, so neither an underflow to 0 nor a saturation at 1 decides the combination, and coin_component_logp() refits a component whose statistic sits exactly on its null expectation against the permutation distribution and uses its mid-p value. See PLAN_ACAT_BOUNDARY.md.

manytestsr 0.0.4.1010

Bug fixes

  • Monte Carlo p-values can no longer be zero. Below simthresh the p-value functions draw their reference distribution with coin::approximate(nresample = sims), and coin reports count/sims, which is exactly zero when no resample beats the observed statistic. Verified directly: with perfect separation coin returns p = 0 at sims = 9 and sims = 49. A zero p-value sits at or below every threshold, so such a node always rejects. Under the null the observed statistic is the most extreme of the sims + 1 exchangeable values with probability about 1/(sims + 1), so any critical value below that had realized size about 1/(sims + 1) rather than its nominal value.

    The functions now report (count + 1)/(sims + 1), the standard convention (Davison and Hinkley 1997; Phipson and Smyth 2010, “Permutation P-values should never be zero”). The observed assignment is itself a draw from the null and belongs in both numerator and denominator. The correction is applied in pOneway(), pWilcox(), pIndepDist(), pTestTwice(), pPolyRank(), and to each component of pCombCauchyDist() before the Cauchy combination. The asymptotic branch is untouched.

    This matters most for the smallest thresholds, which is to say for alpha/m-style corrections, so the old behavior favoured procedures testing near nominal alpha – this package’s own top-down procedure included – over bottom-up competitors. It had to be fixed before any head-to-head benchmark. New tests in tests/testthat/test_mc_pvalue_floor.R (59 assertions), verified failing before the fix.

manytestsr 0.0.4.1009

Documentation

  • The vignettes now follow the 0.0.4.1008 corrections. The detection sections of getting-started and hierarchical-testing-workflow show and explain hit_type and group_p, and both vignettes stop recommending the stream-based sequential alpha procedures (alpha_investing, alpha_saffron, alpha_addis). Those procedures are now labeled experimental in the tree setting – their guarantees are proven for flat streams of p-values, not for gated tree-structured testing – and remain slated for deprecation, matching the README.
  • The advanced-methodologies vignette is archived to archive/ and is no longer built into the package or the pkgdown site. It presented experimental features (use_closed_testing, use_meinshausen, use_evalues) with stronger claims than the package can support, and referenced helper functions that do not exist.

manytestsr 0.0.4.1008

Corrections responding to an anonymous AOAS referee who worked through this package alongside the manuscript and identified genuine errors (2026-08; see FIX_PLAN.md for the mapping). The pre-correction state is tagged pre-referee-fixes so previously published numbers remain reproducible.

Breaking changes

  • compute_error_load() and the adaptive-alpha schedule builders now implement the paper’s Definition 2: per-depth loads sum path power (product of PROPER-ancestor rejection probabilities) over the nodes at each depth, from depth 2. The previous formula multiplied each node’s own theta back in and included a depth-1 term, understating the load and returning needs_adjustment = FALSE in designs that require adjustment. The needs_adjustment gate and the alpha-schedule denominators now use the same quantity.
  • Power calculations behind the error load are now two-tailed: theta = pnorm(delta*sqrt(n) - z) + pnorm(-delta*sqrt(n) - z), so theta at delta = 0 equals the size alpha rather than alpha/2.
  • The error-load/power path refuses nodesize values below 1 with an error. find_blocks()’s default blocksize = "hwt" stores harmonic weights whose node sums lie on the unit interval; feeding them to a power formula produced meaningless near-floor thetas. Pass a headcount column (e.g. blocksize = "nb") for any error-load or adaptive-alpha use; hwt remains valid for splitting and weighting.
  • alpha_adaptive_tree_pruned() now warns at creation that the strong-FWER guarantee for the pruned-load schedule and its switching rule DOES NOT HOLD: the underlying theorem was falsified by exact counterexample (FWER 0.063 at alpha 0.05 with all hypotheses satisfied). Levels are computed as before for reproducibility; for a schedule with a proof use alpha_adaptive_tree() with static budget weights.

Bug fixes

  • report_detections() computed each block’s parent p-value at the GLOBAL maximum depth, so branches whose testing stopped earlier got the wrong parent or NA (surfacing as hit = NA, silently dropped by sum(hit, na.rm = TRUE) callers), and group coverage was missed on any branch shorter than the deepest one. All quantities now use each block’s own final tested depth, and hit is never NA.
  • report_detections() gains hit_type (“single” = the block’s own test rejected; “group” = family parent rejected with no child rejected; “none” otherwise – a sibling’s individual rejection explains the parent, so coverage does not spread to failed siblings) and group_p (the rejecting parent’s p-value for covered blocks). pfinalb is now the running maximum of p-values along the block’s tested path rather than the deepest p reached.
  • Removed a stray positional argument in five coin::approximate() calls that bound to coin’s cl formal; harmless under parallel = "no" but broken under parallel = "snow".

manytestsr 0.0.4.1007

Bug fixes

  • make_results_tree() now propagates block-level truth (nonnull) up the tree instead of labeling leaf nodes only. An internal node is non-null iff a descendant leaf is non-null, and a known null iff all of its descendant leaves are known nulls. Without this, a false rejection of a true internal null hypothesis (rejecting a whole null group/college/region) was dropped from the node-level FWER tally (node_any_false_rejection, node_false_rejection_prop, node_num_false_rejections, node_false_discovery_prop), and correct rejections of non-null ancestors were missing from node_true_discoveries and node_power. The procedure’s rejection decisions are unchanged; only the truth labeling used to score node-level metrics is corrected. New tests in tests/testthat/test_node_truth_propagation.R.

manytestsr 0.0.4.1005

Bug fixes

  • pCombStephenson() is now compatible with CMRSS 0.2.6+, which constrains its internal k argument to 1..m (where m = sum(Z) is the number of treated units) rather than 1..n. The wrapper’s user-facing k argument remains paper-notation in 1..n (the rank index of tau among all units), and the wrapper now translates internally as cmrss_k = k - (n - m) before calling CMRSS::pval_comb_block(). The default k = n still tests Fisher’s sharp null. The degenerate path (k <= n - m) now warns and returns p = 1 without calling CMRSS, avoiding the LP solver on a branch whose test statistic is constant across permutations anyway.

  • pCombStephenson() now validates k <= n and stops with a clear error if k exceeds the total number of units.

Dependency changes

  • Added highs to Suggests. CMRSS 0.2.6+ requires an LP solver (highs or gurobi) for the non-degenerate path of pval_comb_block(). highs is open-source and the recommended solver. Install with install.packages('highs') or via the GitHub-only CMRSS GitHub remote.

manytestsr 0.0.4.1003

New features

  • New exported function pPolyRank() tests Fisher’s sharp null of no effects using multiple polynomial rank score functions simultaneously via coin::independence_test(). Computes within-block polynomial scores at multiple r values (default r = 2, 6, 10) and passes them as a multivariate response, providing adaptive sensitivity to treatment effects without pre-committing to a single rank scoring.

  • New exported function pCombStephenson() provides a formula-based wrapper around CMRSS::pval_comb_block(), the combined Stephenson rank test of Kim, Li, and Bowers. Tests quantile-of-effects hypotheses (whether the k-th largest individual effect exceeds a threshold); defaults to k = n, c = 0 for the sharp null. CMRSS is a Suggests dependency installed from GitHub (bowers-illinois-edu/CMRSS).

Dependency changes

  • Moved 8 packages from Imports to Suggests: stringi, tidygraph, ggraph, digest, ggplot2, Ckmeans.1d.dp, onlineFDR, hommel. Core test statistic functions (pIndepDist, pTestTwice, pCombCauchyDist, pOneway, pWilcox) now install without pulling in heavy tree-testing and visualization libraries. Functions that need the moved packages check with requireNamespace() and give a clear error message if the package is missing.

  • Removed ClusterR from Imports (unused; the code that called it was already commented out).

  • Replaced dataPreparation dependency with an internal which_are_constant() helper (a one-liner that checks for columns with fewer than 2 unique values). dataPreparation remains in Suggests for cross-validation testing only.

  • Added CMRSS to Suggests (GitHub-only: bowers-illinois-edu/CMRSS).

manytestsr 0.0.4.1002

New features

  • New exported factory function alpha_adaptive_tree_pruned() creates a branch-pruning adaptive alpha system for use with find_blocks(). Unlike alpha_adaptive_tree(), which pre-computes a fixed schedule, this version can recompute the schedule on the surviving subtree after each depth — giving more alpha to surviving branches when dead branches are removed. Returns a list with three components: $alphafn (standard closure), $update (recompute on pruned tree), and $reset (restore full tree).

  • find_blocks() now supports list-valued alphafn parameters. When alphafn is a list (as returned by alpha_adaptive_tree_pruned()), find_blocks extracts the $alphafn, $update, and $reset components, calls reset at the start of each run, and calls update after each depth’s testable decisions. Plain function alphafn values continue to work unchanged.

  • New internal helper .get_all_descendants() performs BFS traversal on tree-structured node_dat to find all descendants of given nodes.

manytestsr 0.0.4.1001

New features

  • New exported function compute_adaptive_alphas_tree() computes per-depth adjusted significance levels from an actual (possibly irregular) tree structure. Takes node_dat with per-node sample sizes (as returned by find_blocks()) instead of assuming a regular k-ary tree. The algorithm divides alpha at each depth by the sum of path powers — the expected number of tests conducted at that depth. For regular k-ary trees, this produces identical results to the parametric compute_adaptive_alphas().

  • New exported factory function alpha_adaptive_tree() creates a closure for use with find_blocks(alphafn = ...), using the tree-based alpha schedule from compute_adaptive_alphas_tree(). Drop-in replacement for alpha_adaptive() when the tree has irregular branching or unequal sample sizes across nodes.

Bug fixes

  • Fixed 11 test failures in test_alpha_adaptive.R that referenced the removed tau parameter. Two tests (“tau = 0” and “tau = 1”) were rewritten as error-load equivalents; the rest had tau arguments removed.

manytestsr 0.0.4.1000

New features

  • New exported function compute_error_load() computes the error load at each tree level — the expected number of all-null sibling groups that the procedure tests. When the total error load is at most 1, the unadjusted procedure controls FWER via natural gating; when it exceeds 1, adaptive alpha adjustment is required. Supports both a parametric interface (regular k-ary trees with equal splits) and a tree interface (irregular trees with per-node sample sizes from find_blocks()).

Changes

  • compute_adaptive_alphas() now checks the error load before computing adjusted alphas. When the total error load is at most 1 (natural gating suffices), nominal alpha is returned at every level without adjustment. The tau parameter has been removed; the error load check replaces it.

  • compute_adaptive_alphas() gains an "error_load" attribute on its return value, so callers can inspect the error load diagnostics without a separate call to compute_error_load().

manytestsr 0.0.4.0000

New features

  • Added adaptive alpha adjustment for tree-structured hypothesis testing (compute_adaptive_alphas() and alpha_adaptive()). This implements Algorithm 1 from the paper’s Appendix D, which adjusts significance levels at each tree depth based on estimated power decay. When cumulative power is high, alpha is tightened to account for the multiplicity of tests that power enables. When cumulative power drops below a threshold, natural gating suffices and nominal alpha is used. The procedure supports both constant and variable branching factors.

Interface changes

  • The alphafn interface used by find_blocks() now passes a depth parameter (integer vector of tree depths, 1 = root) to alpha adjustment functions. This enables alpha strategies that depend on tree structure rather than treating p-values as a flat stream.

  • alpha_investing(), alpha_saffron(), and alpha_addis() now accept a depth argument for interface compatibility. They do not use it; their behavior is unchanged.

manytestsr 0.0.3.0000

  • Initial tracked version with find_blocks(), splitting functions (splitCluster, splitEqualApprox, splitLOO, splitSpecifiedFactor, splitSpecifiedFactorMulti), p-value functions (pOneway, pWilcox, pIndepDist, pCombCauchyDist, pTestTwice), online FDR alpha adjustment (alpha_investing, alpha_saffron, alpha_addis), local p-value adjustment (local_hommel_all_ps, local_simes, local_bh_all_ps), and reporting/visualization functions.