Skip to contents

Calculate the error and success proportions of tests for a single iteration

calc_errs.Rd

This function takes output from findBlocks or an equivalent bottom-up testing function such as adjust_block_tests and returns the proportions of errors made. To use this function the input to findBlocks must include a column containing a true block-level effect. Repeated uses of this function allow us to assess false discovery rates and family wise error rates among other metrics of testing success.

Usage

calc_errs(
  testobj,
  truevar_name,
  trueeffect_tol = .Machine$double.eps,
  blockid = "bF",
  thealpha = 0.05,
  fwer = FALSE,
  return_details = FALSE
)

Arguments

testobj

Is an object arising from findBlocks or adjust_block_tests. It will contain block-level results.

truevar_name

Is a string indicating the name of the variable containing the true underlying causal effect (at the block level).

trueeffect_tol

Is the smallest effect size below which we consider the effect to be zero (by default is it floating point zero).

blockid

A character name of the column in idat and bdat indicating the block.

thealpha

Is the error rate for a given test (for cases where alphafn is NULL, or the starting alpha for alphafn not null)

fwer

Indicates that we are trying to control FWER. Right now, we do this by default in report_detections but this indicates when we should be looking at FDR

return_details

TRUE means that the function should return a list of the original data ("detobj"), a summary of the results ("detresults"), a node level dataset ("detnodes"), and a copy of the original object that was provided as input. Default here is FALSE. Only use TRUE when not using simulations.

Value

If hit means that $p $ for a given block or group of blocks where testing has stopped, allnull means that all of the effects in the group of blocks (or the single block) are zero, anynotnull means that at least one block has a non-zero effect, then this is the code that returns the different basic descriptions of errors.


deterrs <- detnodes[, .(
 nreject = sum(hit),
 naccept = sum(1 - hit),
 prop_reject = mean(hit),
 prop_accept = mean(1 - hit), # or 1-prop_reject
 # rejections of false null /detections of true non-null
 # (if any of the blocks have an effect, then we count this as a correct rejection or correct detection)
 true_pos_prop = mean(hit * anynotnull),
 # If we reject but *all* of the blocks have no effects, this is a false positive error
 # If we reject but only one of them has no effects but the other has effects,
 # then this is not an error --- but a correct detection
 false_pos_prop = mean(hit * allnull),
 # If we do not reject and all blocks are truly null, then we have no error.
 true_neg_prop = mean((1 - hit) * allnull),
 # If we do not reject/detect and at least one of the blocks actually has an effect, we have
 # a false negative error --- a failure to detect the truth
 false_neg_prop = mean((1 - hit) * anynotnull),
 # Now look at false and true discoveries: false rejections as a proportion of rejections
 false_disc_prop = sum(hit * allnull) / max(1, sum(hit)),
 true_disc_prop = sum(hit * anynotnull) / max(1, sum(hit)),
 true_nondisc_prop = sum((1 - hit) * allnull) / max(1, sum(1 - hit)),
 false_nondisc_prop = sum((1 - hit) * anynotnull) / max(1, sum(1 - hit)),
 meangrpsize = mean(grpsize),
 medgrpsize = median(grpsize)
)]

We also summarize the average treatment effects in the blocks (or groups of blocks) among blocks where the null hypothesis of no effects has been rejected and where it has not been rejected.

  • "nreject" Number of tests with $p $

  • "naccept" Number of tests with $p > $

  • "prop_reject" Proportion of tests with $p $ out of all tests completed

  • "prop_accept" Proportion of tests with $p > $ out of all tests completed

  • "true_pos_prop" Proportion of tests with $p $ out of all tests of true null hypotheses (rejections of false null hypotheses)

  • "false_pos_prop" Proportion of tests with $p $ out of all tests of false null hypotheses (rejections of true null hypotheses)

  • "true_neg_prop"

  • "false_neg_prop"

  • "false_disc_prop"

  • "true_disc_prop"

  • "true_nondisc_prop"

  • "false_nondisc_prop"

  • "meangrpsize"

  • "medgrpsize"

  • "hit1"

  • "minate1"

  • "meanate1"

  • "medate1"

  • "maxate1"

  • "hit0"

  • "minate0"

  • "meanate0"

  • "medate0"

  • "maxate0"