Reporting class for SimulationFramework results — SimulationFrameworkReport • EDI: Experimental Design and Inference

An R6 class for accessing and summarizing the results of a SimulationFramework run. It can be constructed either from a completed SimulationFramework object (via SimulationFrameworkReport$new(sim)) or by loading results from a previously saved CSV / CSV.BZ2 file (via SimulationFrameworkReport$new("path/to/results.csv")).

Details

When constructed from a SimulationFramework object all design/inference parameter metadata is preserved, so $summarize() can annotate each row with human-readable parameter strings. When constructed from a file only the raw results are available; parameter annotation columns will be empty strings.

Methods

Public methods

SimulationFrameworkReport$new()
SimulationFrameworkReport$get_results()
SimulationFrameworkReport$get_errors()
SimulationFrameworkReport$summarize()
SimulationFrameworkReport$print()
SimulationFrameworkReport$clone()

`SimulationFrameworkReport$new()`

Create a new SimulationFrameworkReport object that stores simulation results, captured errors, and summary helpers returned by SimulationFramework.

Usage

SimulationFrameworkReport$new(sim_or_filename, alpha = NULL)

Arguments

sim_or_filename: Either a completed SimulationFramework object or a character string giving the path to a .csv or .csv.bz2 results file written by SimulationFramework.
alpha: Numeric in $(0,1)$. Significance level for coverage and power calculations. When sim_or_filename is a SimulationFramework object and alpha is NULL (default), the framework's own alpha is used. When loading from a file, defaults to 0.05.

`SimulationFrameworkReport$get_results()`

Get the raw per-replication results.

Usage

SimulationFrameworkReport$get_results()

Returns

A data.table with one row per (replication, design, inference class, inference type).

`SimulationFrameworkReport$get_errors()`

Return all errors captured during the simulation run.

Usage

SimulationFrameworkReport$get_errors()

Returns

A list of named lists, one per captured error. Each element includes the simulation cell metadata, replication number, design / inference path, user-supplied parameters, error stage, and error message. Empty when constructed from a file.

`SimulationFrameworkReport$summarize()`

Aggregate and summarize simulation results.

Usage

SimulationFrameworkReport$summarize()

Returns

A data.table with one row per unique (response_type, cond_exp_func_model, n, p, betaT, design, inference, inference_type) combination. Columns include MSE, coverage, ci_length, and coverage_pval (when CI types were run; coverage_pval is the exact two-sided binomial test p-value of H0: true coverage = 1 - alpha), power (when betaT != 0 and p-value types were run), size and size_pval (when betaT == 0 and p-value types were run; size_pval is the exact two-sided binomial test p-value of H0: true size = alpha, suitable for multiplicity-corrected calibration checks across settings), and parameter annotation strings.

`SimulationFrameworkReport$print()`

Print a concise summary of the report.

Usage

SimulationFrameworkReport$print()

`SimulationFrameworkReport$clone()`

The objects of this class are cloneable with this method.

Usage

SimulationFrameworkReport$clone(deep = FALSE)

Arguments

deep: Whether to make a deep clone.

Examples

# \donttest{
sim <- SimulationFramework$new(
  response_type = "continuous",
  design_classes_and_params = list(DesignFixedBernoulli),
  inference_classes_and_params = list(InferenceAllSimpleAverageDiff),
  n = 20L, Nrep_W = 5L, betaT = 1,
  results_filename = tempfile(fileext = ".csv"),
  verbose = FALSE, continue_from_last_result_row = FALSE
)
sim$run()
report <- SimulationFrameworkReport$new(sim)
report$get_results()
#>     response_type   rep cond_exp_func_model     n     p betaT
#>            <char> <int>              <char> <int> <int> <num>
#>  1:    continuous     1              linear    20     5     1
#>  2:    continuous     1              linear    20     5     1
#>  3:    continuous     1              linear    20     5     1
#>  4:    continuous     1              linear    20     5     1
#>  5:    continuous     1              linear    20     5     1
#>  6:    continuous     1              linear    20     5     1
#>  7:    continuous     2              linear    20     5     1
#>  8:    continuous     2              linear    20     5     1
#>  9:    continuous     2              linear    20     5     1
#> 10:    continuous     2              linear    20     5     1
#> 11:    continuous     2              linear    20     5     1
#> 12:    continuous     2              linear    20     5     1
#> 13:    continuous     3              linear    20     5     1
#> 14:    continuous     3              linear    20     5     1
#> 15:    continuous     3              linear    20     5     1
#> 16:    continuous     3              linear    20     5     1
#> 17:    continuous     3              linear    20     5     1
#> 18:    continuous     3              linear    20     5     1
#> 19:    continuous     4              linear    20     5     1
#> 20:    continuous     4              linear    20     5     1
#> 21:    continuous     4              linear    20     5     1
#> 22:    continuous     4              linear    20     5     1
#> 23:    continuous     4              linear    20     5     1
#> 24:    continuous     4              linear    20     5     1
#> 25:    continuous     5              linear    20     5     1
#> 26:    continuous     5              linear    20     5     1
#> 27:    continuous     5              linear    20     5     1
#> 28:    continuous     5              linear    20     5     1
#> 29:    continuous     5              linear    20     5     1
#> 30:    continuous     5              linear    20     5     1
#>     response_type   rep cond_exp_func_model     n     p betaT
#>            <char> <int>              <char> <int> <int> <num>
#>                   design                     inference inference_type  estimate
#>                   <char>                        <char>         <char>     <num>
#>  1: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 0.9914813
#>  2: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 0.9914813
#>  3: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 0.9914813
#>  4: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 0.9914813
#>  5: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 0.9914813
#>  6: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 0.9914813
#>  7: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 1.2660390
#>  8: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 1.2660390
#>  9: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 1.2660390
#> 10: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 1.2660390
#> 11: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 1.2660390
#> 12: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 1.2660390
#> 13: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 0.8702131
#> 14: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 0.8702131
#> 15: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 0.8702131
#> 16: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 0.8702131
#> 17: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 0.8702131
#> 18: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 0.8702131
#> 19: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 1.7178692
#> 20: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 1.7178692
#> 21: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 1.7178692
#> 22: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 1.7178692
#> 23: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 1.7178692
#> 24: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 1.7178692
#> 25: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 1.7259847
#> 26: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 1.7259847
#> 27: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 1.7259847
#> 28: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 1.7259847
#> 29: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 1.7259847
#> 30: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 1.7259847
#>                   design                     inference inference_type  estimate
#>                   <char>                        <char>         <char>     <num>
#>            ci_lo    ci_hi        pval true_estimand simulation_mode
#>            <num>    <num>       <num>         <num>          <char>
#>  1:           NA       NA 0.128770011             1        standard
#>  2: -0.320200162 2.303163          NA             1        standard
#>  3:           NA       NA 0.163504699             1        standard
#>  4: -0.308273590 2.038567          NA             1        standard
#>  5:           NA       NA 0.129353234             1        standard
#>  6: -0.224185400 2.342222          NA             1        standard
#>  7:           NA       NA 0.052472938             1        standard
#>  8: -0.015636253 2.547714          NA             1        standard
#>  9:           NA       NA 0.054257751             1        standard
#> 10: -0.035412353 2.378094          NA             1        standard
#> 11:           NA       NA 0.049751244             1        standard
#> 12:  0.001264556 2.580090          NA             1        standard
#> 13:           NA       NA 0.257961011             1        standard
#> 14: -0.699835970 2.440262          NA             1        standard
#> 15:           NA       NA 0.200664920             1        standard
#> 16: -0.358387166 2.319841          NA             1        standard
#> 17:           NA       NA 0.189054726             1        standard
#> 18: -1.070809858 2.487732          NA             1        standard
#> 19:           NA       NA 0.023288859             1        standard
#> 20:  0.267453445 3.168285          NA             1        standard
#> 21:           NA       NA 0.009950249             1        standard
#> 22:  0.600117631 3.082132          NA             1        standard
#> 23:           NA       NA 0.019900498             1        standard
#> 24:  0.282011869 3.135079          NA             1        standard
#> 25:           NA       NA 0.029963586             1        standard
#> 26:  0.187384008 3.264585          NA             1        standard
#> 27:           NA       NA 0.051693255             1        standard
#> 28: -0.053305300 2.918371          NA             1        standard
#> 29:           NA       NA 0.059701493             1        standard
#> 30: -0.178900001 3.154648          NA             1        standard
#>            ci_lo    ci_hi        pval true_estimand simulation_mode
#>            <num>    <num>       <num>         <num>          <char>
report$summarize()
#> Key: <response_type, cond_exp_func_model, n, p, betaT, design, inference, inference_type, simulation_mode>
#>    response_type cond_exp_func_model     n     p betaT               design
#>           <char>              <char> <int> <int> <num>               <char>
#> 1:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 2:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 3:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 4:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 5:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 6:    continuous              linear    20     5     1 DesignFixedBernoulli
#>                        inference inference_type simulation_mode       MSE n_est
#>                           <char>         <char>          <char>     <num> <int>
#> 1: InferenceAllSimpleAverageDiff       asymp_ci        standard 0.2260168     5
#> 2: InferenceAllSimpleAverageDiff     asymp_pval        standard 0.2260168     5
#> 3: InferenceAllSimpleAverageDiff        boot_ci        standard 0.2260168     5
#> 4: InferenceAllSimpleAverageDiff      boot_pval        standard 0.2260168     5
#> 5: InferenceAllSimpleAverageDiff        rand_ci        standard 0.2260168     5
#> 6: InferenceAllSimpleAverageDiff      rand_pval        standard 0.2260168     5
#>    coverage n_cov ci_length coverage_pval power n_pow  size n_size size_pval
#>       <num> <int>     <num>         <num> <num> <int> <num>  <int>     <num>
#> 1:        1     5  2.860969             1    NA     0    NA      0        NA
#> 2:       NA     0        NA            NA   0.4     5    NA      0        NA
#> 3:        1     5  2.578453             1    NA     0    NA      0        NA
#> 4:       NA     0        NA            NA   0.2     5    NA      0        NA
#> 5:        1     5  2.978078             1    NA     0    NA      0        NA
#> 6:       NA     0        NA            NA   0.4     5    NA      0        NA
#>    design_params inference_params inference_type_params
#>           <char>           <char>                <char>
#> 1:                                                     
#> 2:                                                     
#> 3:                                                     
#> 4:                                                     
#> 5:                                                     
#> 6:                                                     
# }