Reporting class for SimulationFramework results — SimulationFrameworkReport • EDI

An R6 class for accessing and summarizing the results of a SimulationFramework run. It can be constructed either from a completed SimulationFramework object (via SimulationFrameworkReport$new(sim)) or by loading results from a previously saved CSV / CSV.BZ2 file (via SimulationFrameworkReport$new("path/to/results.csv")).

Details

When constructed from a SimulationFramework object all design/inference parameter metadata is preserved, so $summarize() can annotate each row with human-readable parameter strings. When constructed from a file only the raw results are available; parameter annotation columns will be empty strings.

Methods

Public methods

SimulationFrameworkReport$new()
SimulationFrameworkReport$get_results()
SimulationFrameworkReport$get_errors()
SimulationFrameworkReport$summarize()
SimulationFrameworkReport$print()
SimulationFrameworkReport$clone()

`SimulationFrameworkReport$new()`

Create a new SimulationFrameworkReport object that stores simulation results, captured errors, and summary helpers returned by SimulationFramework.

Usage

SimulationFrameworkReport$new(sim_or_filename, alpha = NULL)

Arguments

sim_or_filename: Either a completed SimulationFramework object or a character string giving the path to a .csv or .csv.bz2 results file written by SimulationFramework.
alpha: Numeric in $(0,1)$. Significance level for coverage and power calculations. When sim_or_filename is a SimulationFramework object and alpha is NULL (default), the framework's own alpha is used. When loading from a file, defaults to 0.05.

`SimulationFrameworkReport$get_results()`

Get the raw per-replication results.

Usage

SimulationFrameworkReport$get_results()

Returns

A data.table with one row per (replication, design, inference class, inference type).

`SimulationFrameworkReport$get_errors()`

Return all errors captured during the simulation run.

Usage

SimulationFrameworkReport$get_errors()

Returns

A list of named lists, one per captured error. Each element includes the simulation cell metadata, replication number, design / inference path, user-supplied parameters, error stage, and error message. Empty when constructed from a file.

`SimulationFrameworkReport$summarize()`

Aggregate and summarize simulation results.

Usage

SimulationFrameworkReport$summarize()

Returns

A data.table with one row per unique (response_type, cond_exp_func_model, n, p, betaT, design, inference, inference_type) combination. Columns include MSE, coverage, ci_length, and coverage_pval (when CI types were run; coverage_pval is the exact two-sided binomial test p-value of H0: true coverage = 1 - alpha), power (when betaT != 0 and p-value types were run), size and size_pval (when betaT == 0 and p-value types were run; size_pval is the exact two-sided binomial test p-value of H0: true size = alpha, suitable for multiplicity-corrected calibration checks across settings), and parameter annotation strings.

`SimulationFrameworkReport$print()`

Print a concise summary of the report.

Usage

SimulationFrameworkReport$print()

`SimulationFrameworkReport$clone()`

The objects of this class are cloneable with this method.

Usage

SimulationFrameworkReport$clone(deep = FALSE)

Arguments

deep: Whether to make a deep clone.

Examples

# \donttest{
sim <- SimulationFramework$new(
  response_type = "continuous",
  design_classes_and_params = list(DesignFixedBernoulli),
  inference_classes_and_params = list(InferenceAllSimpleAverageDiff),
  n = 20L, Nrep_W = 5L, betaT = 1,
  results_filename = tempfile(fileext = ".csv"),
  verbose = FALSE, continue_from_last_result_row = FALSE
)
sim$run()
report <- SimulationFrameworkReport$new(sim)
report$get_results()
#>     response_type   rep cond_exp_func_model     n     p betaT
#>            <char> <int>              <char> <int> <int> <num>
#>  1:    continuous     1              linear    20     5     1
#>  2:    continuous     1              linear    20     5     1
#>  3:    continuous     1              linear    20     5     1
#>  4:    continuous     1              linear    20     5     1
#>  5:    continuous     1              linear    20     5     1
#>  6:    continuous     1              linear    20     5     1
#>  7:    continuous     2              linear    20     5     1
#>  8:    continuous     2              linear    20     5     1
#>  9:    continuous     2              linear    20     5     1
#> 10:    continuous     2              linear    20     5     1
#> 11:    continuous     2              linear    20     5     1
#> 12:    continuous     2              linear    20     5     1
#> 13:    continuous     3              linear    20     5     1
#> 14:    continuous     3              linear    20     5     1
#> 15:    continuous     3              linear    20     5     1
#> 16:    continuous     3              linear    20     5     1
#> 17:    continuous     3              linear    20     5     1
#> 18:    continuous     3              linear    20     5     1
#> 19:    continuous     4              linear    20     5     1
#> 20:    continuous     4              linear    20     5     1
#> 21:    continuous     4              linear    20     5     1
#> 22:    continuous     4              linear    20     5     1
#> 23:    continuous     4              linear    20     5     1
#> 24:    continuous     4              linear    20     5     1
#> 25:    continuous     5              linear    20     5     1
#> 26:    continuous     5              linear    20     5     1
#> 27:    continuous     5              linear    20     5     1
#> 28:    continuous     5              linear    20     5     1
#> 29:    continuous     5              linear    20     5     1
#> 30:    continuous     5              linear    20     5     1
#>     response_type   rep cond_exp_func_model     n     p betaT
#>            <char> <int>              <char> <int> <int> <num>
#>                   design                     inference inference_type  estimate
#>                   <char>                        <char>         <char>     <num>
#>  1: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 1.3313725
#>  2: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 1.3313725
#>  3: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 1.3313725
#>  4: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 1.3313725
#>  5: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 1.3313725
#>  6: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 1.3313725
#>  7: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 2.0657343
#>  8: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 2.0657343
#>  9: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 2.0657343
#> 10: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 2.0657343
#> 11: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 2.0657343
#> 12: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 2.0657343
#> 13: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 0.5363332
#> 14: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 0.5363332
#> 15: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 0.5363332
#> 16: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 0.5363332
#> 17: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 0.5363332
#> 18: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 0.5363332
#> 19: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 0.5276257
#> 20: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 0.5276257
#> 21: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 0.5276257
#> 22: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 0.5276257
#> 23: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 0.5276257
#> 24: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 0.5276257
#> 25: DesignFixedBernoulli InferenceAllSimpleAverageDiff     asymp_pval 1.2505601
#> 26: DesignFixedBernoulli InferenceAllSimpleAverageDiff       asymp_ci 1.2505601
#> 27: DesignFixedBernoulli InferenceAllSimpleAverageDiff      boot_pval 1.2505601
#> 28: DesignFixedBernoulli InferenceAllSimpleAverageDiff        boot_ci 1.2505601
#> 29: DesignFixedBernoulli InferenceAllSimpleAverageDiff      rand_pval 1.2505601
#> 30: DesignFixedBernoulli InferenceAllSimpleAverageDiff        rand_ci 1.2505601
#>                   design                     inference inference_type  estimate
#>                   <char>                        <char>         <char>     <num>
#>            ci_lo    ci_hi        pval true_estimand simulation_mode
#>            <num>    <num>       <num>         <num>          <char>
#>  1:           NA       NA 0.054313845             1        standard
#>  2: -0.027963799 2.690709          NA             1        standard
#>  3:           NA       NA 0.053102859             1        standard
#>  4: -0.009033398 2.724132          NA             1        standard
#>  5:           NA       NA 0.049751244             1        standard
#>  6: -0.346132622 2.589501          NA             1        standard
#>  7:           NA       NA 0.011044937             1        standard
#>  8:  0.555199071 3.576270          NA             1        standard
#>  9:           NA       NA          NA             1        standard
#> 10:  0.693890380 3.620432          NA             1        standard
#> 11:           NA       NA 0.009950249             1        standard
#> 12:  0.355010531 3.877089          NA             1        standard
#> 13:           NA       NA 0.475404836             1        standard
#> 14: -1.011710414 2.084377          NA             1        standard
#> 15:           NA       NA 0.459936372             1        standard
#> 16: -0.862310611 1.805308          NA             1        standard
#> 17:           NA       NA 0.487562189             1        standard
#> 18: -2.017808681 2.451940          NA             1        standard
#> 19:           NA       NA 0.466554099             1        standard
#> 20: -0.971487959 2.026739          NA             1        standard
#> 21:           NA       NA 0.480246683             1        standard
#> 22: -0.871356718 1.735937          NA             1        standard
#> 23:           NA       NA 0.487562189             1        standard
#> 24: -1.941884285 2.379758          NA             1        standard
#> 25:           NA       NA 0.025503988             1        standard
#> 26:  0.174346219 2.326774          NA             1        standard
#> 27:           NA       NA          NA             1        standard
#> 28:  0.263934358 2.245760          NA             1        standard
#> 29:           NA       NA 0.029850746             1        standard
#> 30: -1.033853424 3.534974          NA             1        standard
#>            ci_lo    ci_hi        pval true_estimand simulation_mode
#>            <num>    <num>       <num>         <num>          <char>
report$summarize()
#> Key: <response_type, cond_exp_func_model, n, p, betaT, design, inference, inference_type, simulation_mode>
#>    response_type cond_exp_func_model     n     p betaT               design
#>           <char>              <char> <int> <int> <num>               <char>
#> 1:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 2:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 3:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 4:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 5:    continuous              linear    20     5     1 DesignFixedBernoulli
#> 6:    continuous              linear    20     5     1 DesignFixedBernoulli
#>                        inference inference_type simulation_mode       MSE n_est
#>                           <char>         <char>          <char>     <num> <int>
#> 1: InferenceAllSimpleAverageDiff       asymp_ci        standard 0.3493004     5
#> 2: InferenceAllSimpleAverageDiff     asymp_pval        standard 0.3493004     5
#> 3: InferenceAllSimpleAverageDiff        boot_ci        standard 0.3493004     5
#> 4: InferenceAllSimpleAverageDiff      boot_pval        standard 0.3493004     5
#> 5: InferenceAllSimpleAverageDiff        rand_ci        standard 0.3493004     5
#> 6: InferenceAllSimpleAverageDiff      rand_pval        standard 0.3493004     5
#>    coverage n_cov ci_length coverage_pval power n_pow  size n_size size_pval
#>       <num> <int>     <num>         <num> <num> <int> <num>  <int>     <num>
#> 1:        1     5  2.797297             1    NA     0    NA      0        NA
#> 2:       NA     0        NA            NA   0.4     5    NA      0        NA
#> 3:        1     5  2.583289             1    NA     0    NA      0        NA
#> 4:       NA     0        NA            NA   0.0     3    NA      0        NA
#> 5:        1     5  3.963586             1    NA     0    NA      0        NA
#> 6:       NA     0        NA            NA   0.6     5    NA      0        NA
#>    design_params inference_params inference_type_params
#>           <char>           <char>                <char>
#> 1:                                                     
#> 2:                                                     
#> 3:                                                     
#> 4:                                                     
#> 5:                                                     
#> 6:                                                     
# }