Simulation Framework for Experimental Designs and Inference Methods

An R6 class for benchmarking experimental designs and inference methods by Monte Carlo simulation. Each replication generates synthetic covariates and responses, runs every requested (design, inference) pair, and records point estimates, confidence intervals, and p-values. Raw and aggregated results are available through SimulationFrameworkReport.

Details

Covariates are drawn independently from $\mathrm{Uniform}(0, 1)$.

cond_exp_func_model = "linear": the base continuous signal is $y = X\beta$ where $\beta$ is evenly spaced from 1 to $-1$.
cond_exp_func_model = "nonlinear": the Friedman (1991) function $10\sin(\pi x_1 x_2) + 20(x_3-0.5)^2 + 10x_4 + 5x_5$; requires $p \ge 5$.

The continuous base signal is transformed to the scale appropriate for response_type. Treatment effects are applied per-subject: additive on the linear/logit/ordinal scale, log-multiplicative for count and survival.

For each (design, inference) pair the framework runs whichever of the following are supported by the inference class:

asymptotic (InferenceAsymp subclasses): Wald CI and p-value.
bootstrap (InferenceNonParamBootstrap subclasses): percentile CI and p-value.
randomisation (InferenceRand subclasses): p-value; additionally a test-inversion CI for continuous, proportion, and count response types (InferenceRandCI subclasses).

Incompatible (design, inference) pairs (e.g.\ a KK-specific inference class with a non-KK design) are silently skipped via tryCatch.

Reported summary metrics include:

MSE: $\overline{(\hat\beta_T - \beta_T)^2}$ over reps with a finite point estimate.
coverage: proportion of reps where $\beta_T$ lies inside the CI (NA when no CI is available for that inference type).
power: proportion of p-values $< \alpha$; equals the empirical type-I error rate when betaT = 0.

Methods

Public methods

SimulationFramework$new()
SimulationFramework$run()
SimulationFramework$get_all_intermediate_data()
SimulationFramework$clear_all_intermediate_data_and_gc()
SimulationFramework$clone()

`SimulationFramework$new()`

Create a new SimulationFramework object that stores simulation design settings, response-generation settings, inference methods, and replication controls.

Usage

SimulationFramework$new(
  response_type,
  design_classes_and_params = NULL,
  inference_classes_and_params = NULL,
  n = 100L,
  p = 5L,
  cond_exp_func_model = "linear",
  Nrep_W = 100L,
  Nrep_Y_w = 1L,
  betaT = 1,
  alpha = 0.05,
  B_boot = 201L,
  r_rand = 201L,
  pval_epsilon = 0.02,
  sd_noise = 1,
  n_ordinal_levels = 4L,
  proportion_epsilon = 1e-06,
  phi_proportion = 100,
  k_survival = 2,
  incidence_clamp = 1e-09,
  proportion_clamp = 1e-09,
  count_clamp = 1e-09,
  survival_clamp = 1e-09,
  survival_min_time = 0.1,
  count_min_rate = 0L,
  count_shift = 0,
  norm_sq_beta_vec = 1,
  X_mat = NULL,
  num_cores = 1L,
  seed = NULL,
  cov_draw_method = stats::rnorm,
  cov_draw_method_args = list(mean = 0, sd = 1),
  random_X_draws = TRUE,
  prob_censoring = 0.25,
  custom_replication_data_generator = NULL,
  custom_apply_treatment_and_noise = NULL,
  make_estimand_fn = NULL,
  dgp_params = list(),
  custom_dgp = NULL,
  verbose = TRUE,
  keep_all_intermediate_data = FALSE,
  turn_off_asserts_for_speed = TRUE,
  inference_types_and_params = NULL,
  results_filename = "simulation_framework_results.csv.bz2",
  continue_from_last_result_row = TRUE,
  reuse_cache = TRUE,
  stop_on_error = TRUE,
  save_to_disk_every_n_rep = 25L,
  save_model_control_fits = TRUE
)

Arguments

response_type

(required) Character scalar or vector. The type of outcome variable. One of "continuous", "incidence", "proportion", "count", "survival", "ordinal".

design_classes_and_params

NULL (default) or a list describing design classes and optional constructor parameters. Unnamed R6 class generators use default parameters, for example list(DesignSeqOneByOneKK21, DesignFixedBernoulli). Named entries use the entry name as the design class and the value as the parameter list, for example list(DesignSeqOneByOneUrn = list(alpha = 2, beta = 2)). Duplicate named entries are allowed for repeated designs with different parameters. Each generator must be constructable with only response_type and n plus any extra params supplied in this list. NULL uses the package's standard design set. Designs requiring strata_cols, cluster_col, or factors have sensible defaults auto-injected (first covariate column; second for cluster_col; list(treatment=2) for factors) when not supplied in the parameter list. Example:

design_classes_and_params = list(
  DesignSeqOneByOneKK21 = list(lambda = 0.5, t_0_pct = 0.1),
  DesignSeqOneByOneUrn  = list(alpha  = 2,   beta    = 2),
  DesignFixedBernoulli  # default params
)

Commonly useful design constructor parameters:

lambda: Matching-weight decay for KK14 / KK21 / KK21stepwise.
t_0_pct: Burn-in fraction for KK14 / KK21 / KK21stepwise.
morrison: Logical; Morrison correction for KK14.
alpha, beta: Shape parameters for DesignSeqOneByOneUrn.
preferred_num_bins_for_continuous_covariate: Bin count for DesignFixedBlocking and DesignFixedBlockedCluster.
B_target: Target number of blocks for DesignFixedBlocking.

inference_classes_and_params

NULL (default) or a list describing inference classes and optional constructor parameters. Unnamed R6 class generators use default parameters, for example list(InferenceContinOLS, InferenceContinKKOLSIVWC). Named entries use the entry name as the inference class and the value as the constructor parameter list, for example list(InferenceContinOLS = list(max_resample_attempts = 25L)). Duplicate named entries are allowed for repeated inference classes with different parameters. Supplied parameters must be accepted by the inference class constructor. NULL selects a curated set for the given response_type: several universal classes that work with any design, plus representative KK-specific classes (silently skipped for non-KK designs at runtime).

n

Integer scalar or vector. Sample size per simulation replication. Default 100.

p

Integer scalar or vector. Number of covariates. Must be $\ge 5$ when cond_exp_func_model = "nonlinear". Default 5.

cond_exp_func_model

Character scalar or vector. How the latent continuous signal is constructed before transformation to the response_type scale.

"linear": Linear combination $X\beta$ with coefficients evenly spaced from 1 to $-1$.
"nonlinear": Friedman (1991) function $10\sin(\pi x_1 x_2)+20(x_3-0.5)^2+10x_4+5x_5$; requires $p \ge 5$.

Default "linear".

Nrep_W

Positive integer. Number of treatment-assignment draws (w-reps). Each w-rep generates a fresh covariate matrix X and a new treatment assignment vector w. Default 100L.

Nrep_Y_w

Positive integer. Number of draws of the response per draw of w, the allocation vector. For each w-rep, Nrep_Y_w independent response vectors y are drawn from the same (X, w). The effective total number of replications recorded is Nrep_W * Nrep_Y_w. Default 1 (standard behaviour: one outcome draw per allocation-vector draw).

betaT

Numeric scalar or vector. True treatment effect added to treated subjects' outcomes. The scale is response-type specific: additive for continuous, proportion, and ordinal; on the logit scale for incidence; log-multiplicative for count and survival. Default 1. Set betaT = 0 to check type-I error.

alpha

Numeric in $(0,1)$. Significance level used for all confidence intervals and for computing power ($p < \alpha$). Default 0.05.

B_boot

Positive integer. Bootstrap resamples per CI / p-value call. Default 201.

r_rand

Positive integer. Randomisation draws per rand p-value call, and per bisection step of the rand CI. Default 201.

pval_epsilon

Numeric. Bisection convergence tolerance for randomisation-based CIs (compute_rand_confidence_interval). Default 0.02.

sd_noise

Numeric $> 0$. Standard deviation of independent Gaussian noise added to each subject's outcome. Default 1.

n_ordinal_levels

Positive integer. Number of ordinal categories when response_type = "ordinal". Default 4L.

proportion_epsilon

Numeric scalar. Small value added to proportion base responses to avoid 0 and 1. Default 1e-6.

phi_proportion

Positive numeric scalar. Precision parameter for beta-distributed observed proportion outcomes. The beta mean is y_linear_model[i] + betaT * w[i]. Default 100.

k_survival

Positive numeric scalar. Scale parameter passed to the Weibull draw for observed survival outcomes. Default 2.

incidence_clamp

Numeric scalar in $(0, 0.5)$. Clamp applied to the Bernoulli probability for observed incidence outcomes. Default 1e-9.

proportion_clamp

Numeric scalar in $(0, 0.5)$. Clamp applied to the beta mean for observed proportion outcomes. Default 1e-9.

count_clamp

Positive numeric scalar. Minimum Poisson mean for observed count outcomes. Default 1e-9.

survival_clamp

Positive numeric scalar. Minimum Weibull shape for observed survival outcomes. Default 1e-9.

survival_min_time

Numeric scalar. Minimum survival time and shift for base responses. Default 0.1.

count_min_rate

Integer scalar. Minimum baseline rate for count responses. Default 0L.

count_shift

Numeric scalar. Constant added to counts after zero-centering for base responses. Default 0.

norm_sq_beta_vec

Positive numeric scalar. The desired squared Euclidean norm of the latent linear coefficient vector $\beta$. The generated vector is scaled to match this norm. Default 1.

X_mat

Numeric matrix of dimensions n x p, or NULL (default). If provided, these fixed covariates are used for every replication. In this case, cov_draw_method must be NULL.

num_cores

Positive integer. Number of worker processes for parallel execution of Monte Carlo replications. Note that when num_cores > 1, parallelization *within* individual inference routines (e.g. bootstrap, randomization) is automatically disabled to prevent thread oversubscription.

Unix/Linux (recommended): A makeForkCluster pool is created once at run() start. Workers inherit all pre-generated design and SE caches via copy-on-write with zero serialization overhead. Parallelism operates at the replication level: num_cores replications run simultaneously, each executing all DGP cells serially. This eliminates the per-batch dispatch overhead that would arise from cycling through cells within every replication, and keeps all cores fully subscribed regardless of the number of DGP cells. For best performance, run on a Unix/Linux machine and set num_cores to the number of physical cores available.

Non-Unix (Windows/macOS): mirai daemons are used when available. Every active (replication, DGP-cell) pair is one work unit and num_cores units are kept in flight continuously, so all cores stay subscribed even when the grid has fewer DGP cells than cores. Cell state is pushed to the daemons once at run() start rather than re-serialized per dispatch. If mirai is not installed, execution falls back to serial with a warning. Default 1.

seed

Integer or NULL (default). Random seed for the entire simulation run.

cov_draw_method

A function used to draw n * p i.i.d. covariate values for every replication. The function must accept the total number of values as its first argument, followed by arguments in cov_draw_method_args. Default stats::rnorm. Must be NULL when X_mat is supplied.

cov_draw_method_args

Named list of additional arguments forwarded to cov_draw_method beyond the sample-size first argument. Default is list(mean = 0, sd = 1).

random_X_draws

Logical. If TRUE (default), a new set of covariates is drawn for every single replication. If FALSE, one set is drawn per (n, p) cell and shared across its replications.

prob_censoring

Numeric in $[0,1]$. Per-subject independent censoring probability; applied only when response_type = "survival". Default 0.25.

custom_replication_data_generator

Optional function for custom replication data. When supplied, it is called as fn(state, rep) and must return a list containing at least X and y_linear_model. Any additional fields in the returned list (e.g. latent frailty draws) are passed forward as rep_data to custom_apply_treatment_and_noise and the function built by make_estimand_fn.

custom_apply_treatment_and_noise

Optional function for custom response generation. Signature: fn(y_linear_model, w, rep_data, state). w is in {-1, +1} format; convert with (w+1)/2 for {0,1} semantics. rep_data is the full list returned by custom_replication_data_generator (or NULL for the standard path). Must return a list with components y and dead. Three-argument functions fn(y_linear_model, w, state) are still accepted for backwards compatibility.

make_estimand_fn

Optional factory function for a custom true estimand. Signature: fn(beta_T), returning a function with signature fn(y_linear_model, X, w, rep_data, state). Called once per grid cell with that cell's beta_T so the returned estimand function is always tied to the right effect size (important when betaT is a vector of multiple values). The returned function is invoked once per design class per replication after the design completes, so w (in {-1, +1} format) and X reflect the realized assignment. Must return a numeric scalar. When supplied, its return value is used as the ground truth for all inference classes (overriding the is_mean_diff gate). Three-argument functions fn(y_linear_model, state) are still accepted for backwards compatibility (they will not receive X or w). Default NULL uses beta_T directly as the ground truth.

dgp_params

Optional named list of DGP configuration values (e.g. list(frailty_dist = "gamma", censoring_rate = 0.8)). Injected into state as state\$dgp_params and accessible in all three custom-DGP hooks. Recommended over using closures to pass DGP parameters.

custom_dgp

Optional function for a fully custom DGP. Signature: fn(n, p, rep, state) returning a list with components X (data.frame, n x p), w (integer vector in {0, 1}, length n), y (numeric, length n), dead (integer {0,1} or NULL for non-survival), true_estimand (numeric scalar, optional). When supplied, the design class acts as a data container only; it does not run its own randomization or matching. Requires a fixed design class (not DesignSeqOneByOne variants). Cannot be combined with custom_replication_data_generator or custom_apply_treatment_and_noise.

verbose

Logical. If TRUE, prints a message for every replication and for every (design, inference) pair that is skipped due to an error. Default TRUE.

keep_all_intermediate_data

Logical. If TRUE, the framework saves the instantiated design and inference objects for every replication. These can be retrieved after the run using $get_all_intermediate_data(). Warning: this can consume a lot of memory for many replications. Default FALSE.

turn_off_asserts_for_speed

Logical. If TRUE (default), all checkmate assertions across the package are globally disabled during the simulation run to improve performance.

inference_types_and_params

NULL (default) or a named list from inference type to a named list of arguments for that type's function invocation. The list names control which inference outputs are computed. Valid names are "asymp_ci", "asymp_pval", "exact_ci", "exact_pval", "boot_ci", "boot_pval", "rand_ci", and "rand_pval". Each value must be a named list whose names are accepted by the corresponding inference function. NULL runs all eight types with default invocation arguments. Example:

inference_types_and_params = list(
  asymp_pval = list(delta = 0),
  boot_ci    = list(B = 99, type = "perc"),
  rand_pval  = list(r = 999, transform_responses = TRUE)
)

When no *_ci type is requested, coverage is omitted from SimulationFrameworkReport$summarize(). When no *_pval type is requested, power is omitted.

results_filename

Character scalar. The filename for the results file. Supported extensions are .csv and .csv.bz2. Default "simulation_framework_results.csv.bz2".

continue_from_last_result_row

Logical. If TRUE (default), the framework loads existing results from results_filename and skips previously completed replications.

reuse_cache

Logical. If TRUE (default), expensive pre-generated design / SE cache objects are loaded from disk when available. If FALSE, these cache objects are regenerated from scratch, but each regenerated object is still saved to disk for later restarts.

stop_on_error

Logical. If TRUE (default), any error raised during a simulation path aborts the run immediately. If FALSE, the framework records the error, skips the failing path, and continues with the remaining replications / design / inference combinations. Use $get_errors() after $run() to inspect the captured errors.

save_to_disk_every_n_rep

Positive integer. Results are flushed to the on-disk staging file only once every this many replications, and always after the final replication. Larger values reduce disk I/O overhead at the cost of losing more progress if the run is interrupted. Default 25L.

save_model_control_fits

Logical. If TRUE (default), after the design/SE cache is built, saves the per-subject model-implied potential outcomes under treatment and control as CSV files in a subfolder named <stem>_response_values/ (where <stem> is results_filename with its .csv/.csv.bz2 extension stripped) next to results_filename. One file is written per unique (response_type, cond_exp_func_model, n, p, betaT) cell. Only meaningful when random_X_draws = FALSE; silently skipped otherwise. Column names depend on response_type:

"continuous", "survival": columns yt and yc
"incidence", "proportion": columns pt and pc
"count": columns rt and rc

Default TRUE.

`SimulationFramework$run()`

Execute the configured simulation replications, run each requested design and inference method, collect estimates/p-values/CIs and errors, and return a SimulationFrameworkReport.

Usage

SimulationFramework$run()

Returns

The SimulationFramework object itself (invisibly).

`SimulationFramework$get_all_intermediate_data()`

Retrieve the stored intermediate data (design and inference objects) for every replication. Only available if keep_all_intermediate_data = TRUE was passed to the constructor.

Usage

SimulationFramework$get_all_intermediate_data()

Returns

A nested list containing the intermediate data for each replication, or NULL if not recorded.

`SimulationFramework$clear_all_intermediate_data_and_gc()`

Release all stored intermediate data and invoke the garbage collector. Useful after inspecting intermediate results to free memory before further processing. Sets the internal store to NULL and calls gc().

Usage

SimulationFramework$clear_all_intermediate_data_and_gc()

Returns

The SimulationFramework object itself (invisibly).

`SimulationFramework$clone()`

The objects of this class are cloneable with this method.

Usage

SimulationFramework$clone(deep = FALSE)

Arguments

deep: Whether to make a deep clone.

Examples

# \donttest{
# Simple simulation with two designs and two inference methods.
# n/Nrep_W/num_boot/B_boot/r_rand are kept small so this example runs in a
# few seconds; a real simulation would use much larger values (this
# class's defaults, or larger still) for adequately precise estimates.
sim = SimulationFramework$new(
  response_type = "continuous",
  design_classes_and_params = list(
    DesignSeqOneByOneKK21 = list(lambda = 0.5, num_boot = 50L),
    DesignSeqOneByOneBernoulli = list()
  ),
  inference_classes_and_params = list(
    InferenceContinOLS = list(),
    InferenceContinKKOLSIVWC = list()
  ),
  n = 20, p = 3, Nrep_W = 2L, betaT = 1, B_boot = 50L, r_rand = 101L,
  results_filename = tempfile(fileext = ".csv.bz2"),
  continue_from_last_result_row = FALSE
)
sim$run()
#> simulations: CEF_mod=linear  n=20  p=3  Nrep=2  betaT=1 designs=2 inferences=2 num_cores=1
report = SimulationFrameworkReport$new(sim)
report$summarize()
#> Key: <response_type, cond_exp_func_model, n, p, betaT, design, inference, inference_type, simulation_mode>
#>     response_type cond_exp_func_model     n     p betaT
#>            <char>              <char> <int> <int> <num>
#>  1:    continuous              linear    20     3     1
#>  2:    continuous              linear    20     3     1
#>  3:    continuous              linear    20     3     1
#>  4:    continuous              linear    20     3     1
#>  5:    continuous              linear    20     3     1
#>  6:    continuous              linear    20     3     1
#>  7:    continuous              linear    20     3     1
#>  8:    continuous              linear    20     3     1
#>  9:    continuous              linear    20     3     1
#> 10:    continuous              linear    20     3     1
#> 11:    continuous              linear    20     3     1
#> 12:    continuous              linear    20     3     1
#> 13:    continuous              linear    20     3     1
#> 14:    continuous              linear    20     3     1
#> 15:    continuous              linear    20     3     1
#> 16:    continuous              linear    20     3     1
#> 17:    continuous              linear    20     3     1
#> 18:    continuous              linear    20     3     1
#>                                               design                inference
#>                                               <char>                   <char>
#>  1:                       DesignSeqOneByOneBernoulli       InferenceContinOLS
#>  2:                       DesignSeqOneByOneBernoulli       InferenceContinOLS
#>  3:                       DesignSeqOneByOneBernoulli       InferenceContinOLS
#>  4:                       DesignSeqOneByOneBernoulli       InferenceContinOLS
#>  5:                       DesignSeqOneByOneBernoulli       InferenceContinOLS
#>  6:                       DesignSeqOneByOneBernoulli       InferenceContinOLS
#>  7: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L) InferenceContinKKOLSIVWC
#>  8: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L) InferenceContinKKOLSIVWC
#>  9: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L) InferenceContinKKOLSIVWC
#> 10: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L) InferenceContinKKOLSIVWC
#> 11: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L) InferenceContinKKOLSIVWC
#> 12: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L) InferenceContinKKOLSIVWC
#> 13: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L)       InferenceContinOLS
#> 14: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L)       InferenceContinOLS
#> 15: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L)       InferenceContinOLS
#> 16: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L)       InferenceContinOLS
#> 17: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L)       InferenceContinOLS
#> 18: DesignSeqOneByOneKK21 (lambda=0.5, num_boot=50L)       InferenceContinOLS
#>     inference_type simulation_mode        MSE n_est coverage n_cov ci_length
#>             <char>          <char>      <num> <int>    <num> <int>     <num>
#>  1:       asymp_ci        standard 0.01918429     2        1     2  2.473566
#>  2:     asymp_pval        standard 0.01918429     2       NA     0        NA
#>  3:        boot_ci        standard 0.01918429     2       NA     0        NA
#>  4:      boot_pval        standard 0.01918429     2       NA     0        NA
#>  5:        rand_ci        standard 0.01918429     2        1     2  3.958797
#>  6:      rand_pval        standard 0.01918429     2       NA     0        NA
#>  7:       asymp_ci        standard 0.26173836     2        1     2  2.563949
#>  8:     asymp_pval        standard 0.26173836     2       NA     0        NA
#>  9:        boot_ci        standard 0.26173836     2       NA     0        NA
#> 10:      boot_pval        standard 0.26173836     2       NA     0        NA
#> 11:        rand_ci        standard 0.26173836     2       NA     0        NA
#> 12:      rand_pval        standard 0.26173836     2       NA     0        NA
#> 13:       asymp_ci        standard 0.08867730     2        1     2  1.782832
#> 14:     asymp_pval        standard 0.08867730     2       NA     0        NA
#> 15:        boot_ci        standard 0.08867730     2       NA     0        NA
#> 16:      boot_pval        standard 0.08867730     2       NA     0        NA
#> 17:        rand_ci        standard 0.08867730     2        1     2  2.369487
#> 18:      rand_pval        standard 0.08867730     2       NA     0        NA
#>     coverage_pval power n_pow  size n_size size_pval            design_params
#>             <num> <num> <int> <num>  <int>     <num>                   <char>
#>  1:             1    NA     0    NA      0        NA                         
#>  2:            NA     0     2    NA      0        NA                         
#>  3:            NA    NA     0    NA      0        NA                         
#>  4:            NA    NA     0    NA      0        NA                         
#>  5:             1    NA     0    NA      0        NA                         
#>  6:            NA     0     2    NA      0        NA                         
#>  7:             1    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#>  8:            NA     0     2    NA      0        NA lambda=0.5, num_boot=50L
#>  9:            NA    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 10:            NA    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 11:            NA    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 12:            NA    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 13:             1    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 14:            NA     0     2    NA      0        NA lambda=0.5, num_boot=50L
#> 15:            NA    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 16:            NA    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 17:             1    NA     0    NA      0        NA lambda=0.5, num_boot=50L
#> 18:            NA     1     2    NA      0        NA lambda=0.5, num_boot=50L
#>     inference_params inference_type_params
#>               <char>                <char>
#>  1:                                       
#>  2:                                       
#>  3:                                       
#>  4:                                       
#>  5:                                       
#>  6:                                       
#>  7:                                       
#>  8:                                       
#>  9:                                       
#> 10:                                       
#> 11:                                       
#> 12:                                       
#> 13:                                       
#> 14:                                       
#> 15:                                       
#> 16:                                       
#> 17:                                       
#> 18:                                       
# }