Skip to contents

Design Operating Characteristics

evaluating_properties.Rmd

Design operating characteristics to evaluate the properties of a particular design via simulation the function mams.sim can be used. It allows for flexible numbers of subjects per arm and stage in the form of a J × (K + 1) matrix nMat. In addition to the upper and lower boundaries (u and l), a vector of true success probabilities (pv) is required (or alternatively a vector of true effect sizes (deltav) and a standard deviation (sd)). The parameter ptest allows to specify rejection of which hypotheses should be counted in the power calculation. We evaluate the properties of the two-stage design m2 under the global null hypothesis (i.e., a true effect size of p=0.5p = 0.5 or δ=0δ = 0 for all treatments) with 100,000 simulation runs.

m2sim <- mams.sim(nsim = 1e5, nMat = t(m2$n * m2$rMat), u = m2$u, l = m2$l, 
                  pv = rep(0.5, 3), ptest = 1:2)

m2sim

The probability of rejecting at least one hypothesis is 0.050, and since we simulated under the global H0H_0, this corresponds to a FWER of 5% as desired. The power to reject the first hypothesis when it has the largest estimated effect is 0.016, and the power to reject either H1H_1 or H2H_2 or both of them (as specified by ptest = 1:2) is 0.034. The expected number of patients required for the trial under the global H0H_0 is 244.7 in contrast to the maximum required of 380. The function mams.sim is also useful to simulate and compare expected sample sizes of different designs. We illustrate this for the designs poc, obf and tri (whose boundaries are shown in Multi-stage design) under the LFC of the alternative, i.e., one treatment’s effect size equals.

Design Minimum Maximum Expected (LFC) Expected (H0)
Pocock 132 396 232.2 385.6
O'Brien-Fleming 112 336 259.1 322.0
Triangular 136 408 217.8 222.1

p=0.65p = 0.65 whereas the effect sizes for all other treatments are equal to p0=0.55p_0 = 0.55, using 100,000 simulation runs.

pocsim <- mams.sim(nsim = 1e5, nMat = t(poc$n * poc$rMat), u = poc$u, l = poc$l, 
                   pv = c(0.65, rep(0.55, 2)), ptest = 1)
obfsim <- mams.sim(nsim = 1e5, nMat = t(obf$n * obf$rMat), u = obf$u, l = obf$l, 
                   pv = c(0.65, rep(0.55, 2)), ptest = 1)
trisim <- mams.sim(nsim = 1e5, nMat = t(tri$n * tri$rMat), u = tri$u, l = tri$l, 
                   pv = c(0.65, rep(0.55, 2)), ptest = 1)

Similarly, we can obtain the design properties under the global null hypothesis by setting pv = rep(0.5, 3). Table summarizes minimum, maximum, and expected sample sizes of the three designs. We see that under both the LFC and the global H0H_0 the triangular design is expected to require the lowest number of patients. On the other hand, O’Brien-Fleming has the lowest minimum and maximum but the highest expected sample size under the LFC of all three designs. Under the global H0H_0 both the Pocock and O’Brien-Fleming designs have expected sample sizes that are very close to their respective maxima.