Long time frames to detect the impact of changing COVID-19 measures, Canada, March to July 2020

Background Many countries have implemented population-wide interventions to control COVID-19, with varying extent and success. Many jurisdictions have moved to relax measures, while others have intensified efforts to reduce transmission. Aim We aimed to determine the time frame between a population-level change in COVID-19 measures and its impact on the number of cases. Methods We examined how long it takes for there to be a substantial difference between the number of cases that occur following a change in COVID-19 physical distancing measures and those that would have occurred at baseline. We then examined how long it takes to observe this difference, given delays and noise in reported cases. We used a susceptible-exposed-infectious-removed (SEIR)-type model and publicly available data from British Columbia, Canada, collected between March and July 2020. Results It takes 10 days or more before we expect a substantial difference in the number of cases following a change in COVID-19 control measures, but 20–26 days to detect the impact of the change in reported data. The time frames are longer for smaller changes in control measures and are impacted by testing and reporting processes, with delays reaching ≥ 30 days. Conclusion The time until a change in control measures has an observed impact is longer than the mean incubation period of COVID-19 and the commonly used 14-day time period. Policymakers and practitioners should consider this when assessing the impact of policy changes. Rapid, consistent and real-time COVID-19 surveillance is important to minimise these time frames.

The differential equations for the non-distancing model compartments are: (S1) The differential equations for individuals practising physical distancing are analogous: In these equations, the strength of physical distancing is represented by the parameter 0 ≤ f (t) ≤ 1, with f (t) = 1 indicating no physical distancing. This parameter modulates transmission from within the distancing compartments by infected distancing individuals contributing a reduced portion of the force of infection. In addition, individuals in the distancing compartments are less likely to be encountered by others, and so are less likely to become infected themselves. The value of f may be allowed to vary in time as physical distancing is strengthened or relaxed in the population, and in particular, in this work we defined: Recall also the mean number of new cases reported on day t, µ t : Given a set of daily case count data {C t }, where C t is the number of cases identified on day t, we used a negative binomial likelihood with mean µ t and dispersion parameter φ, NB(C t | µ t , φ) as in [1], to write the likelihood of the data given the model parameters accounting for dispersion in observation: Here T is the final day for which case counts are available (or, of concern), f = (f 0 , f 1 , f 2 ), and θ = (u d , u r , q, γ, k 1 , k 2 , ψ t ) are the remaining model parameters, which were assumed to be fixed in all of our experiments except for the simulations in Figure S1. In this parameterization of the negative binomial distribution, the variance scales with the mean according to the dispersion parameter φ: Var[{C t }] = µ + µ 2 /φ. Values of the model parameters for our analysis using British Columbia case count data are provided in Table S1.

Time to pairwise model divergence
In predictive investigations of the effects of relaxing physical distancing in the main text, we simulated the SEIR-type model described above forwards in time, from 1 May to 1 July. We investigated situations in which the amount of physical distancing was modulated on 17 May through a forcing of the ODE parameter f 2 , and in which many of the ODE parameters were modulated through grid-searches. We also considered the introduction of physical distancing in British Columbia, assuming that physical distancing was introduced 18 March by modulation of parameter f from f 0 = 1 to f 1 = 0.4 and f 1 = 0.7, and compared to a baseline model in which f remained at 1. We investigated confidence intervals around the number of active case in early/mid-April, by considering 100 replicates in the ODE simulations, each with a value of R 0 drawn independently from the prior, with standard deviation 0.15. In Results, we provided evidence that the physical distancing undertaken by people in British Columbia corresponded to a modeled value for f 1 of 0.36. In the relaxation of distancing scenario, we therefore considered a baseline model in which the value of f remained at 0.36 for the duration of the time. We considered alternative relaxation models in which the value of f changed from f 1 = 0.36 to either f 2 = 0.5 (weak relaxation), f 2 = 0.65 (medium relaxation) or f 2 = 0.9 (strong relaxation) on 17 May. For each condition, we again considered 100 replicates each with a value of R 0 drawn from the prior.

Impact of uncertainty in model parameters
We considered two experiments wherein the uncertainty about the ODE parameters was modulated, and recorded the time to detect a threshold of excess in the active cases above baseline models. In our first experiment we focused on R 0 . In the main text, we considered ODEs in which the R 0 parameter was initialized to a draw from a normal distribution with mean 2.57 and standard deviation 0.15. This reflects our broad understanding about the uncertainty in R 0 . In all conditions, the baseline indicated a situation in which physical distancing is not relaxed, and remains in effect. If the standard deviation is varied away from 0.15, then the days until the threshold is reached (i.e., a significant difference between the baseline and a relaxed condition is seen) is affected. The results of varying the standard deviation between 0.01 and 0.4 (considering the values in the set {0.01, 0.11, 0.21, 0.31, 0.41}) are shown in Figure S1a. In that Figure, we see that the days until a significant increase in the active cases can vary from 10 to 50 days. Missing values in the lines in this plot indicate that our methods could not determine whether or not the condition will ever differ from the baseline: if the standard deviation of R 0 is > 0.11, we do not predict that a weak relaxation of physical distancing to f 2 = 0.6 will result in a substantial increase in active cases. However, if the standard deviation of R 0 is < 0.11 a relaxation of physical distancing to f 2 = 0.6 could lead to a significant increase in active case count after 20 or 40 days. This is not because the change itself is different, but is a consequence of uncertainty in the baseline.
We also considered variation of the ODE parameters 1/γ, k 1 , k 2 , q, u d and u r . We varied each of these parameters around their default values (given in Table S1) by X%, with X varied in the  Figure S1: Impact of uncertainty in the model parameters on the time to exceed the active case threshold. (a) Days until active cases in the relaxed model exceed the baseline model (yaxis), with varying levels of the standard deviation for R 0 (sd, x-axis). There are four relaxation levels (all starting relaxation on 17 May). Parameter settings are identical to other figures aside from sd. The missing values indicate that our methods could not differentiate between the models. (b) The variables 1/γ, k 1 , k 2 , q, u d and u r were all varied between 10% and 30% around their fitted values (with the three levels described in the legend). The value that f was relaxed to is shown in the x-axis (before relaxation, f 1 was set to 0.36). So, R 0 was sampled at ODE initialization, but so too now were the 6 variables mentioned (with independent draws from the uniform distribution). For each f 2 level, 100 iterations were considered each with 100 replicates. For f 2 = 0.6, we did not observe enough difference between the model trajectories over the period observed (120 days) to determine the active case threshold. set {10, 20, 30}. For each X (uncertainty) level, we considered simulations in which the ODE was initialized with random parameters. For the basic reproduction number, R 0 was initialized as a normal distribution with mean 2.57 and variance 0.15 (as is done in the main text), and for the parameters 1/γ, k 1 , k 2 , q, u d and u r we initialized each parameter by drawing from a uniform distribution. For each parameter, the range of the uniform distribution had a minimum value of the 'default' value of that parameter (given by Table S1) minus X% of the default value, and maximum value of the 'default' plus X%. We then recorded the days until an active case count excess of 10 was obtained in 95% of the simulations. The number of iterations and the number of replicates was identical to Figure 3 (described in the main text). Density plots for the number of days until the threshold on the excess of active case counts are shown in Figure S1b for a variety of relaxation levels. These results show that if the uncertainty about the parameters of our SEIR-type model is 10%, then relaxing distancing totally f 2 = 1 can result in 10-15 days before the threshold of 10 excess cases is reached. This increases slightly to 15-20 and 17-22 days if the uncertainty is increased to 20% or 30%. In addition, if physical distancing is relaxed more moderately (to f 2 = 0.7 for example), then it may be around 70 days before an excess of 10 active cases is seen, given moderate uncertainty in the model parameters. For relaxation to f 2 = 0.6, we did not observe enough difference between the model trajectories to determine the active case threshold. This may be due to either a) a significant difference in the trajectories only occurred after the 120 day simulation window considered here or b) one model's active cases never become significantly larger than the others a significant percent of the time, given the uncertainty in the model parameters.
The above results indicate that, for smaller or moderate changes in f , uncertainty about the model parameters can substantially increase the number of days until definitive statements about the effects of relaxing physical distancing can be made. For larger changes in f this is less impactful, with the total range of the number of days until effects are seen varying between 10 and 75.

Effect of the incubation period
We varied the incubation period to explore the impact on the time before the active cases threshold of 10 is reached. The incubation period was defined as 1/k 1 + 1/k 2 , with default values k 1 = 1/5 and k 2 = 1 throughout this work. We varied the incubation period between 6 and 10 days by modulating k 1 and k 2 appropriately (with the same scaling factor for each) and show the results in Figure S2. For each incubation period and relaxation f 2 level, we performed 50 replicates and plotted the median number of days until the active cases threshold is reached. We found that there is not much variation in the days to threshold as the incubation period is varied. This suggests that the variance in active case due to uncertainty about R 0 may overwhelm variation in the incubation period, over the range considered.  Figure S2: The incubation period has a limited effect on the time before a substantial difference occurs between two simulated scenarios. The initial value f 1 is 0.36. The curves were obtained under the uncertainty assumption that R 0 is normally distributed with a mean 2.57, and a standard deviation 0.15. The incubation period is 1/k 1 + 1/k 2 . For each setting of the incubation period, we rescaled k 1 and k 2 evenly from their fitted values (k 1 = 1 and k 2 = 1/5) to achieve the incubation period in days (e.g., for an incubation period of 7 days 1/k 1 + 1/k 2 = 7, and so the values for k 1 and k 2 used were 1 * (6/7) and 1/5 * (6/7) respectively). Missing values indicate that our methods could not distinguish between simulations in which physical distancing was relaxed and simulations in which physical distancing was not relaxed (indicating that a relaxation of f 2 to 0.6 may not have an effect for much of the range of incubation periods considered)

Daily MLE using simulated data
In addition to the experiments in the main manuscript which estimated daily MLEs of f 1 or f 2 using reported case counts from British Columbia, we performed a simulation study in which we estimated daily MLEs of f 2 using simulated data on the relaxation of physical distancing to a number of fixed f 2 values (0.5 0.65 and 0.9). The benefit of using simulated data is that we can be confident the change in distancing occurred exactly on the date stated. We set the observation noise and delay as estimated pre-relaxation during March/April in BC (φ = 5, onset-to-reporting delay was Weibull with mean 8.78, variance 27.4), both in the simulation of data and in the daily MLE analysis. The results of this are shown in Figure S3. We found similar time frames to identify the strength of physical distancing as with the observed data: 30 days for f 2 = 0.5, and 23 days for both f 2 = 0.65 and f 2 = 0.9. Smaller changes in distancing take longer time frames to identify.

Estimation of R t
For comparison with the daily MLE method applied to British Columbia case count data in Figures 4 and S3, we also performed estimation of R t using the EpiEstim package [10] in R. Results are shown in Figure S4. Note that EpiEstim assumes that all cases from the first time step are imported cases, leading to inflated R t estimates during the first week. To account for this, when finding the earliest time that R t was less than/greater than 1.0 at the 95% level, we ignored this first week of estimation.  Figure S4: Median estimates of the time-dependent reproduction number R t in British Columbia, using a weekly sliding window and assumed serial interval of 5 (sd 1) days. Top panel: after implementing distancing, using true observed case counts (14 days until R t < 1 at the 95% level). Bottom panel: after relaxing distancing, using simulated data (18, 25 days until R t > 1 at the 95% level for f 2 = 0.9, 0.65, respectively), under the assumption that f 1 = 0.36 and observation noise and delay remained as pre-relaxation during March/April in BC. Coloured bands correspond to 95% quantiles.