### Precision

The timing test class attempts to correctly report the precision of its results by applying the following mechanisms:

- The standard deviation of all samples is taken. This is used to compute a 95% confidence interval for the average.
- Timing overhead is measured by calling the internal timer repeatedly and measuring the time required to do so.

The contribution of timing overhead to errors in event-time measurement is estimated as follows:

The bar represents a single sample. The two boxes labeled a at the ends of the bar represent the timing overhead. The left end of the box is the time at which the call to a timing function is initiated, and the right end of the box is the time at which the call returns. The resultant value may represent time at any point within the box. The minimum measured time is exactly the event time multiplied by the number of iterations. The maximum measured time is the event time multiplied by the number of iterations plus 2a. This is the worst case. The actual event time falls somewhere in between.

ITimingTest reports the most pessimistic interpretation reasonable. It reports each timing measurement as the upper end of the 95% confidence interval for the event time, plus an upper-bound estimate of 2a/n for the timing overhead. A greater sample count decreases error by reducing the size of the confidence interval, a larger timing count decreases error by distributing the timing overhead over more events.

ITimingTest parses command line arguments. These arguments modify the way in which timing is done. They may also be supplied programmatically (see ITest).

Argument | Description
-at tol | Set the tolerance for automatic sampling to tol (default = 0.1).
-c count | Set the operations per sample to count (default = 100).
-cc | Continually calibrate by measuring empty loop time before each sample.
-co correlation | Set the minimum correlation between samples for which to post an error to correlation (default = 0.4).
-m maxTime | Consider this test to fail if the median time per operation is greater than maxTime (default = none).
-na | Turn off automatic timing.
-s n | Take n samples, each sample consisting of many operations (default = 10).
-t | Don't do timing tests; just execute one operation without timing.
-w maxWindows | Stop if times do not converge within maxWindows windows of automatic sampling (default = 10).

The start-stop timing test attempts to correctly report the precision of its results by estimating the dispersion of samples and the sampling error as follows:

- Each sample consists of one or more segments. These segments represent timed events, beginning with a call to start an internal timer, the activity being timed, and a call to stop the internal timer. Let k be the number of segments per sample.
- The Test Framework makes the pessimistic assumption that the measured time for each segment has a potential error range of 2a, where a is the timing overhead.

The contribution of timing overhead to errors in IStartStopTimingTest event-time measurement is also estimated. The boxes labeled a at the ends of the bar represent the timing overhead. This diagram represents a single sample, which consists of one or more segments. Each segment is bracketed on each end by the timing overhead.

- The minimum measured time is exactly the event time multiplied by the number of segments, k. The maximum measured time is the event time times k plus 2ka. Thus, the worst-case error is 2ka.
- As is the case with ITimingTest, increasing the sample count reduces the dispersion between samples. Unlike ITimingTest, however, increasing the timing count does nothing to reduce the sampling error because the timing overhead is incurred repeatedly within the sampling loop, rather than only at the start and finish.