Usage
Plotting
The package contains two functions for creating plots, qq_conf_plot and pp_conf_plot to create QQ plots and PP plots, respectively. These two functions have identical interfaces, with the exception that qq_conf_plot requires the input of a quantile function (e.g. qnorm) and pp_conf_plot requires the input of a distribution function (e.g. pnorm). Thus, any information below about parameters that can be fed to the qq_conf_plot function also apply to pp_conf_plot, and vice versa.
Basics / Parameter Estimation
First, we load in qqconf
require(qqconf)Then, we generate data from a standard normal distribution:
set.seed(0)
sample <- rnorm(n = 100, mean = 0, sd = 1)Now, if we did not know the distribution that our sample came from but instead wanted to test if it was sampled i.i.d from a \(N(0, 1)\) distribution, we could create a QQ plot to compare the quantiles of the sample to the theoretical quantiles of a \(N(0, 1)\) distribution.
To do this, we call qq_conf_plot as follows:
qq_conf_plot(
obs = sample,
distribution = qnorm,
dparams = list(mean = 0, sd = 1)
)
By default, this function creates a QQ plot with ELL testing bounds calculated with a Type I error rate of .05. Note here that dparams is not a required parameter. If we were to call qq_conf_plot without specifying dparams, then our code would estimate the mean and variance of the sample assuming it comes from a normal distribution as is specified by the distribution parameter. More generally, for any continuous distribution that is neither normal nor uniform, we use maximum liklihood estimation to estimate all parameters of the distribution. For the uniform distribution, we do not support parameter estimation but instead default to \(U(0, 1)\) and allow the user to specify a custom max and min. For the normal distribution, because it is so commonly used as the reference distribution in QQ plots, we experimented with different methods of parameter estimation and found that estimating the mean of the distribution as the median of the sample and the standard deviation as \(S_{n}\) as proposed by Rousseeuw and Croux (Rousseeuw, Crow 1993) had well calibrated Type I error rates. When we used the MLEs for the normal distribution, we found that the testing bounds produced were conservative. Note that the testing bounds produced for other distributions with parameters estimated via MLE will also likely be conservative. For more information on parameter estimation, please see section 2.6 of our paper.
Modifying Visual Features
General Plot Features
Parameter estimation aside, we have now created a QQ plot that compares our sample to \(N(0, 1)\). While this plot is informative, there are a number of potential modifications we may want to make to this plot for ease of viewing. First, because the top and the bottom of the confidence bands are cut off in this plot, we may want to expand the y-axis limits. We do this by simply adding a ylim argument to the function call:
qq_conf_plot(
obs = sample,
distribution = qnorm,
dparams = list(mean = 0, sd = 1),
ylim = c(-4, 4)
)
In fact, because qq_conf_plot takes ... as an argument, any other parameter normally passed to the plot function in base can be passed to qq_conf_plot. If, for instance, we wanted to modify the axis labels, we could write:
qq_conf_plot(
obs = sample,
distribution = qnorm,
dparams = list(mean = 0, sd = 1),
ylim = c(-4, 4),
xlab = "More Informative Title"
)
However, while the ... argument can be used for more general features of the plot like axis titles, because there are so many separate objects in the plot (points, lines, bands), we’ve added additional arguments so that the user can easily specify the visual features of any element of the plot.
Points
If we want to modify the appearance of the points on the plot, we can use the optional points_params argument. This argument is a list that takes any arguments that can be passed to the points function in base. For instance, if we wanted to make the points smaller, we could do the following.
qq_conf_plot(
obs = sample,
distribution = qnorm,
dparams = list(mean = 0, sd = 1),
ylim = c(-4, 4),
points_params = list(cex = .5) # makes points smaller
)
For more options, see the documentation of the points function.
Expectation Line
If we want to modify the line in the plot that indicates a perfect fit between the reference distribution and the sample, we can use the line_params argument. This list can take any parameters that can be passed to the lines function and will modify the plot accordingly. If, for instance, we wanted to make the line red, we would do the following:
qq_conf_plot(
obs = sample,
distribution = qnorm,
dparams = list(mean = 0, sd = 1),
ylim = c(-4, 4),
line_params = list(col="red") # makes expectation line red
)
Note, that if we don’t want this line to show up on the plot at all, we can pass in type = "n" to line_params.
qq_conf_plot(
obs = sample,
distribution = qnorm,
dparams = list(mean = 0, sd = 1),
ylim = c(-4, 4),
line_params = list(type="n") # removes expectation line
)
Again, for more information see the documentation for the lines function.
Testing Bands
By default, qq_conf_plot does not plot pointwise testing bands on the plot. Pointwise testing bands, in contrast to simultaneous testing bands, are anti-conservative, as they ignore the multiple testing problem inherent in QQ plots and simply conduct a test on each order statistic. To add these pointwise bands, one can simply set the plot_pointwise parameter to True in the qq_conf_plot function. To modify their appearance, one can use the pointwise_lines_params parameter. Because this parameter list is simply passed to the lines function, it works exactly the same way as the line_params parameter.
Finally, if we want to modify the appearance of the simultaneous testing bands, we can use the polygon_params parameter. As the name suggests, arguments in this parameter list are passed to the polygon function in base and modify the testing bands accordingly. Note that by default, border is set to NA and col is set to grey. If the user overrides the default option, all defaults of polygon are used unless otherwise specified, which does not set border to NA and col to grey. If, for instance, we wanted to change the color of the shading, we would do the following:
qq_conf_plot(
obs = sample,
distribution = qnorm,
dparams = list(mean = 0, sd = 1),
ylim = c(-4, 4),
polygon_params = list(col = 'powderblue', border = NA) # change shading and keep no border
)
Scaling and Data Transformations
In addition to general visual parameters to modify the elements of the plot, we’ve also included a few parameters that we found to be useful in particular cases when making a QQ or PP plot. Especially when \(n\) is large, it can be difficult to see the relevant parts of the plot.
To demonstrate this, we will generate 10,000 points from a \(Beta(1, 1.05)\) distribution and test this sample against a \(U(0, 1)\) distribution.
sample <- rbeta(n = 10000, shape1 = 1.0, shape2 = 1.05)Because a QQ plot and PP plot are equivalent for the uniform distribution, we will switch to PP plots to demonstrate the interface.
Differencing
Normally in a PP plot, the expected probabilities are plotted on the x-axis and the theoretical probabilities are plotted on the y-axis.
pp_conf_plot(
obs = sample,
distribution = punif,
points_params = list(cex=.1)
)
However, in this plot it is very difficult to see anything because the individual points are so small and the magnitude of the deviation from the reference distribution is not very large.
Instead, if we plot the expected probabilities on the x-axis and the observed probabilities - the expected probabilities on the y-axis by setting the difference parameter to TRUE, things become much easier to see:
pp_conf_plot(
obs = sample,
distribution = punif,
points_params = list(cex=.1),
difference = TRUE, # Make y-axis differenced
ylim = c(-.0225, .0225)
)
log Scaling
Sometimes, we are only interested in deviations of the sample from one tail of the reference distribution. For instance, if we have a sample of \(n\) p-values that we would like to compare to the \(U(0, 1)\) distribution for the sake of global null hypothesis testing, we’re generally only interested in p-values that are small. In this case, especially with large \(n\), it can be helpful to transform the probability values onto the \(-log10\) scale.
To demonstrate this, we will generate 10,000 points from a mixture of a \(Beta(.25, 1)\) and a \(U(0, 1)\) and test this sample against a \(U(0, 1)\) reference distribution.
mix <- distr::UnivarMixingDistribution(
distr::Beta(shape1 = .25, shape2 = 1),
distr::Unif(),
mixCoeff=c(.01, .99)
)
sampler <- distr::r(mix)
sample <- sampler(10000)Again, if we make a standard PP plot the relevant parts of the plot are very difficult to make out.
pp_conf_plot(
obs = sample,
distribution = punif,
points_params = list(cex=.1)
)
Even if we set difference to TRUE, it’s still difficult to see the left tail of the distribution.
pp_conf_plot(
obs = sample,
distribution = punif,
points_params = list(cex=.1),
difference = TRUE
)
However, setting log10 to TRUE makes the plot much easier to see.
pp_conf_plot(
obs = sample,
distribution = punif,
points_params = list(cex=.1),
log10 = TRUE
)
Note, that if we want to highlight the right tail of the distribution as opposed to the left tail, we can set log10 to TRUE and the right_tail parameter to TRUE. This results in a plot with the data transformed onto the log10 scale instead of the -log10 scale.
ELL Testing Bounds Generations
In addition to the plotting interfaces explained above, qqconf provides functions to directly produce testing bounds generated by the equal local levels method. Functions exist for both two-sided and one-sided testing. For information on the generation of testing bounds using equal local levels, please see section 2 of our paper.
Two-Sided Testing
Given a desired global Type I error rate \(\alpha\) and a sample size \(n\), we can generate two-sided bounds using the get_bounds_two_sided function. Below, we generate such bounds for \(n = 100\) and \(\alpha = .05\):
bounds <- get_bounds_two_sided(alpha = .05, n = 100)In this case, the bounds object is a list with a number of objects (for more information see the documentation), but the lower bounds can be extracted with bounds$lower_bounds and the upper bounds with bounds$upper_bounds.
One-Sided Testing
In addition to two-sided testing, qqconf also provides functionality for one-sided testing.
Given a desired global Type I error rate \(\alpha\) and a sample size \(n\), we can generate such bounds using the get_bounds_one_sided function. Below, we generate such bounds for \(n = 100\) and \(\alpha = .05\):
bounds <- get_bounds_one_sided(alpha = .05, n = 100)From the bounds object, the upper bounds can be extracted via bounds$bound. Again, this function returns a list and more information can be extracted from bounds.