Data Storage
On a File System
We start by writing the dataset as is with
write.table(), saveRDS(),
write_vc() and write_vc() without storage
optimization. Note that write_vc() uses optimization by
default. Since write_vc() creates two files for each data
set, we take their combined file size into account.
library(git2rdata)
root <- tempfile("git2rdata-efficient")
dir.create(root)write.table(airbag, file.path(root, "base_R.tsv"), sep = "\t")
base_size <- file.size(file.path(root, "base_R.tsv"))
saveRDS(airbag, file.path(root, "base_R.rds"))
rds_size <- file.size(file.path(root, "base_R.rds"))
fn <- write_vc(airbag, "airbag_optimize", root, sorting = "X")
optim_size <- sum(file.size(file.path(root, fn)))
fn <- write_vc(airbag, "airbag_verbose", root, sorting = "X", optimize = FALSE)
verbose_size <- sum(file.size(file.path(root, fn)))Since the data is highly compressible, saveRDS() yields
the smallest file at the cost of having a binary file format. Both
write_vc() formats yield smaller files than
write.table(). Partly because write_vc()
doesn’t store row names and doesn’t use quotes unless needed. The
difference between the optimized and verbose version of
write_vc() is, in this case, solely due to the way
write_vc() stores factors in the data (tsv)
file. The optimized version stores the indices of the factor whereas the
verbose version stores the levels. For example:
airbag$dvcat has 5 levels with short labels (on average 5
character), storing the index requires 1 character. This results in more
compact files.
| method | file_size | relative |
|---|---|---|
| saveRDS() | 313.13 | 0.12 |
| write_vc(), optimized | 1739.99 | 0.64 |
| write_vc(), verbose | 2467.19 | 0.91 |
| write.table() | 2716.93 | 1.00 |
The reduction in file size when storing in factors depends on the length of the labels, the number of levels and the number of observations. The figure below illustrates the strong gain as soon as the level labels contain more than two characters. The gain is less pronounced when the factor has a large number of levels. The optimization fails in extreme cases with short factor labels and a high number of levels.
Effect of the label length on the efficiency of storing factor optimized, assuming 1000 observations
The effect of the number of observations is mainly due to the overhead of storing the metadata. The importance of this overhead increases when the number of observations is small.
Effect of the number of observations on the efficiency of storing factor optimized assuming labels with 10 characters
In Git Repositories
Here we will simulate how much space the data requires to store the
history in a git repository. We will create a git repository for each
method and store different subsets of the same data. Each commit
contains a new version of the data. Each version is a random sample
containing 90% of the observations of the airbag data. Two
consecutive versions of the subset will have about 90% of the
observations in common.
After writing each version, we commit the file, perform garbage
collection (git gc) on the git repository and then
calculate the size of the git history
(git count-objects -v).
library(git2r)
tmp_repo <- function() {
root <- tempfile("git2rdata-efficient-git")
dir.create(root)
repo <- git2r::init(root)
git2r::config(repo, user.name = "me", user.email = "me@me.com")
return(repo)
}
commit_and_size <- function(repo, filename) {
add(repo, filename)
commit(repo, "test", session = TRUE)
git_size <- system(
sprintf("cd %s\ngit gc\ngit count-objects -v", dirname(repo$path)),
intern = TRUE
)
git_size <- git_size[grep("size-pack", git_size)]
as.integer(gsub(".*: (.*)", "\\1", git_size))
}
repo_wt <- tmp_repo()
repo_wts <- tmp_repo()
repo_rds <- tmp_repo()
repo_wvco <- tmp_repo()
repo_wvcv <- tmp_repo()
repo_size <- replicate(
100, {
observed_subset <- rbinom(nrow(airbag), size = 1, prob = 0.9) == 1
this <- airbag[
sample(which(observed_subset)),
sample(ncol(airbag))
]
this_sorted <- airbag[observed_subset, ]
fn_wt <- file.path(workdir(repo_wt), "base_R.tsv")
write.table(this, fn_wt, sep = "\t")
fn_wts <- file.path(workdir(repo_wts), "base_R.tsv")
write.table(this_sorted, fn_wts, sep = "\t")
fn_rds <- file.path(workdir(repo_rds), "base_R.rds")
saveRDS(this, fn_rds)
fn_wvco <- write_vc(this, "airbag_optimize", repo_wvco, sorting = "X")
fn_wvcv <- write_vc(
this, "airbag_verbose", repo_wvcv, sorting = "X", optimize = FALSE
)
c(
write.table = commit_and_size(repo_wt, fn_wt),
write.table.sorted = commit_and_size(repo_wts, fn_wts),
saveRDS = commit_and_size(repo_rds, fn_rds),
write_vc.optimized = commit_and_size(repo_wvco, fn_wvco),
write_vc.verbose = commit_and_size(repo_wvcv, fn_wvcv)
)
})Each version of the data has on purpose a random order of
observations and variables. This is what would happen in a worst case
scenario as it would generate the largest possible diff. We also test
write.table() with a stable ordering of the observations
and variables.
The randomised write.table() yields the largest git
repository, converging to about 6.5 times the size of a git repository
based on the sorted write.table(). saveRDS()
yields a 26% reduction in repository size compared to the randomised
write.table(), but still is 4.8 times larger than the
sorted write.table(). Note that the gain of storing binary
files in a git repository is much smaller than the gain in individual
file size because git compresses its history. The optimized
write_vc() starts at 88% and converges toward 78%, the
verbose version starts at 94% and converges towards 110%. Storage size
is a lot smaller when using write_vc() with optimization.
The verbose option of write_vc() has little the gain in
storage size. Another advantage is that write_vc() stores
metadata.
Size of the git history using the different storage methods.
Relative size of the git repository when compared to write.table().
