Introduce dlookr

Choonghyun Ryu

2022-06-06

Preface

After you have acquired the data, you should do the following:

Diagnose data quality.
- If there is a problem with data quality,
- The data must be corrected or re-acquired.
Explore data to understand the data and find scenarios for performing the analysis.
Derive new variables or perform variable transformations.

The dlookr package makes these steps fast and easy:

Performs a data diagnosis or automatically generates a data diagnosis report.
Discover data in a variety of ways, and automatically generate EDA(exploratory data analysis) report.
Impute missing values and outliers, resolve skewed data, and binaries continuous variables into categorical variables. And generates an automated report to support it.

dlookr increases synergy with dplyr. Particularly in data exploration and data wrangle, it increases the efficiency of the tidyverse package group.

Supported data structures

Data diagnosis supports the following data structures.

data frame : data.frame class.
data table : tbl_df class.
table of DBMS : table of the DBMS through tbl_dbi.
- Use dplyr as the back-end interface for any DBI-compatible database.

List of supported tasks of data analytics

Diagnose Data

Overall Diagnose Data

Tasks	Descriptions	Functions	Support DBI
describe overview of data	Inquire basic information to understand the data in general	`overview()`
summary overview object	summary described overview of data	`summary.overview()`
plot overview object	plot described overview of data	`plot.overview()`
diagnose data quality of variables	The scope of data quality diagnosis is information on missing values and unique value information	`diagnose()`	x
diagnose data quality of categorical variables	frequency, ratio, rank by levels of each variables	`diagnose_category()`	x
diagnose data quality of numerical variables	descriptive statistics, number of zero, minus, outliers	`diagnose_numeric()`	x
diagnose data quality for outlier	number of outliers, ratio, mean of outliers, mean with outliers, mean without outliers	`diagnose_outlier()`	x
plot outliers information of numerical data	box plot and histogram whith outliers, without outliers	`plot_outlier.data.frame()`	x
plot outliers information of numerical data by target variable	box plot and density plot whith outliers, without outliers	`plot_outlier.target_df()`	x
diagnose combination of categorical variables	Check for sparse cases of level combinations of categorical variables	`diagnose_sparese()`

Visualize Missing Values

Tasks	Descriptions	Functions	Support DBI
pareto chart for missing value	visualize pareto chart for variables with missing value.	`plot_na_pareto()`
combination chart for missing value	visualize distribution of missing value by combination of variables.	`plot_na_hclust()`
plot the combination variables that is include missing value	visualize the combinations of missing value across cases..	`plot_na_intersect()`

Reporting

Types	Descriptions	Functions	Support DBI
reporting the information of data diagnosis into pdf file	report the information for diagnosing the quality of the data.	`diagnose_report()`	x
reporting the information of data diagnosis into html file	report the information for diagnosing the quality of the data.	`diagnose_report()`	x
reporting the information of data diagnosis into html file	dynamic report the information for diagnosing the quality of the data.	`diagnose_web_report()`	x
reporting the information of data diagnosis into pdf and html file	paged report the information for diagnosing the quality of the data.	`diagnose_paged_report()`	x

EDA

Univariate EDA

Types	Tasks	Descriptions	Functions	Support DBI
categorical	summaries	frequency tables	`univar_category()`
categorical	summaries	chi-squared test	`summary.univar_category()`
categorical	visualize	bar charts	`plot.univar_category()`
categorical	visualize	bar charts	`plot_bar_category()`
numerical	summaries	descriptive statistics	`describe()`	x
numerical	summaries	descriptive statistics	`univar_numeric()`
numerical	summaries	descriptive statistics of standardized variable	`summary.univar_numeric()`
numerical	visualize	histogram, box plot	`plot.univar_numeric()`
numerical	visualize	Q-Q plots	`plot_qq_numeric()`
numerical	visualize	box plot	`plot_box_numeric()`
numerical	visualize	histogram	`plot_hist_numeric()`

Bivariate EDA

Types	Tasks	Descriptions	Functions	Support DBI
categorical	summaries	frequency tables cross cases	`compare_category()`
categorical	summaries	contingency tables, chi-squared test	`summary.compare_category()`
categorical	visualize	mosaics plot	`plot.compare_category()`
numerical	summaries	correlation coefficient, linear model summaries	`compare_numeric()`
numerical	summaries	correlation coefficient, linear model summaries with threshold	`summary.compare_numeric()`
numerical	visualize	scatter plot with marginal box plot	`plot.compare_numeric()`
numerical	Correlate	correlation coefficient	`correlate()`	x
numerical	Correlate	summaries with correlation matrix	`summary.correlate()`	x
numerical	Correlate	visualization of a correlation matrix	`plot.correlate()`	x
both	PPS	PPS(Predictive Power Score)	`pps()`	x
both	PPS	summaries with PPS	`summary.pps()`	x
both	PPS	visualization of a PPS matrix	`plot.pps()`	x

Normality Test

Types	Tasks	Descriptions	Functions	Support DBI
numerical	summaries	Shapiro-Wilk normality test	`normality()`	x
numerical	summaries	normality diagnosis plot (histogram, Q-Q plots)	`plot_normality()`	x

Relationship between target variable and predictors

Target Variable	Predictor	Descriptions	Functions	Support DBI
categorical	categorical	contingency tables	`relate()`	x
categorical	categorical	mosaics plot	`plot.relate()`	x
categorical	numerical	descriptive statistic for each levels and total observation	`relate()`	x
categorical	numerical	density plot	`plot.relate()`	x
categorical	categorical	bar charts	`plot_bar_category()`
numerical	categorical	ANOVA test	`relate()`	x
numerical	categorical	scatter plot	`plot.relate()`	x
numerical	numerical	simple linear model	`relate()`	x
numerical	numerical	box plot	`plot.relate()`	x
categorical	numerical	Q-Q plots	`plot_qq_numeric()`
categorical	numerical	box plot	`plot_box_numeric()`
categorical	numerical	histogram	`plot_hist_numeric()`

Reporting

Types	Descriptions	Functions	Support DBI
reporting the information of EDA into pdf file	reporting the information of EDA.	`eda_report()`	x
reporting the information of EDA into html file	reporting the information of EDA.	`eda_report()`	x
reporting the information of EDA into pdf file	dynamic reporting the information of EDA.	`eda_web_report()`	x
reporting the information of EDA into html file	paged reporting the information of EDA.	`eda_paged_report()`	x

Transform Data

Find Variables

Types	Descriptions	Functions	Support DBI
missing values	find the variable that contains the missing value in the object that inherits the data.frame	`find_na()`
outliers	find the numerical variable that contains outliers in the object that inherits the data.frame	`find_outliers()`
skewed variable	find the numerical variable that skewed variable that inherits the data.frame	`find_skewness()`

Imputation

Types	Descriptions	Functions	Support DBI
missing values	missing values are imputed with some representative values and statistical methods.	`imputate_na()`
outliers	outliers are imputed with some representative values and statistical methods.	`imputate_outlier()`
summaries	calculate descriptive statistics of the original and imputed values.	`summary.imputation()`
visualize	the imputation of a numerical variable is a density plot, and the imputation of a categorical variable is a bar plot.	`plot.imputation()`

Binning

Types	Descriptions	Functions	Support DBI
binning	converts a numeric variable to a categorization variable	`binning()`
summaries	calculate frequency and relative frequency for each levels(bins)	`summary.bins()`
visualize	visualize two plots on a single screen. The plot at the top is a histogram representing the frequency of the level. The plot at the bottom is a bar chart representing the frequency of the level.	`plot.bins()`
optimal binning	categorizes a numeric characteristic into bins for ulterior usage in scoring modeling	`binning_by()`
summaries	summary metrics to evaluate the performance of binomial classification model	`summary.optimal_bins()`
visualize	generates plots for understand distribution, bad rate, and weight of evidence after running binning_by()	`plot.optimal_bins()`
infogain binning	categorizes a numeric characteristic into bins for multi-class variable using recursive information gain ratio maximization	`binning_rgr()`
visualize	generates plots for understand distribution and distribution by target variable after running binning_rgr()	`plot.infogain_bins()`
evaluate	calculates metrics to evaluate the performance of binned variable for binomial classification model	`performance_bin()`
summaries	summary metrics to evaluate the performance of binomial classification model after performance_bin()	`summary.performance_bin()`
visualize	It generates plots for understand frequency, WoE by bins using performance_bin after running binning_by()	`plot.performance_bin()`
visualize	extract bins from “bins” and “optimal_bins” objects	`extract.bins()`

Diagnose Binned Variable

Types	Descriptions	Functions	Support DBI
diagnosis	performs diagnose performance that calculates metrics to evaluate the performance of binned variable for binomial classification model.	`performance_bin()`
summaries	summary method for “performance_bin”. summary metrics to evaluate the performance of binomial classification model.	`summary.performance_bin()`
visualize	visualize for understand frequency, WoE by bins using performance_bin and something else.	`plot.performance_bin()`

Transformation

Types	Descriptions	Functions	Support DBI
transformation	performs variable transformation for standardization and resolving skewness of numerical variables.	`transform()`
summaries	compares the distribution of data before and after data transformation	`summary.transform()`
visualize	visualize two kinds of plot by attribute of ‘transform’ class. The transformation of a numerical variable is a density plot.	`plot.transform()`

Reporting

Types	Descriptions	Functions	Support DBI
reporting the information of transformation into pdf	reporting the information of transformation.	`transformation_report()`
reporting the information of transformation into html	reporting the information of transformation.	`transformation_report()`
reporting the information of transformation into pdf	dynamic reporting the information of transformation.	`transformation_web_report()`
reporting the information of transformation into html	paged reporting the information of transformation.	`transformation_paged_report()`

Miscellaneous

Statistics

Types	Descriptions	Functions	Support DBI
statistics	calculate the entropy.	`entropy()`
statistics	calculate the skewness of the data.	`skewness()`
statistics	calculate the kurtosis of the data.	`kurtosis()`
statistics	calculate the Jensen-Shannon divergence between two probability distributions.	`jsd()`
statistics	calculate the Kullback-Leibler divergence between two probability distributions.	`kld()`
statistics	calculate the Cramer’s V statistic between two categorical(discrete) variables.	`cramer()`
statistics	calculate the Theil’s U statistic between two categorical(discrete) variables.	`theil()`
statistics	finding percentile of numerical variable.	`get_percentile()`
statistics	transform a numeric vector using several methods like “log”, “sqrt”, “log+1”, “log+a”, “1/x”, “x^2”, “x^3”, “Box-Cox”, “Yeo-Johnson”	`get_transform()`
statistics	calculate the Cramer’s V statistic	`cramer()`
statistics	calculate the Theil’s U statistic	`theil()`

Programming

Types	Descriptions	Functions	Support DBI
programming	extracts variable information having a certain class from an object inheriting data.frame.	`find_class()`
programming	gets class of variables in data.frame or tbl_df.	`get_class()`
programming	retrieves the column information of the DBMS table through the tbl_bdi object of dplyr.	`get_column_info()`
programming	finding Users Machine’s OS.	`get_os()`
programming	import Google Fonts.	`import_google_font()`