Compliance Documentation

Peter Hurford

2022-05-05

Compliance documentation is a premium add-on product to DataRobot. It allows users to automatically generate and download documentation to assist with deploying models in highly regulated industries.

Connect to DataRobot

To explore compliance documentation, let’s first connect to DataRobot. First, you must load the DataRobot R package library.

If you have set up a credentials file, library(datarobot) will initialize a connection to DataRobot automatically. Otherwise, you can specify your endpoint and apiToken as in this example to connect to DataRobot directly. For more information on connecting to DataRobot, see the “Introduction to DataRobot” vignette.

library(datarobot)
endpoint <- "https://<YOUR DATAROBOT URL GOES HERE>/api/v2"
apiToken <- "<YOUR API TOKEN GOES HERE>"
ConnectToDataRobot(endpoint = endpoint, token = apiToken)

Download Compliance Documentation

To download compliance documentation for a particular model, call DownloadComplianceDocumentation on a particular model and specify a filepath to download the documentation to. Note that it downloads in DOCX format.

DownloadComplianceDocumentation(model, "path/to/filename.docx")

Creating a Custom Template

You can also use your own custom compliance documentation templates.

The Default Template

First, let’s get the default template. This can be done just by using GetComplianceDocTemplate. It downloads as a JSON file.

GetComplianceDocTemplate("path/to/filename.json")

Updating the Default Template

A common workflow for building your own template is downloading the default template and modifying it.

DownloadComplianceDocTemplate("path/to/filename.json")
# ...then modify the compliance doc template in your favorite editor.
UploadComplianceDocTemplate(name = "myNewTemplate", filename = "path/to/modified_file.json")

Alternatively, you can construct a template via a list:

sections <- list(list("title" = "Missing Values Report",
                      "highlightedText" = "NOTICE",
                      "regularText" = "This dataset had a lot of Missing Values. See the chart below: {{missingValues}}",
                      "type" = "user"),
                 list("title" = "Blueprints",
                      "regularText" = "{{blueprintDiagram}} /n Blueprint for this model",
                      "type" = "user"))
UploadComplianceDocTemplate(name = "myNewTemplateFromSections", sections = sections)

You can then get and download your template:

myTemplate <- ListComplianceDocTemplates(namePart = "myNewTemplateFromSections")[[1]]
DownloadComplianceDocTemplate(myTemplate)

Creating Custom Compliance Documentation from Custom Template

Once you have a custom template made, you can use it to create custom compliance documentation:

myTemplate <- ListComplianceDocTemplates(namePart = "myNewTemplate")[[1]]
CreateComplianceDocumentation(model, myTemplate)

Keyword Tags

Custom keyword tags are supported for templates, embedding DataRobot generated content into the template. Each tag looks like {{ keyword }} and on generation will be replaced with corresponding content. Below you can find a table of currently supported tags:

+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| Tag                            | Type           | Content                                              | Web Application UI Analog                                      |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ blueprint_diagram }}        | Image          | Graphical representation of the modeling pipeline.   | Leaderboard >> Model >> Describe >> Blueprint                  |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ alternative_models }}       | Table          | Comparison of the model with alternatives            | Leaderboard                                                    |
|                                |                | built in the same project.                           |                                                                |
|                                |                | Also known as challenger models.                     |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ model_features }}           | Table          | Description of the model features                    | Data >> Project Data                                           |
|                                |                | and corresponding EDA statistics.                    |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ missing_values }}           | Table          | Description of the missing values and their          | Leaderboard >> Model >> Describe >> Missing Values             |
|                                |                | processing in the model.                             |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ partitioning }}             | Image          | Graphical representation of the data partitioning.   | Data >> Show Advanced Options >> Partitioning                  |
|                                |                |                                                      | (only available before project start)                          |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ model_scores }}             | Table          | Metric scores of the model on different data sources | Leaderboard >> Model                                           |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ lift_chart }}               | Image          | Lift chart.                                          | Leaderboard >> Model >> Evaluate >> Lift Chart                 |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ feature_impact }}           | Image          | Feature Impact chart.                                | Leaderboard >> Model >> Understand >> Feature Impact           |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ feature_impact_table }}     | Table          | Table representation of Feature Impact data.         | Leaderboard >> Model >> Understand >> Feature Impact >> Export |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ feature_effects }}          | List of images | Feature Effects charts for the top 3 features.       | Leaderboard >> Model >> Understand >> Feature Effects          |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ accuracy_over_time }}       | Image          | Accuracy over time chart.                            | Leaderboard >> Model >> Evaluate >> Accuracy Over Time         |
|                                |                | Available only for datetime partitioned projects.    |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ cv_scores }}                | Table          | Project metric scores for each fold.                 | Currently unavailable in the UI                                |
|                                |                | Available only for projects with cross validation.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ roc_curve }}                |                | ROC Curve.                                           | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
|                                | Image          | Available only for binary classification projects.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ confusion_matrix_summary }} | Table          | Confusion matrix summary for the threshold with      | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
|                                |                | maximal F1 score value (default suggestion in UI).   |                                                                |
|                                |                | Available only for binary classification projects.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ prediction_distribution }}  | Image          | Prediction distribution.                             | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
|                                |                | Available only for binary classification projects.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+