Model Cards for transparent, responsible reporting

What is a “Model Card”?

Good documentation helps us make sense of software, know when and how to use it, and understand its purpose. The same can be true of documentation or reporting for a deployed model, but it can be hard to know where to start. The paper “Model Cards for Model Reporting” (Mitchell et al. 2019) provides a suggested framework for organizing and presenting the essential facts about a deployed machine learning model. The vetiver package provides an R Markdown template for creating a “Model Card” for a published vetiver model. The template automates extracting some information from the model object, and provides structure for the model developer where automation is not possible.

Model developers see a nudge to create a model card when they publish a model; we recommend that you create a model card when you deploy a model for the first time and refresh that model card as needed when new versions are deployed.

library(vetiver)
library(pins)
model_board <- board_temp()

cars_lm <- lm(mpg ~ ., data = mtcars)
v <- vetiver_model(cars_lm, "cars_linear")
vetiver_pin_write(model_board, v)
#> Creating new version '20220811T173414Z-ebc45'
#> Writing to pin 'cars_linear'
#> 
#> Create a Model Card for your published model
#> • Model Cards provide a framework for transparent, responsible reporting
#> • Use the vetiver `.Rmd` template as a place to start

(Learn more about silencing messages like this if desired.)

Accessing the template

To use the vetiver model card template from RStudio, access through File -> New File -> R Markdown. This will open the dialog box where you can select from one of the available templates:

Create a new Model Card

If you are not using RStudio, you’ll also need to install Pandoc. Then, use the rmarkdown::draft() function to create the model card:

rmarkdown::draft(
    "my_model_card.Rmd", 
    template = "vetiver_model_card", 
    package = "vetiver"
)

Model card outline

There are several sections in the model card framework used here.

Can’t I just delete the section on ethical considerations?

It’s possible that a given machine learning model may not have obvious caveats, ethical challenges, or demographic aspects, or that they are largely unknown. However, we strongly advise that instead of deleting any such section because you have incomplete or imprecise information, you note your own process and considerations. Also, consider the possibility of gathering feedback from those impacted by the machine learning system, especially those with marginalized identities.

The process of documenting the extent and limits of a machine learning system is part of transparent, responsible reporting. A model card framework such as this is a helpful tool and some parts of a model card can be automated, but ultimately the extent of its value depends on you. From Mitchell et al. (2019):

Therefore the usefulness and accuracy of a model card relies on the integrity of the creator(s) of the card itself.