Part 6 Proposed interface: using models

6.1 Performance assessment

During the training process for a model family, the fit method will train a large number of models. Presumably it’d be nice to keep track of all these hyperparameter combinations in a tibble.

We also want to keep track of model performance for each hyperparameter combination. To keep data tidy, this information should live in a separate table where rows correspond to unique models and columns correspond to performance on a particular resampled dataset (i.e. fold).

In some cases users will want multiple or even many performance metrics for each fold. I think it makes sense for each metric to get it’s own results table? So users could do

trained_model_family\(rmse # get rmse results on all resampled datasets trained_model_family\)mae # same, but for mae

Presumably these tables should include some summary statistics, etc. Not sure how I feel about requiring the user to know the name of the metric to access the results tables.