Part 6 Proposed interface: using models
6.1 Performance assessment
During the training process for a model family, the fit method will train a large number of models. Presumably it’d be nice to keep track of all these hyperparameter combinations in a tibble.
We also want to keep track of model performance for each hyperparameter combination. To keep data tidy, this information should live in a separate table where rows correspond to unique models and columns correspond to performance on a particular resampled dataset (i.e. fold).
In some cases users will want multiple or even many performance metrics for each fold. I think it makes sense for each metric to get it’s own results table? So users could do
trained_model_family\(rmse # get rmse results on all resampled datasets trained_model_family\)mae # same, but for mae
Presumably these tables should include some summary statistics, etc. Not sure how I feel about requiring the user to know the name of the metric to access the results tables.