2.6 Benchmarking

Comparing the performance of different learners on multiple tasks and/or different resampling schemes is a common task. This operation is usually referred to as “benchmarking” in the field of machine-learning. The mlr3 package offers the benchmark() function for convenience.

2.6.1 Design Creation

In mlr3 we require you to supply a “design” of your benchmark experiment. By “design” we essentially mean the matrix of settings you want to execute. A “design” consists of unique combinations of Task, Learner and Resampling triplets.

Here, we call benchmark() to perform a single holdout split on a single task and two learners. We use the benchmark_grid() function to create an exhaustive design and instantiate the resampling properly, so that all learners are executed on the same train/test split for each task:

library("data.table")
design = benchmark_grid(
  tasks = tsk("iris"),
  learners = list(lrn("classif.rpart"), lrn("classif.featureless")),
  resamplings = rsmp("holdout")
)
print(design)

##             task                     learner          resampling
## 1: <TaskClassif>       <LearnerClassifRpart> <ResamplingHoldout>
## 2: <TaskClassif> <LearnerClassifFeatureless> <ResamplingHoldout>

bmr = benchmark(design)

Instead of using benchmark_grid() you could also create the design manually as a data.table and use the full flexibility of the benchmark() function. The design does not have to be exhaustive, e.g. it can also contain a different learner for each task. However, you should note that benchmark_grid() makes sure to instantiate the resamplings for each task. If you create the design manually, even if the same task is used multiple times, the train/test splits will be different for each row of the design if you do not manually instantiate the resampling before creating the design.

Let’s construct a more complex design to show the full capabilities of the benchmark() function.

# get some example tasks
tasks = lapply(c("german_credit", "sonar"), tsk)

# get some learners and for all learners ...
# * predict probabilities
# * predict also on the training set
library("mlr3learners")
learners = c("classif.featureless", "classif.rpart", "classif.ranger", "classif.kknn")
learners = lapply(learners, lrn,
  predict_type = "prob", predict_sets = c("train", "test"))

# compare via 3-fold cross validation
resamplings = rsmp("cv", folds = 3)

# create a BenchmarkDesign object
design = benchmark_grid(tasks, learners, resamplings)
print(design)

##             task                     learner     resampling
## 1: <TaskClassif> <LearnerClassifFeatureless> <ResamplingCV>
## 2: <TaskClassif>       <LearnerClassifRpart> <ResamplingCV>
## 3: <TaskClassif>      <LearnerClassifRanger> <ResamplingCV>
## 4: <TaskClassif>        <LearnerClassifKKNN> <ResamplingCV>
## 5: <TaskClassif> <LearnerClassifFeatureless> <ResamplingCV>
## 6: <TaskClassif>       <LearnerClassifRpart> <ResamplingCV>
## 7: <TaskClassif>      <LearnerClassifRanger> <ResamplingCV>
## 8: <TaskClassif>        <LearnerClassifKKNN> <ResamplingCV>

2.6.2 Execution and Aggregation of Results

After the benchmark design is ready, we can directly call benchmark():

# execute the benchmark
bmr = benchmark(design)

Note that we did not instantiate the resampling instance manually. benchmark_grid() took care of it for us: Each resampling strategy is instantiated once for each task during the construction of the exhaustive grid.

After the benchmark, we can calculate and aggregate the performance with $aggregate():

# measures:
# * area under the curve (auc) on training
# * area under the curve (auc) on test
measures = list(
  msr("classif.auc", id = "auc_train", predict_sets = "train"),
  msr("classif.auc", id = "auc_test")
)
bmr$aggregate(measures)

## # A tibble: 8 x 8
##      nr resample_result task_id learner_id resampling_id iters auc_train
##   <int> <list>          <chr>   <chr>      <chr>         <int>     <dbl>
## 1     1 <RsmplRsl>      german… classif.f… cv                3     0.5  
## 2     2 <RsmplRsl>      german… classif.r… cv                3     0.803
## 3     3 <RsmplRsl>      german… classif.r… cv                3     0.999
## 4     4 <RsmplRsl>      german… classif.k… cv                3     0.989
## 5     5 <RsmplRsl>      sonar   classif.f… cv                3     0.5  
## 6     6 <RsmplRsl>      sonar   classif.r… cv                3     0.920
## 7     7 <RsmplRsl>      sonar   classif.r… cv                3     1    
## 8     8 <RsmplRsl>      sonar   classif.k… cv                3     0.999
## # … with 1 more variable: auc_test <dbl>

Subsequently, we can aggregate the results further. For example, we might be interested which learner performed best over all tasks simultaneously. Simply aggregating the performances with the mean is usually not statistically sound. Instead, we calculate the rank statistic for each learner grouped by task. Then the calculated ranks grouped by learner are aggregated with data.table. Since the AUC needs to be maximized, we multiply with $-1$ so that the best learner gets a rank of $1$.

tab = bmr$aggregate(measures)
print(tab)

##    nr  resample_result       task_id          learner_id resampling_id iters
## 1:  1 <ResampleResult> german_credit classif.featureless            cv     3
## 2:  2 <ResampleResult> german_credit       classif.rpart            cv     3
## 3:  3 <ResampleResult> german_credit      classif.ranger            cv     3
## 4:  4 <ResampleResult> german_credit        classif.kknn            cv     3
## 5:  5 <ResampleResult>         sonar classif.featureless            cv     3
## 6:  6 <ResampleResult>         sonar       classif.rpart            cv     3
## 7:  7 <ResampleResult>         sonar      classif.ranger            cv     3
## 8:  8 <ResampleResult>         sonar        classif.kknn            cv     3
##    auc_train auc_test
## 1:    0.5000   0.5000
## 2:    0.8028   0.6961
## 3:    0.9986   0.7956
## 4:    0.9892   0.6968
## 5:    0.5000   0.5000
## 6:    0.9196   0.7531
## 7:    1.0000   0.9093
## 8:    0.9985   0.9011

# group by levels of task_id, return columns:
# - learner_id
# - rank of col '-auc_train' (per level of learner_id)
# - rank of col '-auc_test' (per level of learner_id)
ranks = tab[, .(learner_id, rank_train = rank(-auc_train), rank_test = rank(-auc_test)), by = task_id]
print(ranks)

##          task_id          learner_id rank_train rank_test
## 1: german_credit classif.featureless          4         4
## 2: german_credit       classif.rpart          3         3
## 3: german_credit      classif.ranger          1         1
## 4: german_credit        classif.kknn          2         2
## 5:         sonar classif.featureless          4         4
## 6:         sonar       classif.rpart          3         3
## 7:         sonar      classif.ranger          1         1
## 8:         sonar        classif.kknn          2         2

# group by levels of learner_id, return columns:
# - mean rank of col 'rank_train' (per level of learner_id)
# - mean rank of col 'rank_test' (per level of learner_id)
ranks = ranks[, .(mrank_train = mean(rank_train), mrank_test = mean(rank_test)), by = learner_id]

# print the final table, ordered by mean rank of AUC test
ranks[order(mrank_test)]

## # A tibble: 4 x 3
##   learner_id          mrank_train mrank_test
##   <chr>                     <dbl>      <dbl>
## 1 classif.ranger                1          1
## 2 classif.kknn                  2          2
## 3 classif.rpart                 3          3
## 4 classif.featureless           4          4

Unsurprisingly, the featureless learner is outperformed on both training and test set.

2.6.3 Plotting Benchmark Results

Analogously to plotting tasks, predictions or resample results, mlr3viz also provides a autoplot() method for benchmark results.

library("mlr3viz")
library("ggplot2")

autoplot(bmr) + theme(axis.text.x = element_text(angle = 45, hjust = 1))

We can also plot ROC curves. To do so, we first need to filter the BenchmarkResult to only contain a single Task:

autoplot(bmr$clone()$filter(task_id = "german_credit"), type = "roc")

All available types are listed on the manual page of autoplot.BenchmarkResult().

2.6.4 Extracting ResampleResults

A BenchmarkResult object is essentially a collection of multiple ResampleResult objects. As these are stored in a column of the aggregated data.table(), we can easily extract them:

tab = bmr$aggregate(measures)
rr = tab[task_id == "sonar" & learner_id == "classif.ranger"]$resample_result[[1]]
print(rr)

## <ResampleResult> of 3 iterations
## * Task: sonar
## * Learner: classif.ranger
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterations

We can now investigate this resampling and even single resampling iterations using one of the approaches shown in the previous section:

measure = msr("classif.auc")
rr$aggregate(measure)

## classif.auc 
##      0.9093

# get the iteration with worst AUC
perf = rr$score(measure)
i = which.min(perf$classif.auc)

# get the corresponding learner and train set
print(rr$learners[[i]])

## <LearnerClassifRanger:classif.ranger>
## * Model: -
## * Parameters: list()
## * Packages: ranger
## * Predict Type: prob
## * Feature types: logical, integer, numeric, character, factor, ordered
## * Properties: importance, multiclass, oob_error, twoclass, weights

head(rr$resampling$train_set(i))

## [1]  1  2  3 13 17 26

2.6.5 Converting and Merging ResampleResults

It is also possible to cast a single ResampleResult to a BenchmarkResult using the converter as_benchmark_result().

task = tsk("iris")
resampling = rsmp("holdout")$instantiate(task)

rr1 = resample(task, lrn("classif.rpart"), resampling)
rr2 = resample(task, lrn("classif.featureless"), resampling)

# Cast both ResampleResults to BenchmarkResults
bmr1 = as_benchmark_result(rr1)
bmr2 = as_benchmark_result(rr2)

# Merge 2nd BMR into the first BMR
bmr1$combine(bmr2)

bmr1

## <BenchmarkResult> of 2 rows with 2 resampling runs
##  nr task_id          learner_id resampling_id iters warnings errors
##   1    iris       classif.rpart       holdout     1        0      0
##   2    iris classif.featureless       holdout     1        0      0