File size: 5,436 Bytes
7accb91 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
# Dataset splits vs. filtered training / test splits
The NAVSIM framework utilizes several dataset splits for standardized training and evaluating agents.
All of them use the OpenScene dataset that is divided into the dataset splits `mini`,`trainval`,`test`,`private_test_e2e`, which can all be downloaded separately.
It is possible to run trainings and evaluations directly on these sets (see `Standard` in table below).
Alternatively, you can run trainings and evaluations on training and validation splits that were filtered for challenging scenarios (see `NAVSIM` in table below), which is the recommended option for producing comparable and competitive results efficiently.
In contrast to the dataset splits which refer to a downloadable set of logs, the training / test splits are implemented as scene filters, which define how scenes are extracted from these logs.
The NAVSIM training / test splits subsample the OpenScene dataset splits.
Moreover, the NAVSIM splits include overlapping scenes, while the Standard splits are non-overlapping.
Specifically, `navtrain` is based on the `trainval` data and `navtest` on the `test` data.
As the `trainval` sensor data is very large, we provide a separate download link, which loads only the frames needed for `navtrain`.
This eases access for users that only want to run the `navtrain` split and not the `trainval` split. If you already downloaded the full `trainval` sensor data, it is **not necessary** to download the `navtrain` frames as well.
The logs are always the complete dataset split.
## Overview
The Table belows offers an overview on the training and test splits supported by NAVSIM.
In Navsim-v1.1, the training/test split can bet set with a single config parameter given in the table.
<table border="0">
<tr>
<th></th>
<th>Name</th>
<th>Description</th>
<th>Logs</th>
<th>Sensors</th>
<th>Config parameters</th>
</tr>
<tr>
<td rowspan="3">Standard</td>
<td>trainval</td>
<td>Large split for training and validating agents with regular driving recordings. Corresponds to nuPlan and downsampled to 2HZ.</td>
<td>14GB</td>
<td>>2000GB</td>
<td>
train_test_split=trainval
</td>
</tr>
<tr>
<td>test</td>
<td>Small split for testing agents with regular driving recordings. Corresponds to nuPlan and downsampled to 2HZ.</td>
<td>1GB</td>
<td>217GB</td>
<td>
train_test_split=test
</td>
</tr>
<tr>
<td>mini</td>
<td>Demo split for with regular driving recordings. Corresponds to nuPlan and downsampled to 2HZ.</td>
<td>1GB</td>
<td>151GB</td>
<td>
train_test_split=mini
</td>
</tr>
<tr>
<td rowspan="2">NAVSIM</td>
<td>navtrain</td>
<td>Standard split for training agents in NAVSIM with non-trivial driving scenes. Sensors available separately in <a href="https://github.com/autonomousvision/navsim/blob/main/download/download_navtrain.sh">download_navtrain.sh</a>.</td>
<td>-</td>
<td>445GB*</td>
<td>
train_test_split=navtrain
</td>
</tr>
<tr>
<td>navtest</td>
<td>Standard split for testing agents in NAVSIM with non-trivial driving scenes. Available as a filter for test split.</td>
<td>-</td>
<td>-</td>
<td>
train_test_split=navtest
</td>
</tr>
<tr>
<td rowspan="2">Competition</td>
<td>warmup_test_e2e</td>
<td>Warmup test split to validate submission on hugging face. Available as a filter for mini split.</td>
<td>-</td>
<td>-</td>
<td>
train_test_split=warmup_test_e2e
</td>
</tr>
<tr>
<td>private_test_e2e</td>
<td>Private test split for the challenge leaderboard on hugging face.</td>
<td><1GB</td>
<td>25GB</td>
<td>
train_test_split=private_test_e2e
</td>
</tr>
</table>
(*300GB without history)
## Splits
The standard splits `trainval`, `test`, and `mini` are from the OpenScene dataset. Note that the data corresponds to the nuPlan dataset with a lower frequency of 2Hz. You can download all standard splits over Hugging Face with the bash scripts in [download](../download)
NAVSIM provides a subset and filter of the `trainval` split, called `navtrain`. The `navtrain` split facilitates a standardized training scheme and requires significantly less sensor data storage than `travel` (445GB vs. 2100GB). If your agents don't need historical sensor inputs, you can download `navtrain` without history, which requires 300GB of storage. Note that `navtrain` can be downloaded separately via [download_navtrain.sh](https://github.com/autonomousvision/navsim/blob/main/download/download_navtrain.sh) but still requires access to the `trainval` logs. Similarly, the `navtest` split enables a standardized set for testing agents with a provided scene filter. Both `navtrain` and `navtest` are filtered to increase interesting samples in the sets.
For the challenge on Hugging Face, we provide the `warmup_test_e2e` and `private_test_e2e` for the warm-up and challenge track, respectively. Note that `private_test_e2e` requires you to download the data, while `warmup_test_e2e` is a scene filter for the `mini` split.
|