This the multi-page printable view of this section. Click here to print.
Optionally, set up a virtual environment:
Install:
You can change the installation branch with
...@<branch_name>Also use--force-reinstallparameter in this case.
As a standalone tool:
As a python module:
The directory containing Datumaro should be in the
PYTHONPATHenvironment variable orcvat/datumaro/should be the current directory.
As a python library:
List of supported formats:
image_info, instances, person_keypoints, captions, labels,panoptic, stuff)
labels are our extension - like instances with only category_idclassification, detection, segmentation (class, instances),
action_classification, person_layout)
bboxes)
bboxes, masks)
bboxes)
landmarks, bboxes)
classification, detection)
classification (python version))
classification)
classification)
segmentation)
segmentation)
segmentation, detection)
raw/tracklets/velodyne points)
pointcloud)
word_recognition, text_localization, text_segmentation)
person re-identification)
classification, person re-identification, landmarks)
List of supported annotation types:
Datumaro only works with 2d RGB(A) images.
To create an unlabelled dataset from an arbitrary directory with images use
ImageDir format:
or if you work with Datumaro API:
For using with a project:
And for using as a dataset:
This will search for images in the directory recursively and add
them as dataset entries with names like <subdir1>/<subsubdir1>/<image_name1>.
The list of formats matches the list of supported image formats in OpenCV.
.jpg, .jpeg, .jpe, .jp2, .png, .bmp, .dib, .tif, .tiff, .tga, .webp, .pfm,
.sr, .ras, .exr, .hdr, .pic, .pbm, .pgm, .ppm, .pxm, .pnm
After addition into a project, images can be split into subsets and renamed with transformations, filtered, joined with existing annotations etc.
To use a video as an input, one should either create an Extractor plugin, which splits a video into frames, or split the video manually and import images.
The key object is a project, so most CLI commands operate on projects.
However, there are few commands operating on datasets directly.
A project is a combination of a project’s own dataset, a number of
external data sources and an environment.
An empty Project can be created by project create command,
an existing dataset can be imported with project import command.
A typical way to obtain projects is to export tasks in CVAT UI.
If you want to interact with models, you need to add them to project first.
└── project/
├── .datumaro/
| ├── config.yml
│ ├── .git/
│ ├── models/
│ └── plugins/
│ ├── plugin1/
│ | ├── file1.py
│ | └── file2.py
│ ├── plugin2.py
│ ├── custom_extractor1.py
│ └── ...
├── dataset/
└── sources/
├── source1
└── ...
Note: command invocation syntax is subject to change, always refer to command –help output
Available CLI commands:
This command allows to convert a dataset from one format into another.
In fact, this command is a combination of project import and project export
and just provides a simpler way to obtain the same result when no extra options
is needed. A list of supported formats can be found in the --help output of
this command.
Usage:
Example: convert a VOC-like dataset to a COCO-like one:
The command creates an empty project. Once a Project is created, there are a few options to interact with it.
Usage:
Example: create an empty project my_dataset
A Project can contain a number of external Data Sources. Each Data Source
describes a way to produce dataset items. A Project combines dataset items from
all the sources and its own dataset into one composite dataset. You can manage
project sources by commands in the source command line context.
Datasets come in a wide variety of formats. Each dataset format defines its own data structure and rules on how to interpret the data. For example, the following data structure is used in COCO format:
/dataset/
- /images/<id>.jpg
- /annotations/
Supported formats are listed in the command help. Check extending tips for information on extra format support.
Usage:
Example: create a project from a bunch of different annotations and images, and generate TFrecord for TF Detection API for model training
This command allows to create a sub-Project from a Project. The new project includes only items satisfying some condition. XPath is used as a query format.
There are several filtering modes available (-m/--mode parameter).
Supported modes:
i, itemsa, annotationsi+a, a+i, items+annotations, annotations+itemsWhen filtering annotations, use the items+annotations
mode to point that annotation-less dataset items should be
removed. To select an annotation, write an XPath that
returns annotation elements (see examples).
Usage:
Example: extract a dataset with only images which width < height
Example: extract a dataset with only images of subset train.
Example: extract a dataset with only large annotations of class cat and any
non-persons
Example: extract a dataset with only occluded annotations, remove empty images
Item representations are available with --dry-run parameter:
This command updates items in a project from another one (check Merge Projects for complex merging).
Usage:
Example: update annotations in the first_project with annotations
from the second_project and save the result as merged_project
This command creates a Project from an existing dataset.
Supported formats are listed in the command help. Check extending tips for information on extra format support.
Usage:
Example: create a project from COCO-like dataset
An MS COCO-like dataset should have the following directory structure:
COCO/
├── annotations/
│ ├── instances_val2017.json
│ ├── instances_train2017.json
├── images/
│ ├── val2017
│ ├── train2017
Everything after the last _ is considered a subset name in the COCO format.
This command exports a Project as a dataset in some format.
Supported formats are listed in the command help. Check extending tips for information on extra format support.
Usage:
Example: save project as VOC-like dataset, include images, convert images to PNG
This command merges items from 2 or more projects and checks annotations for errors.
Spatial annotations are compared by distance and intersected, labels and
attributes are selected by voting.
Merge conflicts, missing items and annotations, other errors are saved into a .json file.
Usage:
Example: merge 4 (partially-)intersecting projects,
person, hand, head and foot
(? for optional)The command compares two datasets and saves the results in the specified directory. The current project is considered to be “ground truth”.
Example: compare a dataset with model inference
This command outputs project status information.
Usage:
Example:
This command computes various project statistics, such as:
Usage:
Example:
This command inspects annotations with respect to the task type and stores the result in JSON file.
The task types supported are classification, detection, and segmentation.
The validation result contains
annotation statistics based on the task typevalidation reports, such as
summaryUsage:
few_samples_thr : threshold for giving a warning for minimum number of
samples per classimbalance_ratio_thr : threshold for giving imbalance data warningfar_from_mean_thr : threshold for giving a warning that data is far
from meandominance_ratio_thr : threshold for giving a warning bounding box
imbalancetopk_bins : ratio of bins with the highest number of data to total bins
in the histogramExample : give warning when imbalance ratio of data with classification task over 40
Here is the list of validation items(a.k.a. anomaly types).
| Anomaly Type | Description | Task Type |
|---|---|---|
| MissingLabelCategories | Metadata (ex. LabelCategories) should be defined | common |
| MissingAnnotation | No annotation found for an Item | common |
| MissingAttribute | An attribute key is missing for an Item | common |
| MultiLabelAnnotations | Item needs a single label | classification |
| UndefinedLabel | A label not defined in the metadata is found for an item | common |
| UndefinedAttribute | An attribute not defined in the metadata is found for an item | common |
| LabelDefinedButNotFound | A label is defined, but not found actually | common |
| AttributeDefinedButNotFound | An attribute is defined, but not found actually | common |
| OnlyOneLabel | The dataset consists of only label | common |
| OnlyOneAttributeValue | The dataset consists of only attribute value | common |
| FewSamplesInLabel | The number of samples in a label might be too low | common |
| FewSamplesInAttribute | The number of samples in an attribute might be too low | common |
| ImbalancedLabels | There is an imbalance in the label distribution | common |
| ImbalancedAttribute | There is an imbalance in the attribute distribution | common |
| ImbalancedDistInLabel | Values (ex. bbox width) are not evenly distributed for a label | detection, segmentation |
| ImbalancedDistInAttribute | Values (ex. bbox width) are not evenly distributed for an attribute | detection, segmentation |
| NegativeLength | The width or height of bounding box is negative | detection |
| InvalidValue | There’s invalid (ex. inf, nan) value for bounding box info. | detection |
| FarFromLabelMean | An annotation has an too small or large value than average for a label | detection, segmentation |
| FarFromAttrMean | An annotation has an too small or large value than average for an attribute | detection, segmentation |
Validation Result Format:
item_key is defined as,
bbox_template and mask_template are defined as,
numerical_stat_template is defined as,
Supported models:
launchersUsage:
Example: register an OpenVINO model
A model consists of a graph description and weights. There is also a script used to convert model outputs to internal data structures.
Interpretation script for an OpenVINO detection model (convert.py):
You can find OpenVINO model interpreter samples in
datumaro/plugins/openvino/samples (instruction).
This command applies model to dataset images and produces a new project.
Usage:
Example: launch inference on a dataset
Runs an explainable AI algorithm for a model.
This tool is supposed to help an AI developer to debug a model and a dataset. Basically, it executes inference and tries to find problems in the trained model - determine decision boundaries and belief intervals for the classifier.
Currently, the only available algorithm is RISE (article), which runs inference and then re-runs a model multiple times on each image to produce a heatmap of activations for each output of the first inference. As a result, we obtain few heatmaps, which shows, how image pixels affected the inference result. This algorithm doesn’t require any special information about the model, but it requires the model to return all the outputs and confidences. The algorithm only supports classification and detection models.
The following use cases available:
Usage:
Example: run inference explanation on a single image with visualization
Note: this algorithm requires the model to return all (or a reasonable amount) the outputs and confidences unfiltered, i.e. all the
Labelannotations for classification models and all theBboxes for detection models. You can find examples of the expected model outputs intests/test_RISE.py
For OpenVINO models the output processing script would look like this:
Classification scenario:
Object Detection scenario:
This command allows to modify images or annotations in a project all at once.
Example: split a dataset randomly to train and test subsets, ratio is 2:1
Example: split a dataset in task-specific manner. The tasks supported are classification, detection, segmentation and re-identification.
Example: convert polygons to masks, masks to boxes etc.:
Example: remap dataset labels, person to car and cat to dog,
keep bus, remove others
Example: rename dataset items by a regular expression
pattern with replacementframe_ from item idsExample: sampling dataset items as many as the number of target samples with
sampling method entered by the user, divide into sampled and unsampled
subsets
topk: Return the k with high uncertainty datalowk: Return the k with low uncertainty datarandk: Return the random k datamixk: Return half to topk method and the rest to lowk methodrandtopk: First, select 3 times the number of k randomly, and return
the topk among them.Example : control number of outputs to 100 after NDR
random: sample from removed data randomlysimilarity: sample from removed data with ascendinguniform: sample data with uniform distributioninverse: sample data with reciprocal of the numberThere are few ways to extend and customize Datumaro behavior, which is supported by plugins. Check our contribution guide for details on plugin implementation. In general, a plugin is a Python code file. It must be put into a plugin directory:
<project_dir>/.datumaro/plugins for project-specific plugins<datumaro_dir>/plugins for global pluginsDatumaro provides several builtin plugins. Plugins can have dependencies, which need to be installed separately.
The plugin provides support of TensorFlow Detection API format, which includes
boxes and masks. It depends on TensorFlow, which can be installed with pip:
This plugin allows to use Accuracy Checker
to launch deep learning models from various frameworks
(Caffe, MxNet, PyTorch, OpenVINO, …) through Accuracy Checker’s API.
The plugin depends on Accuracy Checker, which can be installed with pip:
This plugin provides support for model inference with OpenVINO™. The plugin depends on the OpenVINO™ Toolkit, which can be installed by following these instructions
Dataset reading is supported by Extractors and Importers. An Extractor produces a list of dataset items corresponding to the dataset. An Importer creates a project from the data source location. It is possible to add custom Extractors and Importers. To do this, you need to put an Extractor and Importer implementation scripts to a plugin directory.
Dataset writing is supported by Converters. A Converter produces a dataset of a specific format from dataset items. It is possible to add custom Converters. To do this, you need to put a Converter implementation script to a plugin directory.
A Transform is a function for altering a dataset and producing a new one. It can update dataset items, annotations, classes, and other properties. A list of available transforms for dataset conversions can be extended by adding a Transform implementation script into a plugin directory.
A list of available launchers for model execution can be extended by adding a Launcher implementation script into a plugin directory.