Skip to content

Module contributor

This section is dedicated for module contributors, basic users that would like to add a new module (dataset, method, metric) to an existing Omnibenchmark. This section will show how to add a new module to an existing Omnibenchmark.

This section will not show you how to create/ setup an Omnibenchmark (see the Benchmarker section) and how to visualize and interpret the evaluation results (see the Method user section).

Warning

Before starting, make sure that you have;

  1. Created an account to access Renku;

  2. Requested an access to the Omnibenchmark Gitlab group (Request Access button. If you don't see it, you were already granted access to it);

  3. A script for importing data, or a wrapper of a method or a metric.

1. Create a module

Any part of Omnibenchmark can be extended with an additional project that will add a new dataset, method or metric to the evaluation.

To ease the addition of a new project to Omnibenchmark, you can select a template when creating a new Renku project.

Info

A template is a project that was already pre-filled and configured to work with Omnibenchmark in a way that you won't have to know too much about Renku. You will just have to add your data and/or your code to project that will extend Omnibenchmark.

The Scribe guide below will show you how to create a new Omnibenchmark project for any module (data, method, metric,...).

If your benchmark is already in the list below, you can click on the module that you would like to add and jump to step 11 of the Scribe guide;

Batch-correction Omnibenchmark

Clustering Omnibenchmark:

Spatial clustering Omnibenchmark:

2. Set up a module

The basic steps required to set up a module are described in the README of your project. Depending on your project, you may also have to modify the following files of your project:

  • Dockerfile; extra linux (ubuntu) software requirements can be specified here (R and Python installed by default).

  • install.R; to manage extra R dependencies

  • requirements.txt; to manage extra Python dependencies

Detailed instruction can be found in the Reku documentation.

Upon any commit, Renku will automatically build a new Docker container based on these files.

Info

Having independent modules also means that you can test your code and work on your project without affecting the omnibenchmark until you submit it. Only when a project is included into an omnibenchmark 'orchestrator', it becomes part of the benchmark itself.

Detailed instructions for

  • Input data; Dataset with known ground-truth to run methods on.

  • Methods; Methods that will be applied on the (processed) datasets and have their output evaluated. Also use their

  • Metrics; Performance metrics used to evaluate methods outputs.

Input data

Data modules are modules that define input datasets and bundle them into a renku dataset, that can be imported by other projects. This page goes into details of some important elements to configure data. Most modules contain 3 main files:

1. The config.yaml file

The config.yaml (in your src directory) defines the metadata associated to your dataset and how it will be run. Upon creation of a dataset project, a src/config.yaml is already created and some metadata already pre-filled. Let's have a look how you can configure it; 

---
data:
    name: "dataset-name"
    title: "dataset title"
    description: "A new dataset module for omnibenchmark"
    keywords: ["MODULE_KEY"]
script: "path/to/module_script"
outputs:
    files:
        data_file1: 
            end: "FILE1_END"
        data_file2:
            end: "FILE2_END"
        data_file3:
            end: "FILE3_END"
benchmark_name: "OMNIBENCHMARK_TYPE"
  • data: name: (mandatory) Specifies the dataset name. Do not use special characters outside - and _.

Warning

Make the name as explicit as possible. If it (even partially) overlaps with other projects names, it might create some conflicts. For example, prefer iris_dataset_clustering instead of iris.

  • data: keywords: (mandatory) A keyword that is used for all datasets of this omnibenchmark. By default/ convention: [omnibenchmark name]_data. If you are not sure, you can check the keywords already in use in existing omnibenchmarks.

  • script: (mandatory) Relative path to the script to execute. More information about the script in the The module script section below.

  • outputs:files: (Mandatory) Name that will be passed to the output of the scripts (e.g. data_file1) and their file extension (end: field). It must match the arguments of you script (see The module script section below).

  • benchmark_name: (Mandatory) The name of the omnibenchmark this dataset belongs to. If you are not sure, you can check the names already in use in existing omnibenchmarks.

  • data: title: (Optional) A human readable title for this dataset. Can be the same as data: name:.

  • data: description: (Optional) Some description about this dataset.

More information about the config.yaml file in the Setup a config.yaml section.

2. The module script

This is the script to load the dataset and to convert its files into the expected format. Omnibenchmark accepts any kind of script and its maintenance and content is up to the module author. By default, an R and Python script are provided in the src folder that you can use to import or create your dataset. Omnibenchmark calls the script (specified in the script: field of config.yaml) from the command line. If you use another language than R or Python (e.g. Julia or bash) specify the interpreter to use in the corresponding field of the config.yaml file (script: field of config.yaml).

Note

All input and output files will automatically be parsed from the command line in the format: --ARGUMENT_NAME ARGUMENT_VALUE

You can find below a dummy example of how a config.yaml and its corresponding script can look like (in Python and R); 

---
data:
    name: "some-dataset"
    title: "some dataset"
    description: " "
    keywords: ["omnibenchmark-X_data"] 
script: "src/dataset.R" # (1)!
outputs:
    template: "data/${name}/${name}_${out_name}.${out_end}"
    files:
        dataset_output: # (2)!
            end: "csv" # (3)!
benchmark_name: "omnibenchmark-X"

  1. Can be replaced by another script or extension.

  2. The argument name that will be parsed when running the script. Has to match the argument name of your script! (showed in the next script)

  3. Specifies the file extension (attached to the argument name).

The corresponding script that creates a dataset_output.csv (following our example config.yaml) can thus look like this; 

# Load module
import argparse

# Get command line arguments and store them in args
parser=argparse.ArgumentParser()
parser.add_argument('--dataset_output', help='name') # (1)!
args=parser.parse_args()

# Call the argument
arg1 = args.argument_name 

# Import or create a dataset
# df=...

# Save as csv using parsed argument
df.to_csv(arg1, index=False) # (2)!

  1. Argument name should correspond to the one in config, here 'dataset_output'.

  2. No need to specify the file extension here. For outputs, it will automatically be parsed based on the 'end:' that you specified in your config.yaml. 

# Load package
library(optparse)

# Get list with command line arguments by name
option_list = list(
    make_option(c("--dataset_output"), # (1)!
            type="character", default=NULL, 
            help=" ", metavar="character")
); 

opt_parser = OptionParser(option_list=option_list);
opt = parse_args(opt_parser);

# Call the argument
arg1 <- opt$argument_name

# Import of create a dataset 
# df <- ...

write.csv(df, file = arg1) # (2)!

  1. Argument name should correspond to the one in config, here 'dataset_output'.

  2. No need to specify the file extension here. For outputs, it will automatically be parsed based on the 'end:' that you specified in your config.yaml.

For multiple outputs, simply add more fields to the outputs: files: of your config.yaml and the same names to argument parser of your script.

Special case: input data as files

In some cases the data have to be uploaded manually to the project instead of being fetched/ downloaded by a script. (Please note however that downloading the data from a script is more reproducible and allows to keep the data, and thus your benchmark, always up-to-date!)

For example, say that we have uploaded 2 data files in an input folder. The corresponding config.yaml and R/Python script would be; 

---
data:
    name: "some-dataset"
    title: "some dataset"
    description: " "
    keywords: ["omnibenchmark-X_data"] 
script: "src/dataset.R" 
inputs: # (1)!
    input_files: 
        import_this_dataset:
            input_data: "input/input_dataset.txt" # (2)!
            input_metadata: "input/input_metadata.txt"
outputs:
    template: "data/${name}/${name}_${out_name}.${out_end}"
    files:
        dataset_output:
            end: "csv" 
benchmark_name: "omnibenchmark-X"

  1. Section to add to your config.yaml

  2. Two new arguments, 'input_data' and 'input_metadata' will now be sent to your script, containing the path to your input files.

And the corresponding script; 

# Load module
import argparse

# Get command line arguments and store them in args
parser=argparse.ArgumentParser()
parser.add_argument('--dataset_output', help='name')
parser.add_argument('--input_data', help='name') # (1)!
parser.add_argument('--input_metadata', help='name') 
args=parser.parse_args()

# Call the argument
arg1 = args.argument_name 

# Import or create a dataset
# df=...

# Save as csv using parsed argument
df.to_csv(arg1, index=False)
  1. Argument names that correspond to the 2 'input:' fields of the config.yaml. 
# Load package
library(optparse)

# Get list with command line arguments by name
option_list = list(
    make_option(c("--dataset_output"),
            type="character", default=NULL, 
            help=" ", metavar="character"),
    make_option(c("--input_data"), # (1)!
            type="character", default=NULL, 
            help=" ", metavar="character"), 
    make_option(c("--input_metadata"),
            type="character", default=NULL, 
            help=" ", metavar="character")
); 

opt_parser = OptionParser(option_list=option_list);
opt = parse_args(opt_parser);

# Call the argument
arg1 <- opt$argument_name

# Import of create a dataset 
# df <- ...

write.csv(df, file = arg1)
  1. Argument name should correspond to the one in config, here 'dataset_output'.

3. The run_workflow.py file

This file is to generate, run and update the modules dataset and workflow. This is a standard script that will adapt to your config.yaml and dataset script and does not need to be (and should not be!) modified. Below is a copy of the run_workflow.py that you have in your project, with some explanations: 

# Load modules
from omnibenchmark.utils.build_omni_object import get_omni_object_from_yaml
from omnibenchmark.renku_commands.general import renku_save

# Build an OmniObject from the config.yaml file
omni_obj = get_omni_object_from_yaml('src/config.yaml') # (1)!

# Create the output dataset
omni_obj.create_dataset() # (2)!
renku_save()

## Run and update the workflow
omni_obj.run_renku()  # (3)!
renku_save()

## Add files to output dataset
omni_obj.update_result_dataset() # (4)!
renku_save()

  1. Load the options of your dataset into an Omnibenchmark object.

  2. Pay attention to any warning messages about names conflicts (it can lead to unpredictable conflicts latter in the benchmark). If it happens, change the 'name' in your config.yaml and rerun all previous command lines.

  3. This will run your script with parsed arguments.

  4. This will upload the created dataset to make it recognizable by further modules.

Once your module is completed and tested, you can submit it to the team.

Method modules

A method module imports all datasets of a benchmark or all preprocessed inputs and runs one benchmarking method on them. Method outputs are added to a renku dataset, that can be imported by metric projects to evaluate them. Benchmark specific requirements like file formats, types and prefixes can be checked here.

To test a method, a parameter dataset is imported to test the method on different parameter settings. Usually the parameter space is defined Valid parameters can be specified in the config.yaml file or on a filtered set of parameters (e.g. parameter limits, specific values and parameter and input file combinations). Usually a method module contains only one workflow, that is automatically run with all valid parameter and input file combinations. Most modules contain 3 main files:

1. The config.yaml file

The config.yaml (in your src directory) defines the metadata associated to your method and how it will be tested. Upon creation of a method project, a src/config.yaml is already created and some metadata already pre-filled. Let's have a look how you can configure it; 

---
data:
    name: "method name" # (1)! 
    title: "method title"
    description: "Short description of the method"
    keywords: ["MODULE_KEY"] # (2)!
script: "path/to/module_script" # (3)!
benchmark_name: "OMNIBENCHMARK_TYPE" # (4)!
inputs:
    keywords: ["INPUT_KEY"] # (5)!
    files: ["input_file_name1", "input_file_name2"]
    prefix:
        input_file_name1: "_INPUT1_"
        input_file_name2: "_INPUT2_"
outputs:
    files:
        method_result1: # (6)!
            end: "FILE1_END"
        method_result2:
            end: "FILE2_END"
parameter:
    keywords: ["PARAMETER_KEY"]# (7)!
    names: ["parameter1", "parameter2"]

  1. Specifies the method name. Do not use special characters outside - and _.

  2. A keyword that is used for all methods of this omnibenchmark. By default/ convention: [omnibenchmark name]_method. If you are not sure, you can check the keywords already in use in existing omnibenchmarks.

  3. Relative path to the script to execute. More information about the script in the The module script section below.

  4. The name of the omnibenchmark this method belongs to. If you are not sure, you can check the names already in use in existing omnibenchmarks.

  5. The keyword(s) of the input dataset to test the method on. If you are not sure, you can check the keywords already in use in existing omnibenchmarks.

  6. Name that will be passed to the output of the scripts (e.g. method_result1) and their file extension (end: field). It must match the arguments of you script (see The module script section below).

  7. The keyword of the input parameters to test the method on and their names. If you are not sure, you can check the keywords already in use in existing omnibenchmarks. You can also specify define specific parameters, see the dedicated documentation of the config.yaml file.

More information about the config.yaml file in the Setup a config.yaml section,

2. The module script

This is the script to apply a metric on test datasets. Omnibenchmark accepts any kind of script and its maintenance and content is up to the module author. By default, an R and Python script are provided in the src folder that you can use to add your method's wrapper. Omnibenchmark calls this script from the command line. If you use another language than R or Python (e.g. Julia or bash) specify the interpreter to use in the corresponding field of the config.yaml file (script: field of config.yaml).

In Python argparse can be used to parse command arguments. In R we recommend to use the optparse package. A basic example is provided in the two example scripts provided in your project.

Note

All input and output files will automatically be parsed from the command line in the format: --ARGUMENT_NAME ARGUMENT_VALUE

Below is a dummy example of how the R/Python scripts can look like given the example config.yaml from the previous section; 

# Load module
import argparse

# Get command line arguments and store them in args
parser=argparse.ArgumentParser()
parser.add_argument('--input_file_name1', help='name') # (1)!
parser.add_argument('--input_file_name2', help='name') 
parser.add_argument('--method_result1', help='name') 
parser.add_argument('--method_result2', help='name') 
parser.add_argument('--parameter1', help='name') 
parser.add_argument('--parameter2', help='name') 
args=parser.parse_args()

# Call the arguments
input_file_name1 = args.input_file_name1 
input_file_name2 = args.input_file_name2 
method_result1 = args.method_result1 
method_result2 = args.method_result2 
parameter1 = args.parameter1 
parameter2 = args.parameter2 

# Run the method to be benchmarked and create 2 outputs
# output1=...
# output2=...

# Save as csv using parsed argument
output1.to_csv(method_result1, index=False) # (2)!
output2.to_csv(method_result2, index=False)

  1. Argument names that correspond to the inputs, outputs and parameter fields specified in config.yaml.

  2. No need to specify the file extension here. For outputs, it will automatically be parsed based on the 'end:' that you specified in your config.yaml. 

# Load package
library(optparse)

# Get list with command line arguments by name
option_list = list(
    make_option(c("--input_file_name1"), # (1)!
        type="character", default=NULL, help=" "), 
    make_option(c("--input_file_name2"),
        type="character", default=NULL, help=" "), 
    make_option(c("--method_result1"),
        type="character", default=NULL, help=" "), 
    make_option(c("--method_result2"),
        type="character", default=NULL, help=" "), 
    make_option(c("--parameter1"),
        type="character", default=NULL, help=" "), 
    make_option(c("--parameter2"),
        type="character", default=NULL, help=" "), 
); 

opt_parser = OptionParser(option_list=option_list);
opt = parse_args(opt_parser);

# Call the argument
input_file_name1 <- opt$input_file_name1
input_file_name2 <- opt$input_file_name2
method_result1 <- opt$method_result1
method_result2 <- opt$method_result2
parameter1 <- opt$parameter1
parameter2 <- opt$parameter2

# Run the method to be benchmarked and create 2 outputs
# output1 <- ...
# output2 <- ...

# Save as csv using parsed argument
write.csv(output1, file = method_result1) # (2)!
write.csv(output2, file = method_result2)

  1. Argument names that correspond to the inputs, outputs and parameter fields specified in config.yaml.

  2. No need to specify the file extension here. For outputs, it will automatically be parsed based on the 'end:' that you specified in your config.yaml.

3. The run_workflow.py file

This file is to run and update the method workflow. This is a standard script that will adapt to your config.yaml and dataset script and does not need to be (and should not be!) modified. Below is a copy of the run_workflow.py that you have in your project, with some explanations: 

from omnibenchmark.utils.build_omni_object import get_omni_object_from_yaml
from omnibenchmark.renku_commands.general import renku_save
import omniValidator as ov
renku_save()

## Load config
omni_obj = get_omni_object_from_yaml('src/config.yaml')  # (1)!

## Update object and download input datasets
omni_obj.update_object(all=True) # (2)! 
renku_save()

## Check object
print(
    f"Object attributes: \n {omni_obj.__dict__}\n"
)
print(
    f"File mapping: \n {omni_obj.outputs.file_mapping}\n"
)
print(
    f"Command line: \n {omni_obj.command.command_line}\n" # (3)!
)

## Create output dataset
omni_obj.create_dataset() # (4)!

## Run workflow
ov.validate_requirements(omni_obj)
omni_obj.run_renku(all=False) # (5)!
renku_save()

## Update Output dataset
omni_obj.update_result_dataset() # (6)!
renku_save()

  1. Load the options of your dataset into an Omnibenchmark object.

  2. Downloads the datasets specified in inputs:. It case there are too many data to download, all can be set to False to only download a subset of the input data.

  3. This will show the command line and the parsed arguments that are going to be called to run the method. Can also be used for debugging.

  4. Pay attention to any warning messages about names conflicts (it can lead to unpredictable conflicts latter in the benchmark). If it happens, change the 'name' in your config.yaml and rerun all previous command lines.

  5. This will run your script with parsed arguments.

  6. This will upload the methods results to make it recognizable by further modules.

Once your module is completed and tested, you can submit it to the team.

Metric modules

A metric module imports method result datasets and runs one evaluation on them. Evaluation results are added to a renku dataset, that can be summarized and explored in an omnibenchmark dashboard. Benchmark specific requirements like file formats, types and prefixes can be checked at the omnibenchmark website. Usually a metric module contains two workflows, one to evaluate the results and one to generate the metric info file. Most metric modules contain 5 main files:

1. The config.yaml file

The config.yaml (in your src directory) defines the metadata associated to your method and how it will be tested. Upon creation of a method project, a src/config.yaml is already created and some metadata already pre-filled. Let's have a look how you can configure it; 

---
data:
    name: "metric name" # (1)! 
    title: "metric title"
    description: "Short description of the metric"
    keywords: ["MODULE_KEY"] # (2)!
script: "path/to/module_script" # (3)!
benchmark_name: "OMNIBENCHMARK_TYPE" # (4)!
inputs:
    keywords: ["INPUT_KEY"] # (5)!
    files: ["input_file_name1", "input_file_name2"]
    prefix:
        input_file_name1: "_INPUT1_"
        input_file_name2: "_INPUT2_"
outputs:
    files:
        metric_result: # (6)!
            end: "FILE1_END"

  1. Specifies the method name. Do not use special characters outside - and _.

  2. A keyword that is used for all metrics of this omnibenchmark. By default/ convention: [omnibenchmark name]_metric_. If you are not sure, you can check the keywords already in use in existing omnibenchmarks.

  3. Relative path to the script to execute. More information about the script in the The module script section below.

  4. The name of the omnibenchmark this method belongs to. If you are not sure, you can check the names already in use in existing omnibenchmarks.

  5. The keyword(s) of the methods results to evaluate. If you are not sure, you can check the keywords already in use in existing omnibenchmarks.

  6. Name that will be passed to the output of the scripts (e.g. metric_result) and their file extension (end: field). It must match the arguments of you script (see The module script section below).

More information about the config.yaml file in the Setup a config.yaml section.

2. The module script

This is the script to load the method results and evaluate them using metrics. Omnibenchmark accepts any kind of script and its maintenance and content is up to the module author. By default, an R and Python script are provided in the src folder that you can use to add your method's wrapper. Omnibenchmark calls this script from the command line. If you use another language than R or Python (e.g. Julia or bash) specify the interpreter to use in the corresponding field of the config.yaml file (script: field of config.yaml).

In Python argparse can be used to parse command arguments. In R we recommend to use the optparse package. A basic example is provided in the two example scripts provided in your project.

Note

All input and output files will automatically be parsed from the command line in the format: --ARGUMENT_NAME ARGUMENT_VALUE

Below is a dummy example of how the R/Python scripts can look like given the example config.yaml from the previous section; 

# Load module
import argparse

# Get command line arguments and store them in args
parser=argparse.ArgumentParser()
parser.add_argument('--input_file_name1', help='Description of the argument') # (1)!
parser.add_argument('--input_file_name2', help='Description of the argument')
parser.add_argument('--metric_result', help='Description of the argument')
args=parser.parse_args()

# Call the argument
input_file_name1 = args.input_file_name1
input_file_name2 = args.input_file_name2
metric_result = args.metric_result

# Run the metric to benchmark the methods results
# output=...

# Save as csv using parsed argument
output.to_csv(metric_result, index=False) # (2)!

  1. Argument names that correspond to the inputs and outputs fields specified in config.yaml.

  2. No need to specify the file extension here. For outputs, it will automatically be parsed based on the 'end:' that you specified in your config.yaml. 

# Load package
library(optparse)

# Get list with command line arguments by name
option_list = list(
    make_option(c("--input_file_name1"), # (1)!
        type="character", default=NULL, help=" "), 
    make_option(c("--input_file_name2"),
        type="character", default=NULL, help=" "), 
    make_option(c("--method_result"),
        type="character", default=NULL, help=" ")
); 

opt_parser = OptionParser(option_list=option_list);
opt = parse_args(opt_parser);

# Call the argument
input_file_name1 <- opt$input_file_name1
input_file_name2 <- opt$input_file_name2
method_result <- opt$method_result

# Run the method to be benchmarked and create 2 outputs
# output <- ...

# Save as csv using parsed argument
write.csv(output, file = metric_result) # (2)!

  1. Argument names that correspond to the inputs and outputs fields specified in config.yaml.

  2. No need to specify the file extension here. For outputs, it will automatically be parsed based on the 'end:' that you specified in your config.yaml.

3. The generate_metric_info.py file (optional)

This file could also be written in R or any other language but one is automatically provided in your src folder. You can change the metadata of the metric (indicated by a +), which will be used for its display into the dashboard showing the metric results. All options can also be left as default. 

import argparse
import json

parser=argparse.ArgumentParser()
parser.add_argument('--metric_info', help='Path to the metric info json file')
args=parser.parse_args()

metric_info = {
    'flip': False, # (1)!
    'max': 1, # (2)!
    'min': 0, # (3)!
    'group': "METRIC_GROUP", # (4)!
    'name': "metric_name", # (5)!
    'input': "metric_input_type" # (6)!
}

with open(args.metric_info, "w") as fp:
    json.dump(metric_info , fp, indent=3)

  1. Logical scalar; whether or not to flip the sign of the metric values on the dashboard app.

  2. Possible maximum value of the metric.

  3. Possible minimum value of the metric.

  4. If different groups of metrics are added, a name can be given for grouping. Can also be left as 'default'.

  5. Metric name that will be displayed on the dashboard.

  6. Prefix to name the result files. Can be left as default; 'metric_res'.

4. The info_config.yaml file (optional)

This file contains file to complement the generate_metric_info.py. All fields can be left as default.

5. The run_workflow.py file

This file is to run and update the metric project. This is a standard script that will adapt to your config.yaml and dataset script and does not need to be (and should not be!) modified. Below is a copy of the run_workflow.py that you have in your project, with some explanations: 

from omnibenchmark.utils.build_omni_object import get_omni_object_from_yaml
from omnibenchmark.renku_commands.general import renku_save
import omniValidator as ov
renku_save()

## Load config
omni_obj = get_omni_object_from_yaml('src/config.yaml')  # (1)!

## Update object and download input datasets
omni_obj.update_object(all=True)  # (2)!
renku_save()

## Check object
print(
    f"Object attributes: \n {omni_obj.__dict__}\n"
)
print(
    f"File mapping: \n {omni_obj.outputs.file_mapping}\n"
)
print(
    f"Command line: \n {omni_obj.command.command_line}\n"  # (3)!
)

## Create output dataset
omni_obj.create_dataset() # (4)!

## Run workflow
ov.validate_requirements(omni_obj)
omni_obj.run_renku(all=False)  # (5)!
renku_save()

## Update Output dataset
omni_obj.update_result_dataset() # (6)!
renku_save()

###################### Generate info file ###################### # (7)!

omni_info = get_omni_object_from_yaml('src/info_config.yaml') 
omni_info.wflow_name = "{{ sanitized_project_name }}_info"

## Check object
print(
    f"Object attributes: \n {omni_info.__dict__}\n"
)
print(
    f"File mapping: \n {omni_info.outputs.file_mapping}\n"
)
print(
    f"Command line: \n {omni_info.command.command_line}\n"
)

## Run workflow
omni_info.run_renku()
renku_save()

## Update Output dataset
omni_info.update_result_dataset()
renku_save()

################################################################

  1. Load the options of your dataset into an Omnibenchmark object.

  2. Downloads the datasets specified in inputs:. It case there are too many data to download, all can be set to False to only download a subset of the input data.

  3. This will show the command line and the parsed arguments that are going to be called to run the method. Can also be used for debugging.

  4. Pay attention to any warning messages about names conflicts (it can lead to unpredictable conflicts latter in the benchmark). If it happens, change the 'name' in your config.yaml and rerun all previous command lines.

  5. This will run your script with parsed arguments.

  6. This will upload the methods results to make it recognizable by further modules.

  7. Similar as before, but for the metrics metadata.

Once your module is completed and tested, you can submit it to the team.

Submit a module to an Omnibenchmark

An omnibenchmark module will be able to import datasets from other modules and export its output to others. However, it still needs to be integrated in an Omnibenchmark orchestrator. An orchestrator is an omnibenchmark project which orchestrates the deployment, running and testing of each pieces of an Omnibenchmark.

Integrate a module to an orchestrator

The list of current Omnibenchmarks and their related orchestrator can be found on the Omnibenchmark dashboard:

To integrate your (populated and tested) module:

  • Follow the link to the relevant orchestrator

  • Open a new issue and describe briefly the aim of your module (data/ method/ metric module ?) and provide a link to it

  • The development team will check your module and integrate it in the Orchestrator worfkflow. When it is done, you will be able to view the result of your module on the shiny app.

Last update: December 11, 2023