Usage

The main form of interaction with duqtools is via the command-line interface. This page documents the different subcommands available.

To get started:

duqtools --help

For information on the subcommands, e.g. create:

duqtools create --help

For information on how to configure duqtools via duqtools.yaml, check out the usage page.

duqtools

For more information, check out the documentation:

https://duqtools.readthedocs.io

Usage:

duqtools [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

Subcommands

clean: Delete generated IDS data and the run dir.
create: Read duqtools.yaml and create the new IMAS data files from template
dash: Open dashboard for evaluating IDS data.
go: Run create, submit, status, dash in succession.
init: Create a default config file.
list-variables: List available variables.
merge: Merge data sets with error propagation.
plot: Generate plots for IMAS data.
recreate: Read runs.yaml and re-create the given runs.
setup: Template substitution for duqtools config.
status: Print the status of the system runs.
submit: Submit system runs to job management system.
sync_prominence: Sync data back from prominence.
version: Print the version and exit.
yolo: Live on the edge, run duqtools go --force --yes --quiet.

duqtools clean

Delete generated IDS data and the run dir.

Usage:

duqtools clean [OPTIONS]

Options:

Name	Type	Description	Default
`--out`	boolean	Remove output data.	`False`
`--force`	boolean	Overwrite backup file if necessary.	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools create

Read duqtools.yaml and create the new IMAS data files from template data.

Usage:

duqtools create [OPTIONS]

Options:

Name	Type	Description	Default
`--force`	boolean	Overwrite existing run directories and IDS data.	`False`
`--no-sampling`	boolean	Create base run (ignores `dimensions`/`sampler`).	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools dash

Open dashboard for evaluating IDS data.

Usage:

duqtools dash [OPTIONS]

Options:

Name	Type	Description	Default
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools go

Run create, submit, status, dash in succession.

Useful for existing tested and working pipelines.

Usage:

duqtools go [OPTIONS]

Options:

Name	Type	Description	Default
`--force`	boolean	Overwrite files when necessary.	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools init

Create a default config file.

Usage:

duqtools init [OPTIONS]

Options:

Name	Type	Description	Default
`-o`, `--out`	text	Path to write config to (default=duqtools.yaml).	`duqtools.yaml`
`--force`	boolean	Overwrite existing config.	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools list-variables

List available variables.

This also picks up variables from duqtools.yaml if it exists in the local directory.

Usage:

duqtools list-variables [OPTIONS]

Options:

Name	Type	Description	Default
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--help`	boolean	Show this message and exit.	`False`

duqtools merge

Merge data sets with error propagation.

Example Merging two IDSes. Run number 8000 is the template. Only The t_e variable is merged.

The resulting IDS is saved to your own test database with shot number 36982 and run number 9999.

Example:

duqtools merge \
    -t g2jcitri/aug/36982/8000 \
    -o test/36982/9999 \
    -h g2jcitri/aug/36982/8001 \
    -h g2jcitri/aug/36982/8000 \
    -v t_e

Note:

The -t, -o and -h options expect an IMAS path formatted as user/db/shot/number or db/shot/number.

By default, duqduq merge attempts to merge all known variables. Use --variable to select which variables to merge.

Usage:

duqtools merge [OPTIONS]

Options:

Name	Type	Description	Default
`-h`, `--handle`	text	IMAS data handles.	None
`-v`, `--variable`	text	Name of the variables.	None
`-t`, `--template`	text	IMAS location to use as the template for the target	_required
`-o`, `--out`	text	IMAS location to store the result in	_required
`-i`, `--input`	text	Input file, i.e. `data.csv` or `runs.yaml`	None
`--force`	boolean	Overwrite existing output dataset.	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools plot

Generate plots for IMAS data.

Examples:

duqtools plot -v t_i_ave -v zeff -i data.csv
duqtools plot -v t_i_ave -v zeff -h user/jet/91234/5
duqtools plot -v zeff -h db/91234/5 -h db/91234/6 -h db/91234/7
duqtools plot -v zeff -h db/91234/5 -i data.csv
duqtools plot -v zeff -h db/91234/5 -o json

Usage:

duqtools plot [OPTIONS]

Options:

Name	Type	Description	Default
`-h`, `--handle`	text	IMAS data handles.	None
`-v`, `--variable`	text	Name of the variables.	None
`-o`, `--format`	text	Output format (json, html, png, svg, pdf), default: html.	`('html',)`
`-e`, `--errorbars`	boolean	Plot the errorbars (if present)	`False`
`-i`, `--input`	text	Input file, i.e. `data.csv` or `runs.yaml`	None
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools recreate

Read runs.yaml and re-create the given runs.

Example:

duqtools recreate run_0003 run_0004 --force

Usage:

duqtools recreate [OPTIONS] [RUNS]...

Options:

Name	Type	Description	Default
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools setup

Template substitution for duqtools config.

Usage:

duqtools setup [OPTIONS]

Options:

Name	Type	Description	Default
`-r`, `--run_name`	text	Name of the run.	`duqtools`
`-t`, `--template`	path	Template duqtools.yaml	`duqtools.template.yaml`
`-h`, `--handle`	text	IMAS data handle.	None
`--force`	boolean	Overwrite existing config	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools status

Print the status of the system runs.

Usage:

duqtools status [OPTIONS]

Options:

Name	Type	Description	Default
`--detailed`	boolean	Detailed info on progress	`False`
`--progress`	boolean	Fancy progress bar	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools submit

Submit system runs to job management system.

This subcommand will read runs.yaml, and start all runs which are not yet running. By default, It will not re-submit running or finished jobs.

There is a scheduler that will continue to submit jobs until the specified maximum number of jobs is running. Once a job has completed, a new job will be submitted from the queue to fill the spot.

Usage:

duqtools submit [OPTIONS]

Options:

Name	Type	Description	Default
`--force`	boolean	Re-submit running or completed jobs.	`False`
`--schedule`	boolean	Schedule and submit jobs automatically.	`False`
`-j`, `--max_jobs`	integer	Maximum number of jobs running simultaneously.	`10`
`--max_array_size`	integer	Maximum array size for slurm (usually 1001, default = 100).	`100`
`-a`, `--array`	boolean	Submit jobs as array.	`False`
`--array-script`	boolean	Create script to submit jobs as array. Like --array, but does not submit.	`False`
`--limit`	integer	Limits total number of jobs to submit.	None
`-r`, `--resubmit`	Path	Case to re-submit, can be specified multiple times	`()`
`-s`, `--status`	text	Only submit jobs with this status.	None
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools sync_prominence

Sync data back from prominence.

This subcommand is meant to be used in combination with the prominence submission system. After the jobs are completed on prominence this command can be used to retrieve and extract the completed runs from prominence to the local machine, so that they can be used in further analysis.

Usage:

duqtools sync_prominence [OPTIONS]

Options:

Name	Type	Description	Default
`--force`	boolean	Overwrite data if necessary	`False`
`--yes`	boolean	Answer yes to questions automatically.	`False`
`--dry-run`	boolean	Execute without any side-effects.	`False`
`-q`, `--quiet`	boolean	Don't output anything to the screen (except mandatory prompts).	`False`
`-c`, `--config`	text	Path to config.	`duqtools.yaml`
`--debug`	boolean	Enable debug print statements.	`False`
`--logfile`, `-l`	text	where to send the logfile, the special values stderr/stdout will send it there respectively.	`duqtools.log`
`--help`	boolean	Show this message and exit.	`False`

duqtools version

Print the version and exit.

Usage:

duqtools version [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

duqtools yolo

Live on the edge, run duqtools go --force --yes --quiet.

Usage:

duqtools yolo [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`