Duqtools demo: single UQ run
This notebook shows how to use duqtools to do validation for a single run.
It will go over the steps required to do uncertainty quantification from a template data set.
We define 2 directories:
- duqtools directory, this is where the duqtools and UQ config resides. This is also the directory we work in with duqtools.
- run directory, this is a directory where slurm has access and where all the simulation files and data are stored.
from pathlib import Path
import os
duqtools_dir = Path('/afs/eufus.eu/user/g/g2ssmee/duqtools_demo')
run_dir = Path('/afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqtools_demo')
assert not duqtools_dir.exists()
duqtools_dir.mkdir()
os.chdir(duqtools_dir)
duqtools help
The main interface for duqtools is via the CLI. You can run duqtools --help to give a list of available subcommands.
!duqtools --help
Each of the subcommands has its own help section.
!duqtools create --help
duqtools init
This is a tool to give you a generic duqtools config file that can be used as a starting point.
!duqtools init --yes
duqtools init will write duqtools.yaml which can be edited with your favourite text editor.
Check out the documentation for an overview of what can be configured.
!gedit duqtools.yaml
Setup create step
Update data in duqtools.yaml:
create:
runs_dir: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqtools_demo
template: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/interpretive_esco02
template_data:
user: g2aho
db: jet
shot: 90350
run: 2
sampler:
method: latin-hypercube
n_samples: 3
dimensions:
- variable: zeff
operator: multiply
values: [0.8, 0.9, 1.0, 1.1, 1.2]
- variable: t_e
operator: multiply
values: [0.8, 0.9, 1.0, 1.1, 1.2]
Prepare a new UQ run using duqtools create
The duqtools config (duqtools.yaml) together with the template run and template data can be used as a starting point to create a new UQ run. The example above will sample 3 runs from a matrix of 25 as the basis for the UQ.
Running duqtools create will create 3 new runs and modify the data according to the specification above.
!duqtools create --yes
Submit to slurm using duqtools submit
Use duqtools submit to submit the jobs to slurm. This tool will find all jobs (.llcmd files in the subdirectories) and submit them to slurm.
Use the --array option to submit the jobs as a slurm array.
!duqtools submit --array --yes
duqtools status
Query the status using duqtools status. This essentially parses all the jetto.status files in the run directory.
!squeue -u g2ssmee
!duqtools status
Overview of run directory
Duqtools writes the data in the directory specified by run_dir in duqtools.yaml.
duqtools.yamlis a copy of the duqtools configlogscontains the logs from the slurm arrayruns.yamlcontains the metadata for each of the runs so that they can be reproducedrun_000{1..3}are the jetto run directories. There are 3 directories corresponding ton_samples: 3in the duqtools config.
os.chdir(run_dir)
!tree -L 1
A jetto run directory.
os.chdir(run_dir / 'run_0000')
!tree -L 1
The data are stored in the local imasdb (support for JINTRAC v220922+).
!tree imasdb
Data visualization with duqtools plot
Duqtools contains a tool to create plots for your data from the command line.
os.chdir(duqtools_dir)
!duqtools plot -i data.csv -v t_i_ave -v t_e -v zeff
Data exploration with duqtools dash
This is a tool for data exploration and visualization.
Use this data.csv:
/afs/eufus.eu/user/g/g2ssmee/jetto_runs/workspace_demo/data.csv
!duqtools dash
Canonical UQ with duqtools setup
Duqtools setup is a tool that can help with canonical UQ.
It takes a templated called duqtools.template.yaml which contains the canonical settings. duqtools setup uses this file to generate a duqtools.yaml file, which is compatible with duqtools create (see steps above).
You can specify the input imas data on the command line. Duqtools will fill some machine specific from the IDS directly. The rules are specified in variables_ids2jetto.yaml. See the documentation for more information about variable specification.
import os
os.chdir('/afs/eufus.eu/user/g/g2ssmee/duqtools_template')
Below is what the template looks like.
The template uses jinja2 as a templating language.
Accolades ({..}) denote placeholders that will be replaced by the templating jinja2. Three objects are currently available (run to denote the run name, handle to reference the imas data, and variables to reference variables from the IDS data. See the documentation for more information about these placeholders.
The template supports functions to modify the placeholders or do some simple logic. The example below does some rounding of the variables using the round() function. The end time is calculated by taking the start time and adding 0.01 s.
%cat duqtools.template.yaml
Using duqtools setup replaces the handle attributes with the imas location given on the CLI, and the run.name with the name of the run.
!duqtools setup --handle g2aho/jet/90350/2 --run_name my_run --yes
Below is what the resulting duqtools.yaml looks like. It can be modified or directly used as input for duqtools create.
%cat my_run/duqtools.yaml