Duqtools demo: single UQ run¶

This notebook shows how to use duqtools to do validation for a single run.

It will go over the steps required to do uncertainty quantification from a template data set.

We define 2 directories:

• duqtools directory, this is where the duqtools and UQ config resides. This is also the directory we work in with duqtools.
• run directory, this is a directory where slurm has access and where all the simulation files and data are stored.

In [1]:

from pathlib import Path

duqtools_dir = Path('/afs/eufus.eu/user/g/g2ssmee/duqtools_demo')
run_dir = Path('/afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqtools_demo')

import os

os.chdir(duqtools_dir)

from pathlib import Path duqtools_dir = Path('/afs/eufus.eu/user/g/g2ssmee/duqtools_demo') run_dir = Path('/afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqtools_demo') import os os.chdir(duqtools_dir)

duqtools help¶

The main interface for duqtools is via the CLI. You can run duqtools --help to give a list of available subcommands.

In [2]:

!duqtools --help

!duqtools --help

Usage: duqtools [OPTIONS] COMMAND [ARGS]...

  For more information, check out the documentation:

  https://duqtools.readthedocs.io

Options:
  --help  Show this message and exit.

Commands:
  clean           Delete generated IDS data and the run dir.
  create          Create the UQ run files.
  dash            Open dashboard for evaluating IDS data.
  go              Run create, submit, status, dash in succession.
  init            Create a default config file.
  list-variables  List available variables.
  merge           Merge data sets with error propagation.
  plot            Plot some IDS data.
  recreate        Read `runs.yaml` and re-create the given runs.
  setup           Template substitution for duqtools config.
  status          Print the status of the UQ runs.
  submit          Submit the UQ runs.
  version         Print the version and exit.
  yolo            Live on the edge, run `duqtools go --force --yes --quiet`.

Each of the subcommands has its own help section.

In [3]:

!duqtools create --help

!duqtools create --help

Usage: duqtools create [OPTIONS]

  Create the UQ run files.

Options:
  --force  Overwrite existing run directories and IDS data.
  --help   Show this message and exit.

Common options:
  --yes               Answer yes to questions automatically.
  --dry-run           Execute without any side-effects.
  -q, --quiet         Don't output anything to the screen (except mandatory
                      prompts).
  -c, --config TEXT   Path to config.
  --debug             Enable debug print statements.
  -l, --logfile TEXT  where to send the logfile, the special values
                      stderr/stdout will send it there respectively.

duqtools init¶

This is a tool to give you a generic duqtools config file that can be used as a starting point.

In [4]:

!duqtools init --yes

!duqtools init --yes

Refusing to overwrite existing CONFIG, duqtools.yaml, use --force if you really want to

duqtools init will write duqtools.yaml which can be edited with your favourite text editor.

Check out the documentation for an overview of what can be configured.

In [5]:

!gedit duqtools.yaml

!gedit duqtools.yaml

Setup create step¶

Update data in duqtools.yaml:

create:
  runs_dir: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqtools_demo
  template: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/interpretive_esco02
  template_data:
    user: g2aho
    db: jet
    shot: 90350
    run: 2
  sampler:
    method: latin-hypercube
    n_samples: 3
  dimensions:
    - variable: zeff
      operator: multiply
      values: [0.8, 0.9, 1.0, 1.1, 1.2]
    - variable: t_e
      operator: multiply
      values: [0.8, 0.9, 1.0, 1.1, 1.2]

Prepare a new UQ run using duqtools create¶

The duqtools config (duqtools.yaml) together with the template run and template data can be used as a starting point to create a new UQ run. The example above will sample 3 runs from a matrix of 25 as the basis for the UQ.

Running duqtools create will create 3 new runs and modify the data according to the specification above.

In [6]:

!duqtools create --yes

!duqtools create --yes

15:16:53 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14
15:16:53 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25

Operations in the Queue:
========================
- Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0000
- Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0001
- Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0002
Applying Operations
21

  0%|                                                    | 0/21 [00:00<?, ?it/s]
Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0000: 
Progress:  10%|███▏                              | 2/21 [00:04<00:44,  2.33s/it]
Progress:  14%|████▊                             | 3/21 [00:04<00:26,  1.46s/it]
Progress:  19%|██████▍                           | 4/21 [00:05<00:19,  1.13s/it]
Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0001: 
Progress:  38%|████████████▉                     | 8/21 [00:10<00:16,  1.28s/it]
Progress:  43%|██████████████▌                   | 9/21 [00:10<00:12,  1.02s/it]
Progress:  48%|███████████████▋                 | 10/21 [00:10<00:09,  1.20it/s]
Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0002: 
Progress:  67%|██████████████████████           | 14/21 [00:11<00:03,  1.84it/s]
Progress:  71%|███████████████████████▌         | 15/21 [00:12<00:02,  2.15it/s]
Progress:  76%|█████████████████████████▏       | 16/21 [00:12<00:02,  2.23it/s]
Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0002: 
Progress: 100%|█████████████████████████████████| 21/21 [00:12<00:00,  1.66it/s]

Submit to slurm using duqtools submit¶

Use duqtools submit to submit the jobs to slurm. This tool will find all jobs (.llcmd files in the subdirectories) and submit them to slurm.

Use the --array option to submit the jobs as a slurm array.

In [7]:

!duqtools submit --array --yes

!duqtools submit --array --yes

15:17:10 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14
15:17:10 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25

Operations in the Queue:
========================
- Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0000')
- Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0001')
- Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqtools_demo/run_0002')
- Submit single array job : duqtools_slurm_array.sh
Applying Operations
4

  0%|                                                     | 0/4 [00:00<?, ?it/s]
Submit single array job : duqtools_slurm_array.sh:                                      
Progress: 100%|███████████████████████████████████| 4/4 [00:00<00:00, 50.03it/s]

duqtools status¶

Query the status using duqtools status. This essentially parses all the jetto.status files in the run directory.

In [8]:

!squeue -u g2ssmee
!duqtools status

!squeue -u g2ssmee !duqtools status

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          216865_0        gw duqtools  g2ssmee  R       0:04      1 r054c05s04
          216865_1        gw duqtools  g2ssmee  R       0:04      1 r054c05s04
          216865_2        gw duqtools  g2ssmee  R       0:04      1 r182c14s04
15:17:16 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14
15:17:16 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25
Total number of directories with submit script     : 3
Total number of directories with unsubmitted jobs  : 3
Total number of directories with status script     : 0
Total number of directories with completed status  : 0
Total number of directories with failed status     : 0
Total number of directories with running status    : 0
Total number of directories with unknown status    : 0

Overview of run directory¶

Duqtools writes the data in the directory specified by run_dir in duqtools.yaml.

• duqtools.yaml is a copy of the duqtools config
• logs contains the logs from the slurm array
• runs.yaml contains the metadata for each of the runs so that they can be reproduced
• run_000{1..3} are the jetto run directories. There are 3 directories corresponding to n_samples: 3 in the duqtools config.

In [9]:

os.chdir(run_dir)
!tree -L 1

os.chdir(run_dir) !tree -L 1

.
├── duqtools.yaml
├── logs
├── run_0000
├── run_0001
├── run_0002
└── runs.yaml

4 directories, 2 files

A jetto run directory.

In [10]:

os.chdir(run_dir / 'run_0000')
!tree -L 1

os.chdir(run_dir / 'run_0000') !tree -L 1

.
├── configs
├── data
├── duqtools.submit.lock
├── envs
├── fort.12
├── fort.13
├── fort.21
├── fort.260 -> jetto.nbip
├── fort.261 -> jetto.nbip1
├── fort.262 -> jetto.nbip2
├── fort.263 -> jetto.nbip3
├── fort.272 -> jetto.tip
├── fort.273 -> jetto.ni1p
├── fort.274 -> jetto.ni2p
├── fort.275 -> jetto.gasp
├── fort.278 -> jetto.eqt
├── fort.280 -> jetto.sadas
├── fort.29 -> jetto.restart
├── fort.300 -> jetto.ext
├── fort.364 -> jetto.beamionsource
├── fort.40 -> jetto.chksum
├── fort.400 -> pion.0
├── fort.401 -> pion.1
├── fort.402 -> pion.2
├── fort.406 -> pion.6
├── fort.407 -> pion.7
├── fort.419 -> pion.19
├── fort.420 -> pion.20
├── fort.45 -> jetto.mhddb
├── fort.451 -> pion.51
├── fort.452 -> pion.52
├── fort.453 -> pion.53
├── fort.454 -> pion.54
├── fort.455 -> pion.55
├── fort.456 -> pion.56
├── fort.470 -> pion.70
├── fort.471 -> pion.71
├── fort.472 -> pion.72
├── fort.48 -> configs/eigen
├── fort.491 -> configs/jht
├── fort.492 -> jetto.jht
├── fort.493 -> configs/jhp
├── fort.494 -> jetto.jhp
├── fort.60 -> jetto.history
├── fort.604 -> gray_central_ray_coord
├── fort.608 -> gray_beam_CS
├── fort.612 -> gray_beam_transv
├── fort.72 -> jetto.ec
├── fort.73 -> flistprm
├── fort.74 -> ntm.print
├── fort.75 -> ntm.out
├── fort.76 -> ntm.greout
├── fort.77 -> jetto.in
├── fort.78 -> jetto.sgrid
├── fort.79 -> jetto.str
├── fort.8 -> jetto.restart
├── fort.80 -> jetto.sin
├── fort.81 -> jetto.srestart
├── fort.82 -> jetto.srestart
├── fort.84 -> jetto.shistory
├── fort.87 -> jetto.shistory
├── fort.88 -> jetto.sout
├── fort.9 -> jetto.bnd
├── fort.95 -> jetto.eqfile
├── fort.96 -> jetto.stat
├── fort.97 -> jetto.eqdsk_out
├── fort.98 -> jetto.eqrestart
├── fort.99 -> jetto.cbank
├── global_configuration
├── hcd_batch
├── hcd_gui
├── hcd_gui.py
├── hcd_nogui
├── helena_bin -> /gss_efgw_work/work/g2fjc/cmg/jams/v220922/depot/helena/bin
├── imasdb
├── interface
├── ionatom -> /gss_efgw_work/work/g2fjc/jintrac/v220922/data/ionatom
├── jetto.catid
├── jetto.chksum
├── jetto.datadir
├── jetto.eqdsk_out
├── jetto.eqrestart
├── jetto.ex
├── jetto.grill -> configs/lhcd6/jetto.grill
├── jetto.in
├── jetto.jse
├── jetto.jset
├── jetto.jsp
├── jetto.jss
├── jetto.jst
├── jetto.out
├── jetto.ray -> configs/lhcd6/jetto.ray
├── jetto.restart
├── jetto.sadas
├── jetto.sgrid
├── jetto.shistory
├── jetto.sin
├── jetto.sout
├── jetto.srestart
├── jetto.ssp
├── jetto.ssp1
├── jetto.ssp2
├── jetto.sst
├── jetto.sst1
├── jetto.sst2
├── jetto.status
├── jintrac_imas_config.cfg
├── jintrac_prov
├── labels.yaml
├── linrad -> /gss_efgw_work/work/g2fjc/jintrac/v220922/data/linrad
├── lookup.json
├── mishka_bin -> /gss_efgw_work/work/g2fjc/cmg/jams/v220922/depot/mishka/bin/mishka
├── python.err
├── QLKNN-data -> /gss_efgw_work/work/g2fjc/jintrac/v220922/jetto/tci/transport/QLKNN/data
├── rjettov
├── serialisation.json
├── _template
├── tools
├── utils_jetto
└── workflow

14 directories, 106 files

The data are stored in the local imasdb (support for JINTRAC v220922+).

In [11]:

!tree imasdb

!tree imasdb

imasdb
└── jet
    └── 3
        ├── 0
        │   ├── ids_903500001.characteristics
        │   ├── ids_903500001.datafile
        │   ├── ids_903500001.tree
        │   ├── ids_903500002.characteristics
        │   ├── ids_903500002.datafile
        │   └── ids_903500002.tree
        ├── 1
        ├── 2
        ├── 3
        ├── 4
        ├── 5
        ├── 6
        ├── 7
        ├── 8
        └── 9

12 directories, 6 files

Data visualization with duqtools plot¶

Duqtools contains a tool to create plots for your data from the command line.

In [12]:

os.chdir(duqtools_dir)

!duqtools plot -i data.csv -v t_i_ave -v t_e -v zeff

os.chdir(duqtools_dir) !duqtools plot -i data.csv -v t_i_ave -v t_e -v zeff

You can now view your plot in your browser:

  rho_tor_norm vs. t_i_ave:

    file:////afs/eufus.eu/g2itmuse/user/g2ssmee/duqtools_demo/chart_rho_tor_norm-t_i_ave.html

You can now view your plot in your browser:

  rho_tor_norm vs. t_e:

    file:////afs/eufus.eu/g2itmuse/user/g2ssmee/duqtools_demo/chart_rho_tor_norm-t_e.html

You can now view your plot in your browser:

  rho_tor_norm vs. zeff:

    file:////afs/eufus.eu/g2itmuse/user/g2ssmee/duqtools_demo/chart_rho_tor_norm-zeff.html

Data exploration with duqtools dash¶

This is a tool for data exploration and visualization.

Use this data.csv:
/afs/eufus.eu/user/g/g2ssmee/jetto_runs/workspace_demo/data.csv

In [13]:

!duqtools dash

!duqtools dash

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://130.186.25.53:8501

^C
  Stopping...

Canonical UQ with duqtools setup¶

Duqtools setup is a tool that can help with canonical UQ.

It takes a templated called duqtools.template.yaml which contains the canonical settings. duqtools setup uses this file to generate a duqtools.yaml file, which is compatible with duqtools create (see steps above).

You can specify the input imas data on the command line. Duqtools will fill some machine specific from the IDS directly. The rules are specified in variables_ids2jetto.yaml. See the documentation for more information about variable specification.

In [14]:

import os

os.chdir('/afs/eufus.eu/user/g/g2ssmee/duqtools_template')

import os os.chdir('/afs/eufus.eu/user/g/g2ssmee/duqtools_template')

Below is what the template looks like.

The template uses jinja2 as a templating language.

Accolades ({..}) denote placeholders that will be replaced by the templating jinja2. Three objects are currently available (run to denote the run name, handle to reference the imas data, and variables to reference variables from the IDS data. See the documentation for more information about these placeholders.

The template supports functions to modify the placeholders or do some simple logic. The example below does some rounding of the variables using the round() function. The end time is calculated by taking the start time and adding 0.01 s.

In [15]:

%cat duqtools.template.yaml

%cat duqtools.template.yaml

tag: {{ run.name }}
create:
  runs_dir: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/{{ run.name }}
  template: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/interpretive_esco02
  template_data:
    user: {{ handle.user }}
    db: {{ handle.db }}
    shot: {{ handle.shot }}
    run: {{ handle.run }}
  operations:
    - variable: major_radius
      operator: copyto
      {# Convert units from IDS (m) to Jetto.jset (cm) -#}
      value: {{ (variables.major_radius * 100) | round(4) }}
    - variable: b_field
      operator: copyto
      value: {{ variables.b_field | round(4) }}
    - variable: t_start
      operator: copyto
      value: {{ variables.t_start | round(4) }}
    - variable: t_end
      operator: copyto
      value: {{ (variables.t_start + 0.01) | round(4) }}
  sampler:
    method: latin-hypercube
    n_samples: 25
  dimensions:
    - variable: zeff
      operator: multiply
      values: [0.8, 0.9, 1.0, 1.1, 1.2]
    - variable: t_e
      operator: multiply
      values: [0.8, 0.9, 1.0, 1.1, 1.2]
system:
  name: jetto-v220922

Using duqtools setup replaces the handle attributes with the imas location given on the CLI, and the run.name with the name of the run.

In [16]:

!duqtools setup --handle g2aho/jet/90350/2 --run_name my_run --yes

!duqtools setup --handle g2aho/jet/90350/2 --run_name my_run --yes

Operations in the Queue:
========================
- Setup run : my_run
Applying Operations
1

  0%|                                                     | 0/1 [00:00<?, ?it/s]
Setup run : my_run: 
Progress: 100%|██████████████████████████████████| 1/1 [00:00<00:00, 415.03it/s]

Below is what the resulting duqtools.yaml looks like. It can be modified or directly used as input for duqtools create.

In [17]:

%cat my_run/duqtools.yaml

%cat my_run/duqtools.yaml

tag: my_run
create:
  runs_dir: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/my_run
  template: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/interpretive_esco02
  template_data:
    user: g2aho
    db: jet
    shot: 90350
    run: 2
  sampler:
    method: latin-hypercube
    n_samples: 25
  dimensions:
    - variable: zeff
      operator: multiply
      values: [0.8, 0.9, 1.0, 1.1, 1.2]
    - variable: t_e
      operator: multiply
      values: [0.8, 0.9, 1.0, 1.1, 1.2]
    - variable: major_radius
      operator: copyto
      values: [ 296.0 ]
    - variable: b_field
      operator: copyto
      values: [ 2.1155 ]
    - variable: t_start
      operator: copyto
      values: [ 45.75 ]
    - variable: t_end
      operator: copyto
      values: [ 45.76 ]
system:
  name: jetto-v220922

In [ ]: