Duqduq demo: large scale validation¶
This notebook shows how to use duqtools to large scale validation.
It will go over the steps required to do uncertainty quantification from a sequence of data sets.
Where duqtools does UQ for a single data set, duqduq loops over multiple datasets to do UQ in sequence.
We define 2 directories:
- duqduq directory, this is where the duqtools and UQ config resides. This is also the directory we work in with duqduq.
- run directory, this is a directory where slurm has access and where all the simulation files and data are stored.
In [1]:
Copied!
from pathlib import Path
duqtools_dir = Path('/afs/eufus.eu/user/g/g2ssmee/duqduq_demo')
duqtools_dir_done = Path('/afs/eufus.eu/user/g/g2ssmee/duqduq_demo_done')
run_dir = Path('/afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long')
import os
os.chdir(duqtools_dir)
from pathlib import Path
duqtools_dir = Path('/afs/eufus.eu/user/g/g2ssmee/duqduq_demo')
duqtools_dir_done = Path('/afs/eufus.eu/user/g/g2ssmee/duqduq_demo_done')
run_dir = Path('/afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long')
import os
os.chdir(duqtools_dir)
duqduq help¶
The main interface for duqduq is via the CLI. You can run duqduq --help to give a list of available subcommands.
You will notice that the subcommands here mimic what is available in duqtools.
In [2]:
Copied!
!duqduq --help
!duqduq --help
Usage: duqduq [OPTIONS] COMMAND [ARGS]... For more information, check out the documentation: https://duqtools.readthedocs.io/large_scale_validation Options: --help Show this message and exit. Commands: create Create data sets for large scale validation. merge Merge data sets with error propagation. setup Set up large scale validation. status Check status large scale validation runs. submit Submit large scale validation runs.
duqduq setup¶
The starting point for duqduq is 2 files:
duqtools.template.yaml, this is the template config thatduqduq setupwill use to generate theduqtools.yamldata.csv, each entry in this csv file corresponds to an IMAS data set
Below is an example data.csv file. This is how you tell duqduq which data to do UQ for.
In [3]:
Copied!
%cat data.csv
%cat data.csv
,user,db,shot,run data_01,g2aho,aug,36982,0002 data_02,g2aho,jet,75225,0002 data_03,g2aho,jet,90350,0002 data_04,g2aho,jet,92432,0002 data_05,g2aho,jet,94875,0001 data_06,g2aho,tcv,64958,0002 data_07,g2aho,west,54568,0001 data_08,g2aho,west,54728,0001 data_09,g2aho,west,55181,0001 data_10,g2aho,west,55525,0001
Below is an example duqtools.template.yaml.
The index of each entry in data.csv file will be used as the run name (run.name).
The details for each entry in data.csv will be written to the template_data section.
Machine/dataset specific parameters, such as major radius or the start time are grabbed from the IDS.
For more information, see the documentation for large scale validation.
In [4]:
Copied!
%cat duqtools.template.yaml
%cat duqtools.template.yaml
tag: {{ run.name }}
create:
runs_dir: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/{{ run.name }}
template: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/interpretive_esco02
template_data:
user: {{ handle.user }}
db: {{ handle.db }}
shot: {{ handle.shot }}
run: {{ handle.run }}
operations:
- variable: major_radius
operator: copyto
{# Convert units from IDS (m) to Jetto.jset (cm) -#}
value: {{ (variables.major_radius * 100) | round(4) }}
- variable: b_field
operator: copyto
value: {{ variables.b_field | round(4) }}
- variable: t_start
operator: copyto
value: {{ variables.t_start | round(4) }}
- variable: t_end
operator: copyto
value: {{ (variables.t_start + 1.0) | round(4) }}
sampler:
method: latin-hypercube
n_samples: 9
dimensions:
- variable: zeff
operator: multiply
values: [0.8, 1.0, 1.2]
- variable: t_e
operator: multiply
values: [0.8, 1.0, 1.2]
system:
name: jetto-v220922
Running duqduq setup will generate a new directory for each dataset in data.csv. Each directory is in itself a valid duqtools directory.
In [5]:
Copied!
!duqduq setup --yes --force
!duqduq setup --yes --force
Operations in the Queue: ======================== - Setup run : data_01 - Setup run : data_02 - Setup run : data_03 - Setup run : data_04 - Setup run : data_05 - Setup run : data_06 - Setup run : data_07 - Setup run : data_08 - Setup run : data_09 - Setup run : data_10 Applying Operations 10 0%| | 0/10 [00:00<?, ?it/s] Setup run : data_10: | 0/10 [00:00<?, ?it/s] Progress: 100%|████████████████████████████████| 10/10 [00:00<00:00, 618.00it/s]
This is what the directory looks like after setup.
In [6]:
Copied!
!tree .
!tree .
. ├── data_01 │ └── duqtools.yaml ├── data_02 │ └── duqtools.yaml ├── data_03 │ └── duqtools.yaml ├── data_04 │ └── duqtools.yaml ├── data_05 │ └── duqtools.yaml ├── data_06 │ └── duqtools.yaml ├── data_07 │ └── duqtools.yaml ├── data_08 │ └── duqtools.yaml ├── data_09 │ └── duqtools.yaml ├── data_10 │ └── duqtools.yaml ├── data.csv ├── duqtools.log └── duqtools.template.yaml 10 directories, 13 files
It creates a duqtools config in each of the subdirectories. At this stage you could modify each of the duqtools.yaml if you wish. The config is no different than for a single UQ run. This means you could docd data_01 and treat it as a single UQ run.
In [7]:
Copied!
%cat data_01/duqtools.yaml
%cat data_01/duqtools.yaml
tag: data_01
create:
runs_dir: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_01
template: /afs/eufus.eu/user/g/g2ssmee/jetto_runs/interpretive_esco02
template_data:
user: g2aho
db: aug
shot: 36982
run: 2
sampler:
method: latin-hypercube
n_samples: 9
dimensions:
- variable: zeff
operator: multiply
values: [0.8, 1.0, 1.2]
- variable: t_e
operator: multiply
values: [0.8, 1.0, 1.2]
- variable: major_radius
operator: copyto
values: [ 165.0 ]
- variable: b_field
operator: copyto
values: [ -2.5725 ]
- variable: t_start
operator: copyto
values: [ 2.875 ]
- variable: t_end
operator: copyto
values: [ 3.875 ]
system:
name: jetto-v220922
Create runs using duqduq create¶
This is the equivalent of duqtools create, but for a large number of runs.
It will take each of the duqtools configs generated and set up the jetto runs and imas data according to the specification.
Since this will take a long time, we will use the --dry_run option.
In [8]:
Copied!
!duqduq create --force --dry-run
!duqduq create --force --dry-run
15:07:19 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14 15:07:19 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25 Operations in the Queue: ======================== - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_06/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0008 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0000 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0001 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0002 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0003 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0004 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0005 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0006 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0007 - Creating run : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0008 Dry run enabled, not applying op_queue
Submit to slurm using duqduq submit¶
Use duqduq submit to submit the jobs to slurm. This tool will find all jobs (.llcmd files in the subdirectories) and submit them to slurm.
Use the --array option to submit the jobs as a slurm array.
In [9]:
Copied!
os.chdir(duqtools_dir_done)
!duqduq submit --array --max_jobs 10 --force --dry-run
os.chdir(duqtools_dir_done)
!duqduq submit --array --max_jobs 10 --force --dry-run
15:07:27 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14 15:07:27 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25 Operations in the Queue: ======================== - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0008') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0000') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0001') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0002') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0003') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0004') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0005') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0006') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0007') - Adding to array : Job('/gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0008') - Submit single array job : duqtools_slurm_array.sh Dry run enabled, not applying op_queue
duqduq status¶
Query the status using duqduq status. This essentially parses all the jetto.status files in the run directory.
In [10]:
Copied!
!duqduq status
!duqduq status
15:07:39 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14 15:07:39 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25 Status codes: _ : no status, . : completed, f : failed, r : running, s : submitted, u : unknown data_01 (data_01): ......... data_02 (data_02): ......... data_03 (data_03): ......... data_04 (data_04): ...f..f.f data_05 (data_05): ff....f.. data_07 (data_07): ......... data_08 (data_08): .f..f..f. data_09 (data_09): ......... data_10 (data_10): ......... Total number of directories with submit script : 81 Total number of directories with unsubmitted jobs : 0 Total number of directories with status script : 81 Total number of directories with completed status : 72 Total number of directories with failed status : 9 Total number of directories with running status : 0 Total number of directories with unknown status : 0 Operations in the Queue: ======================== No actions to execute.
Overview of LSV output directory¶
The output of duqduq differs from a single run in that there is an additional directory layer with the name of the data entry. The logs directory contains the slurm logs.
In [11]:
Copied!
os.chdir(run_dir)
!tree -L 1
os.chdir(run_dir)
!tree -L 1
. ├── data_01 ├── data_02 ├── data_03 ├── data_04 ├── data_05 ├── data_07 ├── data_08 ├── data_09 ├── data_10 └── logs 10 directories, 0 files
Each directory is a run directory as you know it from a single UQ run.
In [12]:
Copied!
!tree 'data_01' -L 1
!tree 'data_01' -L 1
data_01 ├── duqtools.yaml ├── imasdb ├── run_0000 ├── run_0001 ├── run_0002 ├── run_0003 ├── run_0004 ├── run_0005 ├── run_0006 ├── run_0007 ├── run_0008 └── runs.yaml 10 directories, 2 files
Merge data using duqduq merge.¶
In [13]:
Copied!
os.chdir(duqtools_dir_done)
!duqduq merge --force --dry-run
os.chdir(duqtools_dir_done)
!duqduq merge --force --dry-run
Operations in the Queue: ======================== - Merging all known variables - data_01 : Merging 9 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_01/run_0000/imasdb/aug/36982/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_01/imasdb/aug/36982/2 - data_02 : Merging 9 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_02/run_0000/imasdb/jet/75225/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_02/imasdb/jet/75225/2 - data_03 : Merging 9 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_03/run_0000/imasdb/jet/90350/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_03/imasdb/jet/90350/2 - data_04 : Merging 6 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_04/run_0000/imasdb/jet/92432/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_04/imasdb/jet/92432/2 - data_05 : Merging 6 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_05/run_0002/imasdb/jet/94875/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_05/imasdb/jet/94875/2 - data_07 : Merging 9 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_07/run_0000/imasdb/west/54568/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_07/imasdb/west/54568/2 - data_08 : Merging 6 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_08/run_0000/imasdb/west/54728/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_08/imasdb/west/54728/2 - data_09 : Merging 9 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_09/run_0000/imasdb/west/55181/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_09/imasdb/west/55181/2 - data_10 : Merging 9 datasets - Template for merge : /gss_efgw_work/work/g2ssmee/jetto/runs/duqduq_long/data_10/run_0000/imasdb/west/55525/2 - Merging to : /afs/eufus.eu/user/g/g2ssmee/jetto_runs/duqduq_long/data_10/imasdb/west/55525/2 - Writing csv : merge_data.csv Dry run enabled, not applying op_queue
Merged data are stored in in a local imasdb for each data entry in the run directory.
In [14]:
Copied!
os.chdir(run_dir)
!tree 'data_01/imasdb'
os.chdir(run_dir)
!tree 'data_01/imasdb'
data_01/imasdb
└── aug
└── 3
└── 0
├── ids_369820002.characteristics
├── ids_369820002.datafile
└── ids_369820002.tree
3 directories, 3 files
Data exploration with duqtools dash¶
The imas handles for each merged data set are stored in merge_data.csv. They can be visualized using the duqtools dashboard.
In [15]:
Copied!
os.chdir(duqtools_dir_done)
!duqtools dash
os.chdir(duqtools_dir_done)
!duqtools dash
You can now view your Streamlit app in your browser. Local URL: http://localhost:8501 Network URL: http://130.186.25.54:8501 ^C Stopping...
In [ ]:
Copied!