In [1]:

import atexit
import os
import shutil
import stat

import docker

import atexit import os import shutil import stat import docker

Running duqtools with docker containers¶

This Tutorial uses docker to create and submit runs which will be analyzed with duqtools.

Getting started¶

Needed Packages¶

• The first thing we need is to install duqtools, if you haven't done this yet you can install it via pip install duqtools
• We will also need access to the source of duqtools, it can be downloaded from https://github.com/duqtools/duqtools, This notebook was created with version 1.5.0 of duqtools
• docker
  • We also need access to the following images (or equivalent): jintrac-imas. which can be found here https://gitlab.eufus.psnc.pl/containerization/jintrac/jintrac-imas-installer
• etc...

Some example data¶

In this tutorial we will use example data, as IMAS is non-free you will have to get your IMAS data through your preferred supplier. We assume that the system is run with JINTRAC (since JINTRAC has a docker container available), and we use a simple JINTRAC template, which you are of course free to substitute for your own

In [2]:

shutil.rmtree('./example', ignore_errors=True)
os.mkdir('example')
# We assume that the duqtools source is available and that we are in the docs/examples folder
shutil.copytree('../../tests/test_data/template_model', 'example/template');

shutil.rmtree('./example', ignore_errors=True) os.mkdir('example') # We assume that the duqtools source is available and that we are in the docs/examples folder shutil.copytree('../../tests/test_data/template_model', 'example/template');

In [3]:

# Some imas data, substitute with your own, as we are not allowed to provide it
shutil.copytree('../../../containerized_runs/imasdb', 'example/imasdb');

# Some imas data, substitute with your own, as we are not allowed to provide it shutil.copytree('../../../containerized_runs/imasdb', 'example/imasdb');

In [4]:

# A silly script to support venvs
with open("example/entry.sh", "w") as f:
    f.write("""#!/bin/bash
set -e
. /etc/profile.d/modules.sh
module load IMAS
module load fc2k

if [ -d .venv ]; then
 . .venv/bin/activate
fi

exec "${@}"
""")

st = os.stat('example/entry.sh')
os.chmod('example/entry.sh', st.st_mode | stat.S_IEXEC)
!chmod +x example/entry.sh

# A silly script to support venvs with open("example/entry.sh", "w") as f: f.write("""#!/bin/bash set -e . /etc/profile.d/modules.sh module load IMAS module load fc2k if [ -d .venv ]; then . .venv/bin/activate fi exec "${@}" """) st = os.stat('example/entry.sh') os.chmod('example/entry.sh', st.st_mode | stat.S_IEXEC) !chmod +x example/entry.sh

In [5]:

# A simple configuration file for duqtools
with open("example/duqtools.yaml", "w") as f:
    f.write("""system:
  name: jetto
create:
  template: ./template #TODO edit to directory where initial run is stored
  template_data:
    user: "/example/imasdb"
    db: "jet"
    shot: 123
    run: 1
  dimensions:
  - operator: multiply
    scale_to_error: false
    values: [1.1, 1.2, 1.3]
    variable: t_e
  - operator: multiply
    scale_to_error: false
    values: [1.1, 1.2, 1.3]
    variable: zeff
  sampler:
    method: latin-hypercube
    n_samples: 3
submit:
  submit_system: docker
  docker_image: jintrac-imas:latest
""")

# A simple configuration file for duqtools with open("example/duqtools.yaml", "w") as f: f.write("""system: name: jetto create: template: ./template #TODO edit to directory where initial run is stored template_data: user: "/example/imasdb" db: "jet" shot: 123 run: 1 dimensions: - operator: multiply scale_to_error: false values: [1.1, 1.2, 1.3] variable: t_e - operator: multiply scale_to_error: false values: [1.1, 1.2, 1.3] variable: zeff sampler: method: latin-hypercube n_samples: 3 submit: submit_system: docker docker_image: jintrac-imas:latest """)

In [6]:

!tree example #Its okay if this fails if you don't have tree installed

!tree example #Its okay if this fails if you don't have tree installed

example
├── duqtools.yaml
├── entry.sh
├── imasdb
│   └── jet
│       └── 3
│           └── 0
│               ├── ids_1230001.characteristics
│               ├── ids_1230001.datafile
│               └── ids_1230001.tree
└── template
    ├── jetto.eqrestart
    ├── jetto.ex
    ├── jetto.in
    ├── jetto.jset
    ├── jetto.sgrid
    ├── jetto.sin
    ├── jintrac_imas_config.cfg
    ├── rjettov
    └── utils_jetto

5 directories, 14 files

In [7]:

# Get the image and tag it
#!docker login gitlab.eufus.psnc.pl:5050
#!docker pull gitlab.eufus.psnc.pl:5050/containerization/jintrac/jintrac-imas-installer/jintrac-imas:latest
#!docker tag gitlab.eufus.psnc.pl:5050/containerization/jintrac/jintrac-imas-installer/jintrac-imas:latest jintrac-imas:pulled

# Or, build the image yourself
# Follow setup instructions in README.md
!cd ~/local_projects/jintrac-imas-installer && git checkout ab5a6
!cd ~/local_projects/jintrac-imas-installer && git submodule update
!cd ~/local_projects/jintrac-imas-installer && git submodule foreach --recursive git clean -xffd
!cd ~/local_projects/jintrac-imas-installer && DOCKER_BUILDKIT=1 docker build --target jintrac-imas --tag jintrac-imas:ab5a6 . 2> /dev/null

# Get the image and tag it #!docker login gitlab.eufus.psnc.pl:5050 #!docker pull gitlab.eufus.psnc.pl:5050/containerization/jintrac/jintrac-imas-installer/jintrac-imas:latest #!docker tag gitlab.eufus.psnc.pl:5050/containerization/jintrac/jintrac-imas-installer/jintrac-imas:latest jintrac-imas:pulled # Or, build the image yourself # Follow setup instructions in README.md !cd ~/local_projects/jintrac-imas-installer && git checkout ab5a6 !cd ~/local_projects/jintrac-imas-installer && git submodule update !cd ~/local_projects/jintrac-imas-installer && git submodule foreach --recursive git clean -xffd !cd ~/local_projects/jintrac-imas-installer && DOCKER_BUILDKIT=1 docker build --target jintrac-imas --tag jintrac-imas:ab5a6 . 2> /dev/null

HEAD is now at ab5a6fb [Build] Bump jintrac to v220922
Entering 'ITM_FLUSH'
Entering 'adas'
Entering 'adas-data'
Entering 'ascot'
Entering 'edge2d-eirene'
Entering 'eirene-data'
Entering 'eirene_iter'
Entering 'jetto-sanco'
Entering 'jetto-sanco/tci/transport/QLK'
Entering 'jetto-sanco/tci/transport/QLK/QuaLiKiz-matlabtools'
Entering 'jetto-sanco/tci/transport/QLK/QuaLiKiz-pythontools'
Entering 'jetto-sanco/tci/transport/QLK/lib/src/fruit'
Entering 'jetto-sanco/tci/transport/QLK/lib/src/fruitsh'
Entering 'jetto-sanco/tci/transport/QLK/tubs'
Entering 'jetto-sanco/tci/transport/QLKNN'
Entering 'jetto-sanco/tci/transport/QLKNN/lib/src/fruit'
Entering 'jetto-sanco/tci/transport/QLKNN/lib/src/fruitsh'
Entering 'jetto-sanco/tci/transport/QLKNN/tubs'
Entering 'jintrac'
Entering 'nag'
Entering 'pspline'
Entering 'pyal'

In [8]:

# Do some hotfixing
!echo -e "from jintrac-imas:ab5a6
\
run sed -i 's/mpi4py/mpi4py PyYAML/g' /home/docker/jintrac/build/python/run_python_driver
\
run chmod 777 /home/docker" | docker build -t jintrac-imas:latest -

# Do some hotfixing !echo -e "from jintrac-imas:ab5a6 \ run sed -i 's/mpi4py/mpi4py PyYAML/g' /home/docker/jintrac/build/python/run_python_driver \ run chmod 777 /home/docker" | docker build -t jintrac-imas:latest -

Sending build context to Docker daemon  2.048kB
Step 1/3 : from jintrac-imas:ab5a6
 ---> 4cd3896516f7
Step 2/3 : run sed -i 's/mpi4py/mpi4py PyYAML/g' /home/docker/jintrac/build/python/run_python_driver
 ---> Using cache
 ---> 245fba90e06d
Step 3/3 : run chmod 777 /home/docker
 ---> Using cache
 ---> d24430b985f4
Successfully built d24430b985f4
Successfully tagged jintrac-imas:latest

In [9]:

def niceprint(s):
    return print(str(s.output, 'utf-8'))


def run_inside(container, cmd, **kwargs):
    return container.exec_run(' '.join(['/example/entry.sh', cmd]), **kwargs)

def niceprint(s): return print(str(s.output, 'utf-8')) def run_inside(container, cmd, **kwargs): return container.exec_run(' '.join(['/example/entry.sh', cmd]), **kwargs)

In [10]:

# clean up existing container
try:
    container.stop(timeout=0)
except:
    pass

client = docker.from_env()
container = client.containers.run('jintrac-imas:latest',
                                  working_dir="/example",
                                  user=os.getuid(),
                                  detach=True,
                                  entrypoint="/example/entry.sh",
                                  command="tail -f /dev/null",
                                  volumes=[os.getcwd() + "/example:/example"],
                                  auto_remove=True)
atexit.register(lambda: container.stop(timeout=0));

# clean up existing container try: container.stop(timeout=0) except: pass client = docker.from_env() container = client.containers.run('jintrac-imas:latest', working_dir="/example", user=os.getuid(), detach=True, entrypoint="/example/entry.sh", command="tail -f /dev/null", volumes=[os.getcwd() + "/example:/example"], auto_remove=True) atexit.register(lambda: container.stop(timeout=0));

In [11]:

run_inside(container, 'python3 -m venv .venv');

run_inside(container, 'python3 -m venv .venv');

In [12]:

niceprint(run_inside(container, 'python3 -m pip install duqtools -q'))

niceprint(run_inside(container, 'python3 -m pip install duqtools -q'))

WARNING: You are using pip version 20.2.4; however, version 23.0.1 is available.
You should consider upgrading via the '/example/.venv/bin/python3 -m pip install --upgrade pip' command.

In [13]:

niceprint(run_inside(container, 'duqtools create --force --yes'))

niceprint(run_inside(container, 'duqtools create --force --yes'))

15:57:37 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14
15:57:37 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25
15:57:37 [WARNING] Python module 'tkinter' not found. Submodule 'tkinter_helpers' needs it @tkinter_helpers.py:16

Operations in the Queue:
========================
- Creating run : run_0000
- Creating run : run_0001
- Creating run : run_0002
Applying Operations
21

  0%|          | 0/21 [00:00<?, ?it/s]
Creating run : run_0000: 0:00<?, ?it/s]
Progress:  10%|▉         | 2/21 [00:00<00:04,  4.65it/s]
Creating run : run_0001: 0:00<00:01, 10.78it/s]
Progress:  38%|███▊      | 8/21 [00:00<00:01,  9.52it/s]
Creating run : run_0002: 00:01<00:00, 12.94it/s]
Progress:  67%|██████▋   | 14/21 [00:01<00:00, 10.92it/s]
Creating run : run_0002: 00:01<00:00, 13.87it/s]
Progress: 100%|██████████| 21/21 [00:01<00:00, 13.86it/s]

In [14]:

!cd example && duqtools submit --force --yes

!cd example && duqtools submit --force --yes

16:57:40 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14
16:57:40 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25

Operations in the Queue:
========================
- Submitting : Job('/home/vikko/local_projects/duqtools/docs/examples/example/run_0000')
- Submitting : Job('/home/vikko/local_projects/duqtools/docs/examples/example/run_0001')
- Submitting : Job('/home/vikko/local_projects/duqtools/docs/examples/example/run_0002')
Applying Operations
3

  0%|                                                     | 0/3 [00:00<?, ?it/s]
Submitting : Job('/home/vikko/local_projects/duqtools/docs/examples/example/run_0000'): 
Submitting : Job('/home/vikko/local_projects/duqtools/docs/examples/example/run_0001'): 
Submitting : Job('/home/vikko/local_projects/duqtools/docs/examples/example/run_0002'): 
Submitting : Job('/home/vikko/local_projects/duqtools/docs/examples/example/run_0002'): 
Progress: 100%|███████████████████████████████████| 3/3 [00:01<00:00,  2.13it/s]

In [15]:

!cd example && duqtools status

!cd example && duqtools status

16:57:43 [WARNING] Python module 'omas' not found. Submodule 'jams' needs it @jams.py:14
16:57:43 [WARNING] Python module 'netCDF4' not found. Submodule 'transp' needs it @transp.py:25
Total number of directories with submit script     : 3
Total number of directories with unsubmitted jobs  : 0
Total number of directories with status script     : 3
Total number of directories with completed status  : 0
Total number of directories with failed status     : 0
Total number of directories with running status    : 3
Total number of directories with unknown status    : 0

In [16]:

execresults = run_inside(container, 'duqtools dash', stream=True)

execresults = run_inside(container, 'duqtools dash', stream=True)

In [ ]:

for x in execresults.output:
    print(x.decode())

for x in execresults.output: print(x.decode())

  You can now view your Streamlit app in your browser.

  Network URL: http://172.17.0.4:8501
  External URL: http://192.87.183.233:8501