theta.rst 2.86 KB
Newer Older
Marco Govoni's avatar
Marco Govoni committed
1
2
3
4
5
6
.. _theta:

=================
Theta-ALCF (XC40)
=================

7
Theta is a Cray XC40 located at Argonne National Laboratory, maintained by `ALCF <https://www.alcf.anl.gov/>`_. 
Marco Govoni's avatar
Marco Govoni committed
8
9
10

.. code-block:: bash 

Marco Govoni's avatar
Marco Govoni committed
11
   $ ssh -Y <username>@theta.alcf.anl.gov
Marco Govoni's avatar
Marco Govoni committed
12
13
14
15
16
17
18
19

Building WEST
~~~~~~~~~~~~~

WEST executables can be compiled using the following script: 

.. code-block:: bash 

Marco Govoni's avatar
Marco Govoni committed
20
   $ cat build_west.sh
Marco Govoni's avatar
Marco Govoni committed
21
22
   #!/bin/bash

Marco Govoni's avatar
Marco Govoni committed
23
24
25
26
27
   module unload cray-libsci
   module load cray-python/3.6.5.3
   export CRAYPE_LINK_TYPE=dynamic
   export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
   export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
Marco Govoni's avatar
Marco Govoni committed
28
29
   export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH

Marco Govoni's avatar
Marco Govoni committed
30
   ./configure MPIF90=ftn CC=cc CXX=CC --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel SCALAPACK_LIBS="${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.so ${MKLROOT}/lib/intel64/libmkl_intel_thread.so ${MKLROOT}/lib/intel64/libmkl_core.so ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group" FFLAGS=" -xMIC-AVX512 -qopenmp -align array64byte -fp-model fast=2 -no-prec-div -assume byterecl" --with-hdf5=no CFLAGS=" -xMIC-AVX512" LDFLAGS=" -shared-intel -qopenmp" LD_LIBS="`python3-config --ldflags`"
Marco Govoni's avatar
Marco Govoni committed
31
32
33
34

   make pw -j 16

   cd West
Marco Govoni's avatar
Marco Govoni committed
35
36
   make conf PYT=python3
   make all 
Marco Govoni's avatar
Marco Govoni committed
37

Marco Govoni's avatar
Marco Govoni committed
38
39
40
41
42
43
To use the script do: 

.. code-block:: bash 

   $ bash build_west.sh

Marco Govoni's avatar
Marco Govoni committed
44
45
46
Running WEST Jobs
~~~~~~~~~~~~~~~~~

47
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Theta with 64 MPI ranks per node. The <project_name> must be replaced with an active project allocation.
Marco Govoni's avatar
Marco Govoni committed
48
49
50
51
52

.. code-block:: bash 

   $ cat run_west.sh
   #!/bin/bash
Marco Govoni's avatar
Marco Govoni committed
53
54
55
56
57
58
59
60
61
62
   #COBALT -n 2
   #COBALT -t 00:20:00
   #COBALT -q debug-cache-quad
   #COBALT -A <project_name>
   #COBALT -O WEST

   MPIRANKS_PERNODE=64
   MPIRANKS=$((COBALT_PARTSIZE * MPIRANKS_PERNODE))
   NTHREADS=1
   HT=1
Marco Govoni's avatar
Marco Govoni committed
63
64
65
66
67
68

   module unload cray-libsci
   module load cray-python/3.6.5.3
   export CRAYPE_LINK_TYPE=dynamic
   export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
   export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
Marco Govoni's avatar
Marco Govoni committed
69
70
71
   export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH

   echo "Running Cobalt Job $COBALT_JOBID."
Marco Govoni's avatar
Marco Govoni committed
72

Marco Govoni's avatar
Marco Govoni committed
73
74
   export OMP_NUM_THREADS=$NTHREADS
   aprun -n $MPIRANKS -N $MPIRANKS_PERNODE -cc depth -d $NTHREADS -j $HT ./wstat.x -i wstat.in &> wstat.out
Marco Govoni's avatar
Marco Govoni committed
75

Marco Govoni's avatar
Marco Govoni committed
76
77
78
79
80
81
Make the script executable: 

.. code-block:: bash 

   $ chmod +x run_west.sh

Marco Govoni's avatar
Marco Govoni committed
82
83
84
85
86
87
88
89
Job submission is done with the following: 

.. code-block:: bash 

   $ qsub run_west.sh

.. seealso::
   For more information, visit the ALCF user guide (`https://www.alcf.anl.gov/user-guides/xc40-system-overview <https://www.alcf.anl.gov/user-guides/xc40-system-overview/>`_).