theta.rst 2.87 KB
Newer Older
Marco Govoni's avatar
Marco Govoni committed
1
2
3
4
5
6
.. _theta:

=================
Theta-ALCF (XC40)
=================

7
Theta is a Cray XC40 located at Argonne National Laboratory, maintained by `ALCF <https://www.alcf.anl.gov/>`_. 
Marco Govoni's avatar
Marco Govoni committed
8
9
10

.. code-block:: bash 

Marco Govoni's avatar
Marco Govoni committed
11
   $ ssh -Y <username>@theta.alcf.anl.gov
Marco Govoni's avatar
Marco Govoni committed
12
13
14
15
16
17
18
19

Building WEST
~~~~~~~~~~~~~

WEST executables can be compiled using the following script: 

.. code-block:: bash 

Marco Govoni's avatar
Marco Govoni committed
20
   $ cat build_west.sh
Marco Govoni's avatar
Marco Govoni committed
21
22
   #!/bin/bash

Marco Govoni's avatar
Marco Govoni committed
23
24
25
26
27
   module unload cray-libsci
   module load cray-python/3.6.5.3
   export CRAYPE_LINK_TYPE=dynamic
   export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
   export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
Marco Govoni's avatar
Marco Govoni committed
28
29
   export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH

Marco Govoni's avatar
Marco Govoni committed
30
   ./configure MPIF90=ftn CC=cc CXX=CC --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel SCALAPACK_LIBS="${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.so ${MKLROOT}/lib/intel64/libmkl_intel_thread.so ${MKLROOT}/lib/intel64/libmkl_core.so ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group" FFLAGS=" -xMIC-AVX512 -qopenmp -align array64byte -fp-model fast=2 -no-prec-div -assume byterecl" --with-hdf5=no CFLAGS=" -xMIC-AVX512" LDFLAGS=" -shared-intel -qopenmp" 
Marco Govoni's avatar
Marco Govoni committed
31
32
33
34

   make pw -j 16

   cd West
Marco Govoni's avatar
Marco Govoni committed
35
   make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags`"
Marco Govoni's avatar
Marco Govoni committed
36
   make all 
Marco Govoni's avatar
Marco Govoni committed
37

Marco Govoni's avatar
Marco Govoni committed
38
39
40
41
42
43
To use the script do: 

.. code-block:: bash 

   $ bash build_west.sh

Marco Govoni's avatar
Marco Govoni committed
44
45
46
Running WEST Jobs
~~~~~~~~~~~~~~~~~

47
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Theta with 64 MPI ranks per node. The <project_name> must be replaced with an active project allocation.
Marco Govoni's avatar
Marco Govoni committed
48
49
50
51
52

.. code-block:: bash 

   $ cat run_west.sh
   #!/bin/bash
Marco Govoni's avatar
Marco Govoni committed
53
54
55
56
57
58
59
60
61
62
   #COBALT -n 2
   #COBALT -t 00:20:00
   #COBALT -q debug-cache-quad
   #COBALT -A <project_name>
   #COBALT -O WEST

   MPIRANKS_PERNODE=64
   MPIRANKS=$((COBALT_PARTSIZE * MPIRANKS_PERNODE))
   NTHREADS=1
   HT=1
Marco Govoni's avatar
Marco Govoni committed
63
64
65
66
67
68

   module unload cray-libsci
   module load cray-python/3.6.5.3
   export CRAYPE_LINK_TYPE=dynamic
   export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
   export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
Marco Govoni's avatar
Marco Govoni committed
69
70
71
   export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH

   echo "Running Cobalt Job $COBALT_JOBID."
Marco Govoni's avatar
Marco Govoni committed
72

Marco Govoni's avatar
Marco Govoni committed
73
74
   export OMP_NUM_THREADS=$NTHREADS
   aprun -n $MPIRANKS -N $MPIRANKS_PERNODE -cc depth -d $NTHREADS -j $HT ./wstat.x -i wstat.in &> wstat.out
Marco Govoni's avatar
Marco Govoni committed
75

Marco Govoni's avatar
Marco Govoni committed
76
77
78
79
80
81
Make the script executable: 

.. code-block:: bash 

   $ chmod +x run_west.sh

Marco Govoni's avatar
Marco Govoni committed
82
83
84
85
86
87
88
89
Job submission is done with the following: 

.. code-block:: bash 

   $ qsub run_west.sh

.. seealso::
   For more information, visit the ALCF user guide (`https://www.alcf.anl.gov/user-guides/xc40-system-overview <https://www.alcf.anl.gov/user-guides/xc40-system-overview/>`_).