theta.rst 2 KB
Newer Older
Marco Govoni's avatar
Marco Govoni committed
1
2
3
4
5
6
.. _theta:

=================
Theta-ALCF (XC40)
=================

7
Theta is a Cray XC40 located at Argonne National Laboratory, maintained by `ALCF <https://www.alcf.anl.gov/>`_. 
Marco Govoni's avatar
Marco Govoni committed
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

.. code-block:: bash 

   $ ssh -XY <username>@theta.alcf.anl.gov

Building WEST
~~~~~~~~~~~~~

WEST executables can be compiled using the following script: 

.. code-block:: bash 

   $ cat build_theta.sh
   #!/bin/bash

   module load miniconda-3.6/conda-4.5.12

   export BLAS_LIBS="-L$MKLROOT/intel64/lib -Wl,--start-group -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -Wl,--end-group"
   export SCALAPACK_LIBS="-L$MKLROOT/intel64/lib -Wl,--start-group -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -Wl,--end-group"
   export FFT_LIBS=""
   export MPIF90="ftn -g -mkl -dynamic"
   export CC="cc -g -mkl -dynamic"
   export F77="ftn -g -mkl -dynamic"
   export FFLAGS="-xMIC-AVX512 -align array64byte -fp-model fast=2 -no-prec-div -assume byterecl -dynamic"
   
   export CRAYPE_LINK_TYPE=dynamic

   ./install/configure --host=x86_64-build-linux-gnu --build=x86_64-target-linux-gnu --enable-parallel --with-scalapack --enable-openmp LD_LIBS="`python3-config --ldflags`"

   make pw -j 16

   cd West
   make

Running WEST Jobs
~~~~~~~~~~~~~~~~~

45
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Theta with 64 MPI ranks per node. The <project_name> must be replaced with an active project allocation.
Marco Govoni's avatar
Marco Govoni committed
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

.. code-block:: bash 

   $ cat run_west.sh
   #!/bin/bash
   #COBALT -n 2 -t 10 -q debug-cache-quad -A <project_name> -O WEST
   module load miniconda-3.6/conda-4.5.12 
   aprun -n 128 -N 64 -d 1 --cc depth -e OMP_NUM_THREADS=1 -j 1 ./wstat.x -i wstat.in > wstat.out

Job submission is done with the following: 

.. code-block:: bash 

   $ qsub run_west.sh

.. seealso::
   For more information, visit the ALCF user guide (`https://www.alcf.anl.gov/user-guides/xc40-system-overview <https://www.alcf.anl.gov/user-guides/xc40-system-overview/>`_).