Commit ff00fc36 authored by Marco Govoni's avatar Marco Govoni

Merge branch 'develop' into 'master'

Version 4.3.0

See merge request west-devel/West!42
parents feea2ef5 e4829510
Pipeline #3265 passed with stages
in 12 minutes and 22 seconds
This diff is collapsed.
......@@ -9,19 +9,21 @@ This file lists the authors of the WEST code as well as the known contributors.
The current list of developers, in alphabetical order:
- Sijia Dong (Argonne National Laboratory)
- Marco Govoni (Argonne National Laboratory and University of Chicago)
- Lan Huang (Argonne National Laboratory)
- Yu Jin (University of Chicago)
- He Ma (University of Chicago)
- Nan Sheng (University of Chicago)
- Han Yang (University of Chicago)
- Wenzhe Yu (Argonne National Laboratory)
# Former Developers
The former list of developers, in alphabetical order:
- Nicholas Brawand (2016-2018)
- Sijia Dong (2019-2020)
- Matteo Gerosa (2017-2018)
- Lan Huang (2019-2020)
- He Ma (2017-2020)
- Ryan McAvoy (2017-2018)
- Ngoc Linh Nguyen (2017-2018)
- Peter Scherpelz (2016-2017)
......@@ -29,7 +31,7 @@ The former list of developers, in alphabetical order:
# Contributors
We would like to thank the following people for their contibution (beside generic comments and bugfixes), in alphabetical order:
We would like to thank the following people for their contributions, in alphabetical order:
- Francois Gygi (University of California, Davis)
- Ikutaro Hamada (Osaka University)
......
Change Log
==========
v4.3.0 (2021/05/26)
-------------------
- Introduced new data layout. Parallelization over bands allows to distribute data in a more flexible way in `wstat`. This feature also helps reduce memory per image. Band parallelization is enabled by specifying -nb xxx from the command line.
- Introduced checkpointing in `wstat`. With the new keyword `n_steps_write_restart` one can control how often the code produces restarts (default value is `1`).
- Improved I/O in `wfreq`. The number of I/O operations is reduced in `solve_wfreq`, and in the gamma case of `solve_gfreq`.
- Updated library dependency to Json-Fortran 8.2.1, resolving compilation issues with PGI 19.10.
- Updated the initialization of Forpy, such that in case of a module import error, the code has a better chance to print a clear error message.
- Updated build. Now the code builds with the NVIDIA/PGI Fortran compiler.
- Bug fix. Fixed an undefined variable in `Tools/set_npwq.f90`. Added `IMPLICIT NONE to all program units to let the compiler catch such errors in the future.
- Bug fix. Avoid overflow in `Wstat/wstat_memory_report.f90` in large-scale runs.
- Bug fix. Reset permissions to all source files in Westpp. All files appeared to be executable before.
- Bug fix. Reset `make.depend` files, added support to `make -j`.
- Updated CI/CD. Use updated Docker images.
- Updated CI/CD. Check numerical results and fail the CI if results don't match (see `check.py`). DFT checks error in total energy, WSTAT checks maximum error in PDEP eigenvalues, WFREQ checks maximum error in QP energies.
- Updated CI/CD. Added tests of images and OpenMP threads to nightly tests.
- Updated documentation. Fixed doc build with sphinx 3.5.0+.
- Updated documentation. Update build instructions for ALCF/Theta, NERSC/Cori, UChicago/RCC/Midway3, macOS.
- Updated documentation. Added more tutorials.
- Updated documentation. Updated manual.
v4.2.1 (2020/10/19)
-------------------
......
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......@@ -169,6 +169,8 @@ MODULE class_coulomb
USE gvect, ONLY : g, ngm
USE types_bz_grid, ONLY : q_grid
!
IMPLICIT NONE
!
! I/O
!
CLASS(coulomb) :: this
......@@ -304,6 +306,8 @@ MODULE class_coulomb
USE io_global, ONLY : stdout
USE types_bz_grid, ONLY : q_grid
!
IMPLICIT NONE
!
! I/O
!
CLASS(coulomb) :: this
......
store_sqvc.o : ../../Modules/cell_base.o
store_sqvc.o : ../../Modules/constants.o
store_sqvc.o : ../../Modules/io_global.o
store_sqvc.o : ../../Modules/kind.o
store_sqvc.o : ../../Modules/recvec.o
store_sqvc.o : ../Tools/class_bz_grid.o
store_sqvc.o : ../Tools/types_bz_grid.o
store_sqvc.o : ../../PW/src/exx.o
store_sqvc.o : ../../PW/src/pwcom.o
store_sqvc.o : ../../West/Modules/westcom.o
store_vspx.o : ../../Modules/cell_base.o
store_vspx.o : ../../Modules/constants.o
store_vspx.o : ../../Modules/kind.o
store_vspx.o : ../../Modules/recvec.o
store_vspx.o : ../../PW/src/exx.o
store_vspx.o : ../../PW/src/pwcom.o
class_coulomb.o : ../../Modules/cell_base.o
class_coulomb.o : ../../Modules/constants.o
class_coulomb.o : ../../Modules/control_flags.o
class_coulomb.o : ../../Modules/gvecw.o
class_coulomb.o : ../../Modules/io_global.o
class_coulomb.o : ../../Modules/kind.o
class_coulomb.o : ../../Modules/mp.o
class_coulomb.o : ../../Modules/mp_global.o
class_coulomb.o : ../../Modules/mp_world.o
class_coulomb.o : ../../Modules/random_numbers.o
class_coulomb.o : ../../Modules/recvec.o
class_coulomb.o : ../Modules/westcom.o
class_coulomb.o : ../Tools/types_bz_grid.o
types_coulomb.o : ../../Modules/kind.o
types_coulomb.o : class_coulomb.o
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......
......@@ -9,6 +9,7 @@ MODFLAGS= $(MOD_FLAG)../../iotk/src $(MOD_FLAG)../../Modules $(MOD_FLAG)../../LA
$(MOD_FLAG)../Tools \
$(MOD_FLAG)../FFT_kernel \
$(MOD_FLAG)../Coulomb_kernel \
$(MOD_FLAG)../Para_kernel \
$(MOD_FLAG).
IFLAGS=
......
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......@@ -54,7 +54,11 @@ SUBROUTINE apply_sternheimerop_to_m_wfcs(nbndval, psi, hpsi, e, alpha, m)
IF(l_kinetic_only) THEN
CALL k_psi( npwx, npw, m, psi, hpsi )
ELSE
CALL h_psi( npwx, npw, m, psi, hpsi )
!
! use h_psi_, i.e. h_psi without band parallelization, as wstat
! handles band parallelization separately in dfpt_module
!
CALL h_psi_( npwx, npw, m, psi, hpsi )
ENDIF
!
! then we compute the operator H-epsilon S
......
This diff is collapsed.
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......
apply_sternheimerop_to_m_wfcs.o : ../../Modules/becmod.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/control_flags.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/kind.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/mp.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/mp_global.o
......@@ -7,34 +6,37 @@ apply_sternheimerop_to_m_wfcs.o : ../../Modules/noncol.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/uspp.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/wavefunctions.o
apply_sternheimerop_to_m_wfcs.o : ../../PW/src/pwcom.o
apply_sternheimerop_to_m_wfcs.o : ../../West/Modules/westcom.o
dfpt.o : ../../Modules/constants.o
dfpt.o : ../../Modules/control_flags.o
dfpt.o : ../../Modules/fft_base.o
dfpt.o : ../../Modules/io_files.o
dfpt.o : ../../Modules/io_global.o
dfpt.o : ../../Modules/kind.o
dfpt.o : ../../Modules/mp.o
dfpt.o : ../../Modules/mp_global.o
dfpt.o : ../../Modules/noncol.o
dfpt.o : ../../Modules/recvec.o
dfpt.o : ../../Modules/uspp.o
dfpt.o : ../../Modules/wavefunctions.o
dfpt.o : ../../PW/src/buffers.o
dfpt.o : ../../PW/src/pwcom.o
dfpt.o : ../../West/FFT_kernel/fft_at_gamma.o
dfpt.o : ../../West/FFT_kernel/fft_at_k.o
dfpt.o : ../../West/Modules/westcom.o
dfpt.o : ../../West/Tools/bar.o
dfpt.o : ../../West/Tools/io_push.o
dfpt.o : ../../West/Tools/types_bz_grid.o
dfpt.o : ../../PW/src/exx.o
apply_sternheimerop_to_m_wfcs.o : ../Modules/westcom.o
dfpt_module.o : ../../Modules/cell_base.o
dfpt_module.o : ../../Modules/control_flags.o
dfpt_module.o : ../../Modules/fft_base.o
dfpt_module.o : ../../Modules/io_global.o
dfpt_module.o : ../../Modules/kind.o
dfpt_module.o : ../../Modules/mp.o
dfpt_module.o : ../../Modules/mp_global.o
dfpt_module.o : ../../Modules/mp_world.o
dfpt_module.o : ../../Modules/noncol.o
dfpt_module.o : ../../Modules/recvec.o
dfpt_module.o : ../../Modules/uspp.o
dfpt_module.o : ../../Modules/wavefunctions.o
dfpt_module.o : ../../PW/src/buffers.o
dfpt_module.o : ../../PW/src/pwcom.o
dfpt_module.o : ../FFT_kernel/fft_at_gamma.o
dfpt_module.o : ../FFT_kernel/fft_at_k.o
dfpt_module.o : ../Modules/westcom.o
dfpt_module.o : ../Para_kernel/class_idistribute.o
dfpt_module.o : ../Para_kernel/distribution_center.o
dfpt_module.o : ../Tools/bar.o
dfpt_module.o : ../Tools/io_push.o
dfpt_module.o : ../Tools/types_bz_grid.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/control_flags.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/kind.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/mp.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/mp_global.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/noncol.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/recvec.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/wavefunctions.o
linsolve_sternheimer_m_wfcts.o : ../../PW/src/pwcom.o
linsolve_sternheimer_m_wfcts.o : ../../West/Modules/westcom.o
linsolve_sternheimer_m_wfcts.o : ../Modules/westcom.o
precondition_m_wfcts.o : ../../Modules/kind.o
precondition_m_wfcts.o : ../../Modules/noncol.o
precondition_m_wfcts.o : ../../PW/src/pwcom.o
!
! Copyright (C) 2015-2017. Govoni
! Copyright (C) 2015-2021. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......
......@@ -15,10 +15,11 @@ Project Leads
Developers
----------
- Sijia Dong (Argonne National Laboratory)
- Marco Govoni (Argonne National Laboratory and University of Chicago)
- He Ma (University of Chicago)
- Yu Jin (University of Chicago)
- Nan Sheng (University of Chicago)
- Han Yang (University of Chicago)
- Wenzhe Yu (Argonne National Laboratory)
Contributors
------------
......@@ -35,7 +36,10 @@ Former Developers
-----------------
- Nicholas Brawand (2016-2018)
- Sijia Dong (2019-2020)
- Matteo Gerosa (2017-2018)
- Lan Huang (2019-2020)
- He Ma (2017-2020)
- Ryan McAvoy (2017-2018)
- Ngoc Linh Nguyen (2017-2018)
- Peter Scherpelz (2016-2018)
......
......@@ -60,7 +60,7 @@ master_doc = 'index'
# General information about the project.
project = data["name"]
copyright = u'2020, Marco Govoni'
copyright = u'2021, Marco Govoni'
author = u'Marco Govoni'
# The version info for the project you're documenting, acts as replacement for
......@@ -157,11 +157,7 @@ html_logo = "images/logo.jpg"
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_context = {
'css_files': [
'_static/theme_overrides.css', # override wide tables in RTD theme
],
}
html_ccs_files = ['theme_overrides.css'] # override wide tables in RTD theme
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
......@@ -315,6 +311,5 @@ texinfo_documents = [
#texinfo_no_detailmenu = False
intersphinx_mapping = {
'python': ('https://docs.python.org/3.6', None),
'pymongo': ('https://api.mongodb.com/python/current/', None)
'python': ('https://docs.python.org/3.6', None)
}
......@@ -4,10 +4,12 @@
Installation
============
In order to install WEST you need to download the `QuantumEspresso 6.1 <https://gitlab.com/QEF/q-e/-/archive/qe-6.1.0/q-e-qe-6.1.0.tar>`_.
In order to install WEST you need to download the `QuantumEspresso 6.1.0 <https://gitlab.com/QEF/q-e/-/archive/qe-6.1.0/q-e-qe-6.1.0.tar>`_.
To compute absorption spectra (BSE), you also need to download and install `Qbox <http://qboxcode.org>`_.
`QuantumEspresso <http://www.quantum-espresso.org/>`_ (QE) is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale, based on density-functional theory (DFT), plane waves (PW), and pseudopotentials (PP).
Configure QuantumEspresso by running the ``configure`` script that comes with the QE distribution. WEST requires `MPI <https://en.wikipedia.org/?title=Message_Passing_Interface>`_ support (`Scalapack <http://www.netlib.org/scalapack/>`_ and `OpenMP <http://openmp.org/>`_ support is also recommended, but optional). If all the environment variables (compilers, libraries etc.) have been set according to the QE configure guide, this would simply be:
Configure QuantumEspresso by running the ``configure`` script that comes with the QE distribution. WEST requires `MPI <https://en.wikipedia.org/?title=Message_Passing_Interface>`_ support (`ScaLAPACK <http://www.netlib.org/scalapack/>`_ and `OpenMP <http://openmp.org/>`_ support is also recommended, but optional). If all the environment variables (compilers, libraries etc.) have been set according to the QE configure guide, this would simply be:
.. code-block:: bash
......@@ -17,7 +19,7 @@ Configure QuantumEspresso by running the ``configure`` script that comes with th
$ ./configure
.. note::
Note that since v4.0.0 WEST requires dynamic linking and python3.
Note that since v4.0.0 WEST requires dynamic linking and Python3.
It's now time to create the ``pw.x``, ``wstat.x``, ``wfreq.x``, and ``westpp.x`` executables by doing:
......@@ -49,6 +51,7 @@ Suggested configuration options
.. toctree::
:maxdepth: 1
installations/cori.rst
installations/theta.rst
installations/midway.rst
installations/macosx.rst
installations/midway3.rst
installations/macos.rst
.. _cori:
=================
Cori-NERSC (XC40)
=================
Cori is a Cray XC40 located at National Energy Research Scientific Computing Center (`NERSC <https://www.nersc.gov/>`_).
.. code-block:: bash
$ ssh <username>@cori.nersc.gov
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script (tested on May 12, 2021):
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module unload cray-libsci
module load cray-python/3.8.2.1
export CRAYPE_LINK_TYPE=dynamic
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export MPIF90=ftn
export F77=ftn
export CC=cc
export SCALAPACK_LIBS="$MKLROOT/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.so $MKLROOT/lib/intel64/libmkl_intel_thread.so $MKLROOT/lib/intel64/libmkl_core.so $MKLROOT/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group"
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel --with-hdf5=no
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Cori (Haswell partition) with 32 MPI ranks per node. The <project_name> must be replaced with an active project allocation.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --job-name=WEST
#SBATCH --time=00:20:00
#SBATCH --account=<project_name>
#SBATCH --constraint=haswell
#SBATCH --qos=debug
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
module unload cray-libsci
module load cray-python/3.8.2.1
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export OMP_NUM_THREADS=1
export OMP_PLACE=threads
export OMP_PROC_BIND=spread
export MKL_NUM_THREADS=$OMP_NUM_THREADS
NTASKS=$(($SLURM_NTASKS_PER_NODE * $SLURM_JOB_NUM_NODES))
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS_PER_NODE -c $SLURM_CPUS_PER_TASK ./wstat.x -i wstat.in &> wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the `NERSC user guide <https://docs.nersc.gov/systems/cori/>`_.
.. _macosx:
.. _macos:
======
MacOSX
======
=====
macOS
=====
The following instructions have been tested on MacOSX 10.14.6.
The following instructions have been tested on macOS 11.3.
Requirements:
- Gcc/Gfortran 9
- gcc/gfortran (e.g. GCC 9)
- MPICH
- Blas/Lapack/Scalapack
- BLAS/LAPACK/ScaLAPACK
- FFTW3
- Python3
......@@ -40,11 +40,12 @@ WEST executables can be compiled using the following script:
export SCALAPACK_LIBS=${MY_LIB_PATH}/SCALAPACK/libscalapack.a
export FFT_LIBS="${MY_LIB_PATH}/FFTW3/lib/libfftw3.a ${MY_LIB_PATH}/FFTW3/lib/libfftw3_omp.a"
./configure --with-scalapack --enable-openmp
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack --with-hdf5=no
make -j 4 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
......
.. _midway:
===============
Midway-UChicago
===============
Midway is the HPC cluster of the University of Chicago, maintained by UChicago's `RCC <https://rcc.uchicago.edu/>`_.
.. code-block:: bash
$ ssh -Y <username>@midway.rcc.uchicago.edu
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module load intelmpi/5.1+intel-16.0 mkl/2017.up4 python/cpython-3.8.5
export F77=mpiifort
export CC=mpiicc
export MPIF90=mpiifort
export FC=mpiifort
export CFLAGS="-O3 -fno-alias -ansi-alias -g -mkl -Bdynamic"
export FFLAGS="-O3 -fno-alias -ansi-alias -g -mkl -Bdynamic"
export BLAS_LIBS_SWITCH="external"
export BLAS_LIBS=" -lmkl_intel_lp64 -lmkl_sequential -lmkl_core"
export LAPACK_LIBS_SWITCH="external"
export LAPACK_LIBS=" "
export SCALAPACK_LIBS=" -lmkl_scalapack_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group"
./configure --enable-openmp=yes --enable-parallel=yes --with-scalapack=intel --with-hdf5=no
make -j 2 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of the `broadwl` partition with 28 MPI ranks per node.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --time=00:30:00
#SBATCH --partition=broadwl
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=28
#SBATCH --cpus-per-task=1
module load intelmpi/5.1+intel-16.0 mkl/2017.up4 python/cpython-3.8.5
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
NTASKS=$(($SLURM_NTASKS_PER_NODE * $SLURM_JOB_NUM_NODES))
mpirun -np $NTASKS ./wstat.x -i wstat.in > wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the RCC user guide (`https://rcc.uchicago.edu/docs/ <https://rcc.uchicago.edu/docs/>`_).
.. _midway3:
================
Midway3-UChicago
================
Midway3 is the HPC cluster of the University of Chicago, maintained by UChicago's `RCC <https://rcc.uchicago.edu/>`_.
.. code-block:: bash
$ ssh <username>@midway3.rcc.uchicago.edu
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module load intel/19.1.1
module load intelmpi/2019.up7+intel-19.1.1
module load mkl/2020.up1
module load python/anaconda-2020.11
export MPIF90=mpiifort
export F90=ifort
export F77=ifort
export CC=icc
export SCALAPACK_LIBS="-lmkl_scalapack_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group"
./configure --enable-parallel --with-scalapack --enable-openmp
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
sed -i 's/-L.*config-3.8-x86_64-linux-gnu //' west_make.inc
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Midway3 with 32 MPI ranks per node. The <project_name> and <account_name> must be replaced with an active project allocation.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --time=00:20:00
#SBATCH --partition=<partition_name>
#SBATCH --account=<account_name>
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --cpus-per-task=1
module load intel/19.1.1
module load intelmpi/2019.up7+intel-19.1.1
module load mkl/2020.up1
module load python/anaconda-2020.11
export I_MPI_PMI_LIBRARY=/software/slurm-current-$DISTARCH/lib/libpmi.so
export LD_LIBRARY_PATH=/software/python-anaconda-2020.11-el8-x86_64/lib:$LD_LIBRARY_PATH
export OMP_NUM_THREADS=1
srun -n 96 -N 2 ./wstat.x -i wstat.in > wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the `RCC user guide <https://rcc.uchicago.edu/docs/>`_.
......@@ -8,12 +8,12 @@ Theta is a Cray XC40 located at Argonne National Laboratory, maintained by `ALCF
.. code-block:: bash
$ ssh -Y <username>@theta.alcf.anl.gov
$ ssh <username>@theta.alcf.anl.gov
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
WEST executables can be compiled using the following script (tested on May 12, 2021):
.. code-block:: bash
......@@ -21,18 +21,24 @@ WEST executables can be compiled using the following script:
#!/bin/bash
module unload cray-libsci
module load cray-python/3.6.5.3
module load cray-python/3.8.2.1
export CRAYPE_LINK_TYPE=dynamic
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export MPIF90=ftn
export F77=ftn
export CC=cc
export SCALAPACK_LIBS="$MKLROOT/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.so $MKLROOT/lib/intel64/libmkl_intel_thread.so $MKLROOT/lib/intel64/libmkl_core.so $MKLROOT/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group"
./configure MPIF90=ftn CC=cc CXX=CC --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel SCALAPACK_LIBS="${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.so ${MKLROOT}/lib/intel64/libmkl_intel_thread.so ${MKLROOT}/lib/intel64/libmkl_core.so ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group" FFLAGS=" -xMIC-AVX512 -qopenmp -align array64byte -fp-model fast=2 -no-prec-div -assume byterecl" --with-hdf5=no CFLAGS=" -xMIC-AVX512" LDFLAGS=" -shared-intel -qopenmp"
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel --with-hdf5=no
make pw -j 16
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags`"
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
......@@ -62,11 +68,11 @@ The following is an example executable script `run_west.sh` to run the `wstat.x`
HT=1
module unload cray-libsci
module load cray-python/3.6.5.3
export CRAYPE_LINK_TYPE=dynamic
module load cray-python/3.8.2.1
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
echo "Running Cobalt Job $COBALT_JOBID."
......@@ -77,7 +83,7 @@ Make the script executable:
.. code-block:: bash
$ chmod +x run_west.sh
$ chmod 755 run_west.sh
Job submission is done with the following:
......@@ -86,4 +92,4 @@ Job submission is done with the following:
$ qsub run_west.sh