Commit ff00fc36 authored by Marco Govoni's avatar Marco Govoni

Merge branch 'develop' into 'master'

Version 4.3.0

See merge request west-devel/West!42
parents feea2ef5 e4829510
Pipeline #3265 passed with stages
in 12 minutes and 22 seconds
This diff is collapsed.
......@@ -9,19 +9,21 @@ This file lists the authors of the WEST code as well as the known contributors.
The current list of developers, in alphabetical order:
- Sijia Dong (Argonne National Laboratory)
- Marco Govoni (Argonne National Laboratory and University of Chicago)
- Lan Huang (Argonne National Laboratory)
- Yu Jin (University of Chicago)
- He Ma (University of Chicago)
- Nan Sheng (University of Chicago)
- Han Yang (University of Chicago)
- Wenzhe Yu (Argonne National Laboratory)
# Former Developers
The former list of developers, in alphabetical order:
- Nicholas Brawand (2016-2018)
- Sijia Dong (2019-2020)
- Matteo Gerosa (2017-2018)
- Lan Huang (2019-2020)
- He Ma (2017-2020)
- Ryan McAvoy (2017-2018)
- Ngoc Linh Nguyen (2017-2018)
- Peter Scherpelz (2016-2017)
......@@ -29,7 +31,7 @@ The former list of developers, in alphabetical order:
# Contributors
We would like to thank the following people for their contibution (beside generic comments and bugfixes), in alphabetical order:
We would like to thank the following people for their contributions, in alphabetical order:
- Francois Gygi (University of California, Davis)
- Ikutaro Hamada (Osaka University)
......
Change Log
==========
v4.3.0 (2021/05/26)
-------------------
- Introduced new data layout. Parallelization over bands allows to distribute data in a more flexible way in `wstat`. This feature also helps reduce memory per image. Band parallelization is enabled by specifying -nb xxx from the command line.
- Introduced checkpointing in `wstat`. With the new keyword `n_steps_write_restart` one can control how often the code produces restarts (default value is `1`).
- Improved I/O in `wfreq`. The number of I/O operations is reduced in `solve_wfreq`, and in the gamma case of `solve_gfreq`.
- Updated library dependency to Json-Fortran 8.2.1, resolving compilation issues with PGI 19.10.
- Updated the initialization of Forpy, such that in case of a module import error, the code has a better chance to print a clear error message.
- Updated build. Now the code builds with the NVIDIA/PGI Fortran compiler.
- Bug fix. Fixed an undefined variable in `Tools/set_npwq.f90`. Added `IMPLICIT NONE to all program units to let the compiler catch such errors in the future.
- Bug fix. Avoid overflow in `Wstat/wstat_memory_report.f90` in large-scale runs.
- Bug fix. Reset permissions to all source files in Westpp. All files appeared to be executable before.
- Bug fix. Reset `make.depend` files, added support to `make -j`.
- Updated CI/CD. Use updated Docker images.
- Updated CI/CD. Check numerical results and fail the CI if results don't match (see `check.py`). DFT checks error in total energy, WSTAT checks maximum error in PDEP eigenvalues, WFREQ checks maximum error in QP energies.
- Updated CI/CD. Added tests of images and OpenMP threads to nightly tests.
- Updated documentation. Fixed doc build with sphinx 3.5.0+.
- Updated documentation. Update build instructions for ALCF/Theta, NERSC/Cori, UChicago/RCC/Midway3, macOS.
- Updated documentation. Added more tutorials.
- Updated documentation. Updated manual.
v4.2.1 (2020/10/19)
-------------------
......
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......@@ -169,6 +169,8 @@ MODULE class_coulomb
USE gvect, ONLY : g, ngm
USE types_bz_grid, ONLY : q_grid
!
IMPLICIT NONE
!
! I/O
!
CLASS(coulomb) :: this
......@@ -304,6 +306,8 @@ MODULE class_coulomb
USE io_global, ONLY : stdout
USE types_bz_grid, ONLY : q_grid
!
IMPLICIT NONE
!
! I/O
!
CLASS(coulomb) :: this
......
store_sqvc.o : ../../Modules/cell_base.o
store_sqvc.o : ../../Modules/constants.o
store_sqvc.o : ../../Modules/io_global.o
store_sqvc.o : ../../Modules/kind.o
store_sqvc.o : ../../Modules/recvec.o
store_sqvc.o : ../Tools/class_bz_grid.o
store_sqvc.o : ../Tools/types_bz_grid.o
store_sqvc.o : ../../PW/src/exx.o
store_sqvc.o : ../../PW/src/pwcom.o
store_sqvc.o : ../../West/Modules/westcom.o
store_vspx.o : ../../Modules/cell_base.o
store_vspx.o : ../../Modules/constants.o
store_vspx.o : ../../Modules/kind.o
store_vspx.o : ../../Modules/recvec.o
store_vspx.o : ../../PW/src/exx.o
store_vspx.o : ../../PW/src/pwcom.o
class_coulomb.o : ../../Modules/cell_base.o
class_coulomb.o : ../../Modules/constants.o
class_coulomb.o : ../../Modules/control_flags.o
class_coulomb.o : ../../Modules/gvecw.o
class_coulomb.o : ../../Modules/io_global.o
class_coulomb.o : ../../Modules/kind.o
class_coulomb.o : ../../Modules/mp.o
class_coulomb.o : ../../Modules/mp_global.o
class_coulomb.o : ../../Modules/mp_world.o
class_coulomb.o : ../../Modules/random_numbers.o
class_coulomb.o : ../../Modules/recvec.o
class_coulomb.o : ../Modules/westcom.o
class_coulomb.o : ../Tools/types_bz_grid.o
types_coulomb.o : ../../Modules/kind.o
types_coulomb.o : class_coulomb.o
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......
......@@ -9,6 +9,7 @@ MODFLAGS= $(MOD_FLAG)../../iotk/src $(MOD_FLAG)../../Modules $(MOD_FLAG)../../LA
$(MOD_FLAG)../Tools \
$(MOD_FLAG)../FFT_kernel \
$(MOD_FLAG)../Coulomb_kernel \
$(MOD_FLAG)../Para_kernel \
$(MOD_FLAG).
IFLAGS=
......
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......@@ -7,7 +7,7 @@
!
! This file is part of WEST.
!
! Contributors to this file:
! Contributors to this file:
! Marco Govoni
!
!-----------------------------------------------------------------------
......@@ -40,7 +40,7 @@ SUBROUTINE apply_sternheimerop_to_m_wfcs(nbndval, psi, hpsi, e, alpha, m)
! input: the vector
! output: the operator applied to the vector
!
! Workspace
! Workspace
!
INTEGER :: ibnd, ig
COMPLEX(DP) :: za
......@@ -50,11 +50,15 @@ SUBROUTINE apply_sternheimerop_to_m_wfcs(nbndval, psi, hpsi, e, alpha, m)
! compute the product of the hamiltonian with the h vector
!
hpsi=(0.0_DP,0.0_DP)
!
!
IF(l_kinetic_only) THEN
CALL k_psi( npwx, npw, m, psi, hpsi )
ELSE
CALL h_psi( npwx, npw, m, psi, hpsi )
!
! use h_psi_, i.e. h_psi without band parallelization, as wstat
! handles band parallelization separately in dfpt_module
!
CALL h_psi_( npwx, npw, m, psi, hpsi )
ENDIF
!
! then we compute the operator H-epsilon S
......@@ -74,4 +78,4 @@ SUBROUTINE apply_sternheimerop_to_m_wfcs(nbndval, psi, hpsi, e, alpha, m)
!
CALL stop_clock ('stern')
!
END SUBROUTINE
END SUBROUTINE
This diff is collapsed.
!
! Copyright (C) 2015-2017 M. Govoni
! Copyright (C) 2015-2021 M. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......
apply_sternheimerop_to_m_wfcs.o : ../../Modules/becmod.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/control_flags.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/kind.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/mp.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/mp_global.o
......@@ -7,34 +6,37 @@ apply_sternheimerop_to_m_wfcs.o : ../../Modules/noncol.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/uspp.o
apply_sternheimerop_to_m_wfcs.o : ../../Modules/wavefunctions.o
apply_sternheimerop_to_m_wfcs.o : ../../PW/src/pwcom.o
apply_sternheimerop_to_m_wfcs.o : ../../West/Modules/westcom.o
dfpt.o : ../../Modules/constants.o
dfpt.o : ../../Modules/control_flags.o
dfpt.o : ../../Modules/fft_base.o
dfpt.o : ../../Modules/io_files.o
dfpt.o : ../../Modules/io_global.o
dfpt.o : ../../Modules/kind.o
dfpt.o : ../../Modules/mp.o
dfpt.o : ../../Modules/mp_global.o
dfpt.o : ../../Modules/noncol.o
dfpt.o : ../../Modules/recvec.o
dfpt.o : ../../Modules/uspp.o
dfpt.o : ../../Modules/wavefunctions.o
dfpt.o : ../../PW/src/buffers.o
dfpt.o : ../../PW/src/pwcom.o
dfpt.o : ../../West/FFT_kernel/fft_at_gamma.o
dfpt.o : ../../West/FFT_kernel/fft_at_k.o
dfpt.o : ../../West/Modules/westcom.o
dfpt.o : ../../West/Tools/bar.o
dfpt.o : ../../West/Tools/io_push.o
dfpt.o : ../../West/Tools/types_bz_grid.o
dfpt.o : ../../PW/src/exx.o
apply_sternheimerop_to_m_wfcs.o : ../Modules/westcom.o
dfpt_module.o : ../../Modules/cell_base.o
dfpt_module.o : ../../Modules/control_flags.o
dfpt_module.o : ../../Modules/fft_base.o
dfpt_module.o : ../../Modules/io_global.o
dfpt_module.o : ../../Modules/kind.o
dfpt_module.o : ../../Modules/mp.o
dfpt_module.o : ../../Modules/mp_global.o
dfpt_module.o : ../../Modules/mp_world.o
dfpt_module.o : ../../Modules/noncol.o
dfpt_module.o : ../../Modules/recvec.o
dfpt_module.o : ../../Modules/uspp.o
dfpt_module.o : ../../Modules/wavefunctions.o
dfpt_module.o : ../../PW/src/buffers.o
dfpt_module.o : ../../PW/src/pwcom.o
dfpt_module.o : ../FFT_kernel/fft_at_gamma.o
dfpt_module.o : ../FFT_kernel/fft_at_k.o
dfpt_module.o : ../Modules/westcom.o
dfpt_module.o : ../Para_kernel/class_idistribute.o
dfpt_module.o : ../Para_kernel/distribution_center.o
dfpt_module.o : ../Tools/bar.o
dfpt_module.o : ../Tools/io_push.o
dfpt_module.o : ../Tools/types_bz_grid.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/control_flags.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/kind.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/mp.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/mp_global.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/noncol.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/recvec.o
linsolve_sternheimer_m_wfcts.o : ../../Modules/wavefunctions.o
linsolve_sternheimer_m_wfcts.o : ../../PW/src/pwcom.o
linsolve_sternheimer_m_wfcts.o : ../../West/Modules/westcom.o
linsolve_sternheimer_m_wfcts.o : ../Modules/westcom.o
precondition_m_wfcts.o : ../../Modules/kind.o
precondition_m_wfcts.o : ../../Modules/noncol.o
precondition_m_wfcts.o : ../../PW/src/pwcom.o
!
! Copyright (C) 2015-2017. Govoni
! Copyright (C) 2015-2021. Govoni
! This file is distributed under the terms of the
! GNU General Public License. See the file `License'
! in the root directory of the present distribution,
......
......@@ -15,10 +15,11 @@ Project Leads
Developers
----------
- Sijia Dong (Argonne National Laboratory)
- Marco Govoni (Argonne National Laboratory and University of Chicago)
- He Ma (University of Chicago)
- Yu Jin (University of Chicago)
- Nan Sheng (University of Chicago)
- Han Yang (University of Chicago)
- Wenzhe Yu (Argonne National Laboratory)
Contributors
------------
......@@ -35,7 +36,10 @@ Former Developers
-----------------
- Nicholas Brawand (2016-2018)
- Sijia Dong (2019-2020)
- Matteo Gerosa (2017-2018)
- Lan Huang (2019-2020)
- He Ma (2017-2020)
- Ryan McAvoy (2017-2018)
- Ngoc Linh Nguyen (2017-2018)
- Peter Scherpelz (2016-2018)
......
......@@ -60,7 +60,7 @@ master_doc = 'index'
# General information about the project.
project = data["name"]
copyright = u'2020, Marco Govoni'
copyright = u'2021, Marco Govoni'
author = u'Marco Govoni'
# The version info for the project you're documenting, acts as replacement for
......@@ -157,11 +157,7 @@ html_logo = "images/logo.jpg"
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_context = {
'css_files': [
'_static/theme_overrides.css', # override wide tables in RTD theme
],
}
html_ccs_files = ['theme_overrides.css'] # override wide tables in RTD theme
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
......@@ -315,6 +311,5 @@ texinfo_documents = [
#texinfo_no_detailmenu = False
intersphinx_mapping = {
'python': ('https://docs.python.org/3.6', None),
'pymongo': ('https://api.mongodb.com/python/current/', None)
'python': ('https://docs.python.org/3.6', None)
}
......@@ -4,24 +4,26 @@
Installation
============
In order to install WEST you need to download the `QuantumEspresso 6.1 <https://gitlab.com/QEF/q-e/-/archive/qe-6.1.0/q-e-qe-6.1.0.tar>`_.
In order to install WEST you need to download the `QuantumEspresso 6.1.0 <https://gitlab.com/QEF/q-e/-/archive/qe-6.1.0/q-e-qe-6.1.0.tar>`_.
To compute absorption spectra (BSE), you also need to download and install `Qbox <http://qboxcode.org>`_.
`QuantumEspresso <http://www.quantum-espresso.org/>`_ (QE) is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale, based on density-functional theory (DFT), plane waves (PW), and pseudopotentials (PP).
Configure QuantumEspresso by running the ``configure`` script that comes with the QE distribution. WEST requires `MPI <https://en.wikipedia.org/?title=Message_Passing_Interface>`_ support (`Scalapack <http://www.netlib.org/scalapack/>`_ and `OpenMP <http://openmp.org/>`_ support is also recommended, but optional). If all the environment variables (compilers, libraries etc.) have been set according to the QE configure guide, this would simply be:
Configure QuantumEspresso by running the ``configure`` script that comes with the QE distribution. WEST requires `MPI <https://en.wikipedia.org/?title=Message_Passing_Interface>`_ support (`ScaLAPACK <http://www.netlib.org/scalapack/>`_ and `OpenMP <http://openmp.org/>`_ support is also recommended, but optional). If all the environment variables (compilers, libraries etc.) have been set according to the QE configure guide, this would simply be:
.. code-block:: bash
.. code-block:: bash
$ git clone -b 'qe-6.1.0' --single-branch --depth 1 https://gitlab.com/QEF/q-e.git QEdir
$ cd QEdir
$ git clone -b 'v4.2.1' --single-branch --depth 1 http://greatfire.uchicago.edu/west-public/West.git West
$ ./configure
$ ./configure
.. note::
Note that since v4.0.0 WEST requires dynamic linking and python3.
.. note::
Note that since v4.0.0 WEST requires dynamic linking and Python3.
It's now time to create the ``pw.x``, ``wstat.x``, ``wfreq.x``, and ``westpp.x`` executables by doing:
.. code-block:: bash
.. code-block:: bash
$ cd QEdir
$ make pw
......@@ -31,7 +33,7 @@ It's now time to create the ``pw.x``, ``wstat.x``, ``wfreq.x``, and ``westpp.x``
You have succefully installed QuantumEspresso and WEST if you see the executables ``pw.x``, ``wstat.x``, ``wfreq.x``, and ``westpp.x`` created in the QEdir/bin directory.
.. code-block:: bash
.. code-block:: bash
$ ls QEdir/bin/
pw.x
......@@ -40,7 +42,7 @@ You have succefully installed QuantumEspresso and WEST if you see the executable
westpp.x
... (other content) ...
Congratulations, you are all set for running QuantumEspresso and WEST!
Congratulations, you are all set for running QuantumEspresso and WEST!
Suggested configuration options
......@@ -49,6 +51,7 @@ Suggested configuration options
.. toctree::
:maxdepth: 1
installations/cori.rst
installations/theta.rst
installations/midway.rst
installations/macosx.rst
installations/midway3.rst
installations/macos.rst
.. _cori:
=================
Cori-NERSC (XC40)
=================
Cori is a Cray XC40 located at National Energy Research Scientific Computing Center (`NERSC <https://www.nersc.gov/>`_).
.. code-block:: bash
$ ssh <username>@cori.nersc.gov
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script (tested on May 12, 2021):
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module unload cray-libsci
module load cray-python/3.8.2.1
export CRAYPE_LINK_TYPE=dynamic
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export MPIF90=ftn
export F77=ftn
export CC=cc
export SCALAPACK_LIBS="$MKLROOT/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.so $MKLROOT/lib/intel64/libmkl_intel_thread.so $MKLROOT/lib/intel64/libmkl_core.so $MKLROOT/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group"
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel --with-hdf5=no
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Cori (Haswell partition) with 32 MPI ranks per node. The <project_name> must be replaced with an active project allocation.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --job-name=WEST
#SBATCH --time=00:20:00
#SBATCH --account=<project_name>
#SBATCH --constraint=haswell
#SBATCH --qos=debug
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
module unload cray-libsci
module load cray-python/3.8.2.1
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export OMP_NUM_THREADS=1
export OMP_PLACE=threads
export OMP_PROC_BIND=spread
export MKL_NUM_THREADS=$OMP_NUM_THREADS
NTASKS=$(($SLURM_NTASKS_PER_NODE * $SLURM_JOB_NUM_NODES))
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS_PER_NODE -c $SLURM_CPUS_PER_TASK ./wstat.x -i wstat.in &> wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the `NERSC user guide <https://docs.nersc.gov/systems/cori/>`_.
.. _macosx:
.. _macos:
======
MacOSX
======
=====
macOS
=====
The following instructions have been tested on MacOSX 10.14.6.
The following instructions have been tested on macOS 11.3.
Requirements:
Requirements:
- Gcc/Gfortran 9
- MPICH
- Blas/Lapack/Scalapack
- gcc/gfortran (e.g. GCC 9)
- MPICH
- BLAS/LAPACK/ScaLAPACK
- FFTW3
- Python3
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
WEST executables can be compiled using the following script:
.. code-block:: bash
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
......@@ -40,17 +40,18 @@ WEST executables can be compiled using the following script:
export SCALAPACK_LIBS=${MY_LIB_PATH}/SCALAPACK/libscalapack.a
export FFT_LIBS="${MY_LIB_PATH}/FFTW3/lib/libfftw3.a ${MY_LIB_PATH}/FFTW3/lib/libfftw3_omp.a"
./configure --with-scalapack --enable-openmp
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack --with-hdf5=no
make -j 4 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
To use the script do:
.. code-block:: bash
.. code-block:: bash
$ bash build_west.sh
......@@ -60,7 +61,7 @@ Running WEST
We can run the `wstat.x` WEST executables on 2 cores using the following command:
.. code-block:: bash
.. code-block:: bash
$ export OMP_NUM_THREADS=1
$ mpirun -np 2 ./wstat.x -i wstat.in > wstat.out
.. _midway:
===============
Midway-UChicago
===============
Midway is the HPC cluster of the University of Chicago, maintained by UChicago's `RCC <https://rcc.uchicago.edu/>`_.
.. code-block:: bash
$ ssh -Y <username>@midway.rcc.uchicago.edu
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module load intelmpi/5.1+intel-16.0 mkl/2017.up4 python/cpython-3.8.5
export F77=mpiifort
export CC=mpiicc
export MPIF90=mpiifort
export FC=mpiifort
export CFLAGS="-O3 -fno-alias -ansi-alias -g -mkl -Bdynamic"
export FFLAGS="-O3 -fno-alias -ansi-alias -g -mkl -Bdynamic"
export BLAS_LIBS_SWITCH="external"
export BLAS_LIBS=" -lmkl_intel_lp64 -lmkl_sequential -lmkl_core"
export LAPACK_LIBS_SWITCH="external"
export LAPACK_LIBS=" "
export SCALAPACK_LIBS=" -lmkl_scalapack_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group"
./configure --enable-openmp=yes --enable-parallel=yes --with-scalapack=intel --with-hdf5=no
make -j 2 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of the `broadwl` partition with 28 MPI ranks per node.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --time=00:30:00
#SBATCH --partition=broadwl
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=28
#SBATCH --cpus-per-task=1
module load intelmpi/5.1+intel-16.0 mkl/2017.up4 python/cpython-3.8.5
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
NTASKS=$(($SLURM_NTASKS_PER_NODE * $SLURM_JOB_NUM_NODES))
mpirun -np $NTASKS ./wstat.x -i wstat.in > wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the RCC user guide (`https://rcc.uchicago.edu/docs/ <https://rcc.uchicago.edu/docs/>`_).
.. _midway3:
================
Midway3-UChicago
================
Midway3 is the HPC cluster of the University of Chicago, maintained by UChicago's `RCC <https://rcc.uchicago.edu/>`_.
.. code-block:: bash
$ ssh <username>@midway3.rcc.uchicago.edu
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module load intel/19.1.1
module load intelmpi/2019.up7+intel-19.1.1
module load mkl/2020.up1
module load python/anaconda-2020.11
export MPIF90=mpiifort
export F90=ifort
export F77=ifort
export CC=icc
export SCALAPACK_LIBS="-lmkl_scalapack_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group"
./configure --enable-parallel --with-scalapack --enable-openmp
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
sed -i 's/-L.*config-3.8-x86_64-linux-gnu //' west_make.inc
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Midway3 with 32 MPI ranks per node. The <project_name> and <account_name> must be replaced with an active project allocation.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --time=00:20:00
#SBATCH --partition=<partition_name>
#SBATCH --account=<account_name>
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --cpus-per-task=1
module load intel/19.1.1
module load intelmpi/2019.up7+intel-19.1.1
module load mkl/2020.up1
module load python/anaconda-2020.11