Commit bd1b893f authored by Victor Yu's avatar Victor Yu
Browse files

Update installation instructions

* Theta
* Cori
* Midway3
* macOS 11
parent e20e0db3
......@@ -7,21 +7,21 @@ Installation
In order to install WEST you need to download the `QuantumEspresso 6.1 <https://gitlab.com/QEF/q-e/-/archive/qe-6.1.0/q-e-qe-6.1.0.tar>`_.
`QuantumEspresso <http://www.quantum-espresso.org/>`_ (QE) is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale, based on density-functional theory (DFT), plane waves (PW), and pseudopotentials (PP).
Configure QuantumEspresso by running the ``configure`` script that comes with the QE distribution. WEST requires `MPI <https://en.wikipedia.org/?title=Message_Passing_Interface>`_ support (`Scalapack <http://www.netlib.org/scalapack/>`_ and `OpenMP <http://openmp.org/>`_ support is also recommended, but optional). If all the environment variables (compilers, libraries etc.) have been set according to the QE configure guide, this would simply be:
Configure QuantumEspresso by running the ``configure`` script that comes with the QE distribution. WEST requires `MPI <https://en.wikipedia.org/?title=Message_Passing_Interface>`_ support (`ScaLAPACK <http://www.netlib.org/scalapack/>`_ and `OpenMP <http://openmp.org/>`_ support is also recommended, but optional). If all the environment variables (compilers, libraries etc.) have been set according to the QE configure guide, this would simply be:
.. code-block:: bash
.. code-block:: bash
$ git clone -b 'qe-6.1.0' --single-branch --depth 1 https://gitlab.com/QEF/q-e.git QEdir
$ cd QEdir
$ git clone -b 'v4.2.1' --single-branch --depth 1 http://greatfire.uchicago.edu/west-public/West.git West
$ ./configure
$ ./configure
.. note::
Note that since v4.0.0 WEST requires dynamic linking and python3.
.. note::
Note that since v4.0.0 WEST requires dynamic linking and Python3.
It's now time to create the ``pw.x``, ``wstat.x``, ``wfreq.x``, and ``westpp.x`` executables by doing:
.. code-block:: bash
.. code-block:: bash
$ cd QEdir
$ make pw
......@@ -31,7 +31,7 @@ It's now time to create the ``pw.x``, ``wstat.x``, ``wfreq.x``, and ``westpp.x``
You have succefully installed QuantumEspresso and WEST if you see the executables ``pw.x``, ``wstat.x``, ``wfreq.x``, and ``westpp.x`` created in the QEdir/bin directory.
.. code-block:: bash
.. code-block:: bash
$ ls QEdir/bin/
pw.x
......@@ -40,7 +40,7 @@ You have succefully installed QuantumEspresso and WEST if you see the executable
westpp.x
... (other content) ...
Congratulations, you are all set for running QuantumEspresso and WEST!
Congratulations, you are all set for running QuantumEspresso and WEST!
Suggested configuration options
......@@ -49,6 +49,7 @@ Suggested configuration options
.. toctree::
:maxdepth: 1
installations/cori.rst
installations/theta.rst
installations/midway.rst
installations/macosx.rst
installations/midway3.rst
installations/macos.rst
.. _cori:
=================
Cori-NERSC (XC40)
=================
Cori is a Cray XC40 located at National Energy Research Scientific Computing Center (`NERSC <https://www.nersc.gov/>`_).
.. code-block:: bash
$ ssh <username>@cori.nersc.gov
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script (tested on May 12, 2021):
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module unload cray-libsci
module load cray-python/3.8.2.1
export CRAYPE_LINK_TYPE=dynamic
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export MPIF90=ftn
export F77=ftn
export CC=cc
export SCALAPACK_LIBS="$MKLROOT/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.so $MKLROOT/lib/intel64/libmkl_intel_thread.so $MKLROOT/lib/intel64/libmkl_core.so $MKLROOT/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group"
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel --with-hdf5=no
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Cori (Haswell partition) with 32 MPI ranks per node. The <project_name> must be replaced with an active project allocation.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --job-name=WEST
#SBATCH --time=00:20:00
#SBATCH --account=<project_name>
#SBATCH --constraint=haswell
#SBATCH --qos=debug
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
module unload cray-libsci
module load cray-python/3.8.2.1
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2019.3.199/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export OMP_NUM_THREADS=1
export OMP_PLACE=threads
export OMP_PROC_BIND=spread
export MKL_NUM_THREADS=$OMP_NUM_THREADS
NTASKS=$(($SLURM_NTASKS_PER_NODE * $SLURM_JOB_NUM_NODES))
srun -N $SLURM_JOB_NUM_NODES -n $SLURM_NTASKS_PER_NODE -c $SLURM_CPUS_PER_TASK ./wstat.x -i wstat.in &> wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the `NERSC user guide <https://docs.nersc.gov/systems/cori/>`_.
.. _macosx:
.. _macos:
======
MacOSX
======
=====
macOS
=====
The following instructions have been tested on MacOSX 10.14.6.
The following instructions have been tested on macOS 11.3.
Requirements:
Requirements:
- Gcc/Gfortran 9
- MPICH
- Blas/Lapack/Scalapack
- gcc/gfortran (e.g. GCC 9)
- MPICH
- BLAS/LAPACK/ScaLAPACK
- FFTW3
- Python3
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
WEST executables can be compiled using the following script:
.. code-block:: bash
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
......@@ -40,17 +40,18 @@ WEST executables can be compiled using the following script:
export SCALAPACK_LIBS=${MY_LIB_PATH}/SCALAPACK/libscalapack.a
export FFT_LIBS="${MY_LIB_PATH}/FFTW3/lib/libfftw3.a ${MY_LIB_PATH}/FFTW3/lib/libfftw3_omp.a"
./configure --with-scalapack --enable-openmp
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack --with-hdf5=no
make -j 4 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
To use the script do:
.. code-block:: bash
.. code-block:: bash
$ bash build_west.sh
......@@ -60,7 +61,7 @@ Running WEST
We can run the `wstat.x` WEST executables on 2 cores using the following command:
.. code-block:: bash
.. code-block:: bash
$ export OMP_NUM_THREADS=1
$ mpirun -np 2 ./wstat.x -i wstat.in > wstat.out
.. _midway:
===============
Midway-UChicago
===============
Midway is the HPC cluster of the University of Chicago, maintained by UChicago's `RCC <https://rcc.uchicago.edu/>`_.
.. code-block:: bash
$ ssh -Y <username>@midway.rcc.uchicago.edu
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module load intelmpi/5.1+intel-16.0 mkl/2017.up4 python/cpython-3.8.5
export F77=mpiifort
export CC=mpiicc
export MPIF90=mpiifort
export FC=mpiifort
export CFLAGS="-O3 -fno-alias -ansi-alias -g -mkl -Bdynamic"
export FFLAGS="-O3 -fno-alias -ansi-alias -g -mkl -Bdynamic"
export BLAS_LIBS_SWITCH="external"
export BLAS_LIBS=" -lmkl_intel_lp64 -lmkl_sequential -lmkl_core"
export LAPACK_LIBS_SWITCH="external"
export LAPACK_LIBS=" "
export SCALAPACK_LIBS=" -lmkl_scalapack_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group"
./configure --enable-openmp=yes --enable-parallel=yes --with-scalapack=intel --with-hdf5=no
make -j 2 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of the `broadwl` partition with 28 MPI ranks per node.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --time=00:30:00
#SBATCH --partition=broadwl
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=28
#SBATCH --cpus-per-task=1
module load intelmpi/5.1+intel-16.0 mkl/2017.up4 python/cpython-3.8.5
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
NTASKS=$(($SLURM_NTASKS_PER_NODE * $SLURM_JOB_NUM_NODES))
mpirun -np $NTASKS ./wstat.x -i wstat.in > wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the RCC user guide (`https://rcc.uchicago.edu/docs/ <https://rcc.uchicago.edu/docs/>`_).
.. _midway3:
================
Midway3-UChicago
================
Midway3 is the HPC cluster of the University of Chicago, maintained by UChicago's `RCC <https://rcc.uchicago.edu/>`_.
.. code-block:: bash
$ ssh <username>@midway3.rcc.uchicago.edu
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module load intel/19.1.1
module load intelmpi/2019.up7+intel-19.1.1
module load mkl/2020.up1
module load python/anaconda-2020.11
export MPIF90=mpiifort
export F90=ifort
export F77=ifort
export CC=icc
export SCALAPACK_LIBS="-lmkl_scalapack_lp64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group"
./configure --enable-parallel --with-scalapack --enable-openmp
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
sed -i 's/-L.*config-3.8-x86_64-linux-gnu //' west_make.inc
make all
To use the script do:
.. code-block:: bash
$ bash build_west.sh
Running WEST Jobs
~~~~~~~~~~~~~~~~~
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Midway3 with 32 MPI ranks per node. The <project_name> and <account_name> must be replaced with an active project allocation.
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
#SBATCH --time=00:20:00
#SBATCH --partition=<partition_name>
#SBATCH --account=<account_name>
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=1
module load intel/19.1.1
module load intelmpi/2019.up7+intel-19.1.1
module load mkl/2020.up1
module load python/anaconda-2020.11
export LD_LIBRARY_PATH=/software/python-anaconda-2020.11-el8-x86_64/lib:$LD_LIBRARY_PATH
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
NTASKS=$(($SLURM_NTASKS_PER_NODE * $SLURM_JOB_NUM_NODES))
mpirun -np $NTASKS ./wstat.x -i wstat.in > wstat.out
Job submission is done with the following:
.. code-block:: bash
$ sbatch run_west.sh
.. seealso::
For more information, visit the `RCC user guide <https://rcc.uchicago.edu/docs/>`_.
......@@ -4,40 +4,46 @@
Theta-ALCF (XC40)
=================
Theta is a Cray XC40 located at Argonne National Laboratory, maintained by `ALCF <https://www.alcf.anl.gov/>`_.
Theta is a Cray XC40 located at Argonne National Laboratory, maintained by `ALCF <https://www.alcf.anl.gov/>`_.
.. code-block:: bash
.. code-block:: bash
$ ssh -Y <username>@theta.alcf.anl.gov
$ ssh <username>@theta.alcf.anl.gov
Building WEST
~~~~~~~~~~~~~
WEST executables can be compiled using the following script:
WEST executables can be compiled using the following script (tested on May 12, 2021):
.. code-block:: bash
.. code-block:: bash
$ cat build_west.sh
#!/bin/bash
module unload cray-libsci
module load cray-python/3.6.5.3
module load cray-python/3.8.2.1
export CRAYPE_LINK_TYPE=dynamic
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
export MPIF90=ftn
export F77=ftn
export CC=cc
export SCALAPACK_LIBS="$MKLROOT/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group $MKLROOT/lib/intel64/libmkl_intel_lp64.so $MKLROOT/lib/intel64/libmkl_intel_thread.so $MKLROOT/lib/intel64/libmkl_core.so $MKLROOT/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group"
./configure MPIF90=ftn CC=cc CXX=CC --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel SCALAPACK_LIBS="${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.so -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.so ${MKLROOT}/lib/intel64/libmkl_intel_thread.so ${MKLROOT}/lib/intel64/libmkl_core.so ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.so -Wl,--end-group" FFLAGS=" -xMIC-AVX512 -qopenmp -align array64byte -fp-model fast=2 -no-prec-div -assume byterecl" --with-hdf5=no CFLAGS=" -xMIC-AVX512" LDFLAGS=" -shared-intel -qopenmp"
./configure --enable-openmp=yes --enable-parallel=yes --enable-shared=yes --with-scalapack=intel --with-hdf5=no
make pw -j 16
make -j 8 pw
cd West
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags`"
make all
To use the script do:
make conf PYT=python3 PYT_LDFLAGS="`python3-config --ldflags --embed`"
make all
To use the script do:
.. code-block:: bash
.. code-block:: bash
$ bash build_west.sh
......@@ -46,7 +52,7 @@ Running WEST Jobs
The following is an example executable script `run_west.sh` to run the `wstat.x` WEST executable on two nodes of Theta with 64 MPI ranks per node. The <project_name> must be replaced with an active project allocation.
.. code-block:: bash
.. code-block:: bash
$ cat run_west.sh
#!/bin/bash
......@@ -62,28 +68,28 @@ The following is an example executable script `run_west.sh` to run the `wstat.x`
HT=1
module unload cray-libsci
module load cray-python/3.6.5.3
export CRAYPE_LINK_TYPE=dynamic
module load cray-python/3.8.2.1
export LD_LIBRARY_PATH=$MKLROOT/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.6.5.3/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/python/3.8.2.1/lib:$LD_LIBRARY_PATH
echo "Running Cobalt Job $COBALT_JOBID."
export OMP_NUM_THREADS=$NTHREADS
aprun -n $MPIRANKS -N $MPIRANKS_PERNODE -cc depth -d $NTHREADS -j $HT ./wstat.x -i wstat.in &> wstat.out
Make the script executable:
Make the script executable:
.. code-block:: bash
.. code-block:: bash
$ chmod +x run_west.sh
$ chmod 755 run_west.sh
Job submission is done with the following:
Job submission is done with the following:
.. code-block:: bash
.. code-block:: bash
$ qsub run_west.sh
.. seealso::
For more information, visit the ALCF user guide (`https://www.alcf.anl.gov/user-guides/xc40-system-overview <https://www.alcf.anl.gov/user-guides/xc40-system-overview/>`_).
For more information, visit the `ALCF user guide <https://www.alcf.anl.gov/user-guides/xc40-system-overview/>`_.
......@@ -3,19 +3,19 @@
Manual
======
The complete **WEST** reference for input parameters.
The complete **WEST** reference for input parameters.
.. contents:: :local:
:depth: 1
.. note::
Not all input parameters listed below are mandatory. Check :ref:`quickreference` or :ref:`tutorial` pages to see examples of input files.
.. note::
Not all input parameters listed below are mandatory. Check :ref:`quickreference` or :ref:`tutorial` pages to see examples of input files.
.. seealso::
**WESTpy** is a Python package, designed to assist users of the WEST code in pre- and post-process massively parallel calculations. Click `here <http://www.west-code.org/doc/westpy/latest/>`_ to know more.
.. seealso::
The input file is given according to the YAML Notation (`https://yaml.org/ <https://yaml.org//>`_).
The input file is given according to the YAML Notation (`https://yaml.org/ <https://yaml.org//>`_).
|
......@@ -48,7 +48,7 @@ input_west
* - **Default**
- "west"
* - **Description**
- Prefix prepended to the WEST save and restart folders.
- Prefix prepended to the WEST save and restart folders.
.. data:: outdir
......@@ -62,8 +62,8 @@ input_west
* - **Default**
- "./"
* - **Description**
- Directory for: input, temporary, and output files.
- Directory for: input, temporary, and output files.
|
......@@ -86,8 +86,8 @@ wstat_control
- "S" : Start from scratch.
- "R" : Restart from an interrupted run. You should restart with the same number of cores, and images.
- "E" : Calculation of the response is external, i.e. outsourced to a server.
- "E" : Calculation of the response is external, i.e. outsourced to a server.
.. data:: n_pdep_eigen
......@@ -284,7 +284,7 @@ wfreq_control
- "XWO" : Compute the optical properties.
- "XWGQP" : Compute the QP corrections, and plot spectral functions.
- "XWGQOP" : Compute all.
.. data:: n_pdep_eigen_to_use
......@@ -382,7 +382,7 @@ wfreq_control
* - **Type**
- float
* - **Default**
- dynamically se to the cutoff energy of the density, read from the ground state
- dynamically set to the cutoff energy of the density, read from the ground state
* - **Description**
- Cutoff for the imaginary frequencies (in Ry).
......@@ -568,7 +568,7 @@ westpp_control
- "C"
* - **Description**
- Available options for the output fortmat are:
- "c" : Cube.
- "x" : Planar average yz.
- "y" : Planar average xz.
......@@ -679,5 +679,5 @@ server_control
- The document is serialized into a JSON string and passed to the server (see `West/Pytools/west_clientserver.py`).
|
%% Cell type:markdown id: tags:
This tutorial can be downloaded [link](http://greatfire.uchicago.edu/west-public/West/raw/master/doc/tutorials/west_100.ipynb).
%% Cell type:markdown id: tags:
# 1.0 Getting Started: GW calculation
%% Cell type:markdown id: tags:
In order to compute the GW electronic structure of the silane molecule you need to run `pw.x`, `wstat.x` and `wfreq.x` in sequence.
%% Cell type:markdown id: tags:
## Step 1: Ground State
%% Cell type:markdown id: tags:
The ground state electronic structure of silane molecule with [QuantumEspresso](https://www.quantum-espresso.org/) is obtained by running `pw.x`. The pseudopotential files for **Si** and **H** in UPF format can be downloaded from: [QE-PP](https://www.quantum-espresso.org/pseudopotentials) database, or from [SG15](http://www.quantum-simulation.org/potentials/sg15_oncv/upf/) database. Check out the `pw.x` [input description](https://www.quantum-espresso.org/Doc/INPUT_PW.html) in order to generate an input file for QuantumEspresso called `pw.in`.
%% Cell type:markdown id: tags:
Download these files in your current working directory:
%% Cell type:code id: tags:
``` python
%%bash
wget -N -q http://www.west-code.org/doc/training/silane/pw.in
wget -N -q http://www.quantum-simulation.org/potentials/sg15_oncv/upf/H_ONCV_PBE-1.0.upf
wget -N -q http://www.quantum-simulation.org/potentials/sg15_oncv/upf/Si_ONCV_PBE-1.1.upf
```
%% Cell type:markdown id: tags:
Let's inspect the ``pw.in`` file, input for ``pw.x``.
%% Cell type:code id: tags:
``` python
%%bash
cat pw.in
```
%%%% Output: stream
&control
calculation = 'scf'
restart_mode = 'from_scratch'
pseudo_dir = './'
outdir = './'
prefix = 'silane'
wf_collect = .TRUE.
/
&system
ibrav = 1
celldm(1) = 20
nat = 5
ntyp = 2
ecutwfc = 25.0
nbnd = 10
assume_isolated ='mp'
/
&electrons
diago_full_acc = .TRUE.
/
ATOMIC_SPECIES
Si 28.0855 Si_ONCV_PBE-1.1.upf
H 1.00794 H_ONCV_PBE-1.0.upf
ATOMIC_POSITIONS bohr
Si 10.000000 10.000000 10.000000
H 11.614581 11.614581 11.614581
H 8.385418 8.385418 11.614581
H 8.385418 11.614581 8.385418
H 11.614581 8.385418 8.385418
K_POINTS {gamma}
%% Cell type:markdown id: tags:
Run ``pw.in`` on 2 cores.