Ondřej Šrámek <ondrej.sramek at gmail.com> on April 1 (no joke) and April 8, 2015
Department of Geophysics, Faculty of Mathematics and Physics, Charles University in Prague
Source for this Sphinx generated html: index.rst, conf.py, Makefile
See Láďa’s Ubuntu 14.04 update!
CitcomS is a 3-D spherical-shell finite element code for modeling solid-state convection in planetary mantles. It was developed by Louis Moresi (2-D, 3-D Cartesian) and Shijie Zhong (parallelized 3-D spherical) and donated to CIG, or Computational Infrastructure in Geodynamics (http://geodynamics.org/), for further development and public relsease. See Manual for more detailed history and complete list of developers. We will be using the latest CIG version.
Download code from https://geodynamics.org/cig/software/citcoms/, e.g., CitcomS-3.3.1.tar.gz is version 3.3.1 source downloaded on 2/24/2015.
There is also a wiki site http://wiki.geodynamics.org/software:citcoms:start.
In /nfs1/pc/install/citcoms:
Here written in non-dimensional form as presented in the Manual – except in invariant form and with no phase change (which is also included in CitcomS):
Rayleigh number ... \(Ra = \frac{\rho g \alpha \Delta T R^2}{\kappa \eta}\) (for Earth, 10.7 times larger than Ra calculated with mantle thickness)
Chemical Rayleigh number ... \(Ra_c = Ra\frac{\delta\rho_c}{\rho \alpha \Delta T}\)
Dissipation number ... \(Di = \frac{\alpha g R}{c_P}\) (Earth: \(Di \approx 0.5\))
Gruneisen parameter ... \(\gamma = \frac{\alpha K_S}{\rho c_P}\) (Earth: \(\gamma \approx 1\))
Dimensionless surface temperature ... \(T_s = \frac{T_\text{surf}}{\Delta T}\)
Dimensionless internal heating rate ... \(H=\frac{\rho R^2 \tilde{H}}{k \Delta T}\), where \(\tilde{H}\) is dimensional heating rate (in W/kg)
Setting gruneisen=0 (understood as infinite \(\gamma\)) in input file computes an incompressible extended-Boussinesq case (with adiabatic heating, dissipative heating, latent heating of phase change), additionally setting dissipation_number=0 gives classical Boussinesq.
Requires C compiler and MPI library
Standard ”./configure – make – make install” process (last step optional)
Extract CitcomS-3.3.1.tar.gz and step into directory CitcomS-3.3.1 in /nfsz1/sramek (will refer to this as $install)
$ ./configure –-help shows configure options
$ ./configure
================ Configuration Summary ================
CC: mpicc
CFLAGS: -g -O2
CPPFLAGS: -DUSE_GZDIR
LDFLAGS:
LIBS: -lz -lhdf5
with-hdf5: yes
with-ggrd: no
$ make
Upon successful build:
Created subdirectory /nfsz1/sramek/citcomsruns, also /nfsz1/sramek/oneeye, to run examples (calling it $runs from now on)
Input file $install/examples/example0 (regional) included in CIG distribution
Running in directory $runs/Example0
$ mpirun -np 1 CitcomSRegional example0
or simply
$ CitcomSRegional example0
Output:
Input file $install/examples/example1 (regional)
Running in directory $runs/Example1
$ mpirun -np 4 CitcomSRegional example1
On geofxyz’s (with 12 cores), I can run on 8 cores too...
Needs password-less ssh access between machines.
Infiniband for fast network communication, shell variables (hopefully) properly set on geofxyz (see /etc/bash.bashrc):
export OMPI_MCA_btl_tcp_if_include=eth0
export OMPI_MCA_btl=openib,tcp,sm,self
Input file $install/examples/Full/input.sample, where I changed datadir=/home/sramek/citcomsruns/Full0 (the containing directory must exist on each machine)
Make sure all machines see the executable and the input file:
$ prog=/nfsz1/sramek/CitcomS-3.3.1/bin/CitcomSFull
$ mach=/nfsz1/sramek/citcomsruns/mach_z12
$ input=/nfsz1/sramek/citcomsruns/Full0.input
$ np=12
$ mpirun -np $np -machinefile $mach $prog $input
which I scripted in run.full0.sh
Machinefile used:
geofz1
geofz2
Can run from arbitrary directory, into which the pid file will be written, but all other output will go into datadir.
Input file $install/examples/Cookbook1/cookbook1
datadir=/home/sramek/citcomsruns/Cookbook1
run.cookbook1.sh
Anselm documentation: https://docs.it4i.cz/anselm-cluster-documentation
Before compilation/execution, load modules which “sets up the GNU development environment in conjunction with the bullx MPI library”:
$ module load PrgEnv-gnu
I spent little time playing with various hdf5-related modules, but couldn’t get CitcomS to compile with hdf5.
Jobs on anselm should be run on multiples of 16 of cores. CitcomS needs multiples of 12.
In CitcomS input file:
datadir="."
script to send to the queue ($ qsub script):
#!/bin/bash
#PBS -q qexp
#PBS -N CitcomS
#PBS -l select=3:ncpus=16:mpiprocs=16,walltime=00:10:00
#PBS -A OPEN-4-16
#PBS -m ae
case=MarsTherm8Test
program=$HOME/CitcomS-3.3.1-work/bin/CitcomSFull
inputfile=/scratch/$USER/$case/$case.input
outputfile=/scratch/$USER/$case/$case.output
# change to scratch directory, exit on failure
scrdir=/scratch/$USER/$case
cd $scrdir || exit
# load modules
module load PrgEnv-gnu
# execute the calculation
mpiexec $program $inputfile &> $outputfile
# exit
exit
output_format=ascii (default) or ascii-gz or hdf5
output_optional="surf,botm,tracer" (default), also possible to include horiz_avg, geoid, seismic, stress, pressure, connectivity, heating, comp_el, comp_nd
Visualization: OpenDX, MayaVi2, GMT
PREFIX.log
PREFIX.domain ... binary file on rank 0 processor, contains domain bounds of processors
PREFIX.QUANTITY.CORE.TIMESTEP (e.g., example0.velo.3.100):
*.chkpt.* ... binary files, for restart
*.botm.*
*.qt.dat
Use autocombine.py in $install/visual to prepare data files for OpenDX from CitcomS output files (Manual section 5.2):
(Changed remote_shell to ssh from the default rsh in batchcombine.py)
Fixed output_optional_EL issue in autocombine.py
I couldn’t get it to work to collect datafiles from multiple machines:
autocombine.py machinefile pidfile timestep
It worked to copy output files to one location ($runs/Cookbook1/visual), modify the pid file and run:
autocombine.py localhost pidfile timestep
Creates files PREFIX.capXX.TIMESTEP and PREFIX.capXX.TIMESTEP.general
Use OpenDX script visFull.net in $install/visual to plot temperature isosurface (Manual section 5.5). If you need to modify things substantially, have fun* tweaking the OpenDX script... (*sarcasm)
Straightforward but a bit tedious trick: Copy and modify selected output files to the format one would get with ascii output:
copy coord.* files and *.TIMESTEP files to a directory
prepend a prefix to filenames:
$ for f in *.56000 ; do mv "$f" "m8b.$f" ; done
$ for f in coord.* ; do mv "$f" "m8b.$f" ; done
and modify pid* file accordingly
follow steps for ascii visualization
CitcomS input files is an ascii file with lines of format parameter=value. Any text following # is a comment. Most parameters have default vaules set by the code, so a minimal input file may be used for some problems. An alternative practice is to use an input file with all parameter explicitly set. One can start with a pid* file generated by a test run and modify it into a desired input.
Few notes:
To start a “new” calculation but read temperature from an “old” one:
datadir="NEW"
datafile="new"
datadir_old="OLD"
datafile_old="old"
...
solution_cycles_init=21000
zero_elapsed_time=on # time counts from 0
tic_method=-1 # -1 .. read from *.velo.* files
To start a calculation from a supplied radial temperature profile (+perturbations):
Surprisingly, this initial condition on temperature was not implemented in the CIG version. I tweaked a function from my Boulder CitcomS version and included this option. You can check the source code in lib/Initial_temperature.c in my CitcomS installation in /nfsz1/sramek/CitcomS-3.3.1-work, search for osr which labels my modifications/additions (also little things is some other source and header files).
To use this initial condition, use this in input file:
tic_method=99
background_profile_file="file_rT.xy"
The Arrhenius-type viscosity relation, in non-dimensional form and with \(\eta_\text{cmb}=1\) constraint, is
Mini-script nondim_EVQ_4_CitcomS.py (in /nfs1/pc/install/citcoms) calculates the non-dimensional parameters.
struct All_variables *E; defined in global_defs.h contains all global data and is passed between functions.
For example E->monitor.solution_cycles is the current time step (integer).
Input file marstherm8, input parameters for case TC1 from Šrámek & Zhong 2010 JGR doi:10.1029/2010JE003597
Testing 10000 steps of Cookbook 1 case on geofz1 and geofz2.
| Number of cores | 12 | 24 |
| CPU time per velocity step [sec] | 0.219 | 0.135 |
| Wall clock time [sec] | 2155 | 1370 |
| Speed up relative to 12 cores | 1 | 1.63/1.57 |
When using CIG CitcomS, asked to acknowledge CIG and to cite these papers (the third one when using tracers):
I believe the presentation in this book inspired the early developers:
CitcomS 3-D spherical shell reference study:
C programming language:
[email of 8/30/2015]
CitcomS 3.3.1 je na všech geofech s Ubuntu 14.04. Instalace vyžadovala nějaké úpravy pro configure i v pythonovských skriptech, autocombine.py nyní pracuje i mezi stroji. OpenDX vykresluje ascii data, patrně i HDF data; HDF mezi více stroji chodit nemůže, nemáme potřebný “parallel filesystem”. GMT kreslí. Tvoje dokumentace by novou situaci mohla reflektovat v níže shrnutých bodech.
CitcomS na klastru KG
Uniprocessor example
Single-machine multi-core example
Multi-machine example
OpenDX visualization