Bořivoj’s CitcomS @ KG’s notes

Ondřej Šrámek <ondrej.sramek at> on April 1 (no joke) and April 8, 2015

Department of Geophysics, Faculty of Mathematics and Physics, Charles University in Prague

Source for this Sphinx generated html: index.rst,, Makefile


See Láďa’s Ubuntu 14.04 update!

CitcomS is a 3-D spherical-shell finite element code for modeling solid-state convection in planetary mantles. It was developed by Louis Moresi (2-D, 3-D Cartesian) and Shijie Zhong (parallelized 3-D spherical) and donated to CIG, or Computational Infrastructure in Geodynamics (, for further development and public relsease. See Manual for more detailed history and complete list of developers. We will be using the latest CIG version.

Download code from, e.g., CitcomS-3.3.1.tar.gz is version 3.3.1 source downloaded on 2/24/2015.

User Manual is accessible from that site as well. Current direct link is
Note the manual lags by one release number behind the code. As the code is being continually developed, there are occasional inconsistencies between the code and the manual.

There is also a wiki site

In /nfs1/pc/install/citcoms:

  • citcoms-3.3.0-manual.pdf ... CitcomS Manual from CIG
  • CitcomS-3.3.1.tar.gz ... CitcomS code downloaded from CIG.
  • CitcomS_notes_KG_2014.odt ... My own notes when I was working in Shijie’s group that reflect the (non-CIG) version I was using.
  • input_marstherm8 ... Commented input file for thermal convection calculation, case TC1 from Šrámek & Zhong 2010 JGR, doi:10.1029/2010JE003597
  • ... dim-to-nondim calculation of T- and P-dependent viscosity parameters
  • Zhong_etal_2008_G3_CitcomS_benchmark.pdf ... CitcomS benchmark paper, Zhong et al. 2008 G3 doi:10.1029/2008GC002048


Here written in non-dimensional form as presented in the Manual – except in invariant form and with no phase change (which is also included in CitcomS):

Truncated Anelastic Liquid Approximation
“Truncated” because pressure effect in buoyancy force is neglected.
\[\begin{split}\boldsymbol\nabla\cdot(\bar\rho \mathbf{v}) = 0 \\\end{split}\]
\[-\boldsymbol\nabla P + \boldsymbol\nabla\cdot \left[ \eta \left( \boldsymbol\nabla\mathbf{v} + \boldsymbol\nabla^T\mathbf{v} - \frac{2}{3}\boldsymbol\nabla\cdot\mathbf{v}\mathbf{I} \right) \right] + (Ra \bar\rho \alpha T - Ra_c C) g \mathbf{e}_r = 0\]
\[\bar\rho c_P \left( \frac{\partial T}{\partial t} + \mathbf{v} \cdot \boldsymbol\nabla T \right) = \bar\rho c_P \kappa \nabla^2 T - \bar\rho \alpha g v_r Di (T+T_s) + \frac{Di}{Ra}\Phi + \bar\rho H\]

Rayleigh number ... \(Ra = \frac{\rho g \alpha \Delta T R^2}{\kappa \eta}\) (for Earth, 10.7 times larger than Ra calculated with mantle thickness)

Chemical Rayleigh number ... \(Ra_c = Ra\frac{\delta\rho_c}{\rho \alpha \Delta T}\)

Dissipation number ... \(Di = \frac{\alpha g R}{c_P}\) (Earth: \(Di \approx 0.5\))

Gruneisen parameter ... \(\gamma = \frac{\alpha K_S}{\rho c_P}\) (Earth: \(\gamma \approx 1\))

Dimensionless surface temperature ... \(T_s = \frac{T_\text{surf}}{\Delta T}\)

Dimensionless internal heating rate ... \(H=\frac{\rho R^2 \tilde{H}}{k \Delta T}\), where \(\tilde{H}\) is dimensional heating rate (in W/kg)

Setting gruneisen=0 (understood as infinite \(\gamma\)) in input file computes an incompressible extended-Boussinesq case (with adiabatic heating, dissipative heating, latent heating of phase change), additionally setting dissipation_number=0 gives classical Boussinesq.


  • Streamline upwind Petrov-Galerkin approx (SUPG) for the energy equation
  • Mixed formulation in primitive variables (P,v) and Uzawa algorithm with two-loop iterations for Stokes problem
  • Uzawa: outer loop (P) – preconditioned conjugate gradient method; inner loop (v) – full multigrid methods
  • Gauss-Seidel iteration for inner nodes, Jacobi iteration for shared nodes
  • brick elements, 8 velocity nodes w/ trilinear interpolation, 1 constant pressure node
  • Predictor–corrector and 2nd order Runge-Kutta to advect tracers, ratio method used to map tracers to composition
  • Poisson equation for gravitational potential solved with a spectral method

CitcomS on geofxyz


Requires C compiler and MPI library

Standard ”./configure – make – make install” process (last step optional)

Extract CitcomS-3.3.1.tar.gz and step into directory CitcomS-3.3.1 in /nfsz1/sramek (will refer to this as $install)

$ ./configure –-help shows configure options

$ ./configure

================ Configuration Summary ================
     CC:  mpicc
     CFLAGS:  -g -O2
     LIBS:  -lz -lhdf5
     with-hdf5:  yes
     with-ggrd:  no

$ make

Upon successful build:

  • Executables CitcomSFull and CitcomSRegional placed in directory $install/bin
  • Main function int main() in file Citcom.c in $install/bin
  • The rest of source (*.c) and header (*.h) files in $install/lib
  • Visualization aid in $install/visual
  • Examples in $install/examples

Created subdirectory /nfsz1/sramek/citcomsruns, also /nfsz1/sramek/oneeye, to run examples (calling it $runs from now on)

Uniprocessor example

Input file $install/examples/example0 (regional) included in CIG distribution

Running in directory $runs/Example0

$ mpirun -np 1 CitcomSRegional example0

or simply

$ CitcomSRegional example0


  • runtime output on screen (probably want to redirect to a file)
  • pid* file list all parameter values, including those omitted in input file
  • PREFIX.log file, similar to screen output
  • output data files

Single-machine multi-core example

Input file $install/examples/example1 (regional)

Running in directory $runs/Example1

$ mpirun -np 4 CitcomSRegional example1

On geofxyz’s (with 12 cores), I can run on 8 cores too...

Multi-machine example

Needs password-less ssh access between machines.

Infiniband for fast network communication, shell variables (hopefully) properly set on geofxyz (see /etc/bash.bashrc):

export OMPI_MCA_btl_tcp_if_include=eth0
export OMPI_MCA_btl=openib,tcp,sm,self

Input file $install/examples/Full/input.sample, where I changed datadir=/home/sramek/citcomsruns/Full0 (the containing directory must exist on each machine)

Make sure all machines see the executable and the input file:

$ prog=/nfsz1/sramek/CitcomS-3.3.1/bin/CitcomSFull
$ mach=/nfsz1/sramek/citcomsruns/mach_z12
$ input=/nfsz1/sramek/citcomsruns/Full0.input
$ np=12
$ mpirun -np $np -machinefile $mach $prog $input

which I scripted in

Machinefile used:


Can run from arbitrary directory, into which the pid file will be written, but all other output will go into datadir.

Cookbook examples

  1. Global model
  2. Regional model with velocity boundary conditions
  3. Temperature-dependent viscosity (regional)
  4. Regionally refined meshes
  5. Subduction models with trench rollback
  6. Pseudo-free-surface formulation
  7. Thermo-chemical convection (global)
  8. Compressible steady-state convection

Cookbook 1

Input file $install/examples/Cookbook1/cookbook1


CitcomS on anselm supercomputer

Anselm documentation:

Before compilation/execution, load modules which “sets up the GNU development environment in conjunction with the bullx MPI library”:

$ module load PrgEnv-gnu

I spent little time playing with various hdf5-related modules, but couldn’t get CitcomS to compile with hdf5.

Jobs on anselm should be run on multiples of 16 of cores. CitcomS needs multiples of 12.

In CitcomS input file:


script to send to the queue ($ qsub script):

#PBS -q qexp
#PBS -N CitcomS
#PBS -l select=3:ncpus=16:mpiprocs=16,walltime=00:10:00
#PBS -A OPEN-4-16
#PBS -m ae



# change to scratch directory, exit on failure
cd $scrdir || exit

# load modules
module load PrgEnv-gnu

# execute the calculation
mpiexec $program $inputfile &> $outputfile

# exit

Output files, post-processing, visualization

output_format=ascii (default) or ascii-gz or hdf5

output_optional="surf,botm,tracer" (default), also possible to include horiz_avg, geoid, seismic, stress, pressure, connectivity, heating, comp_el, comp_nd

Visualization: OpenDX, MayaVi2, GMT

If ASCII output


step total_t delta_t total_cpu_time step_cpu_time

PREFIX.domain ... binary file on rank 0 processor, contains domain bounds of processors

PREFIX.coord.CAP ... output at 0th time step, one header line, then lines with node coordinates:

PREFIX.QUANTITY.CORE.TIMESTEP (e.g., example0.velo.3.100):

*.chkpt.* ... binary files, for restart


topography heat_flux vel_colat vel_lon
*.velo.* ... two header lines, then lines with:
vel_colat vel_lon vel_r temperature
*.visc.* one header line, then lines with:
*.horiz_avg.* ... lines with:
radius temperature rms(v_horiz) rms(v_vert)


*.qb.dat ... lines with:
time heat_flux rms(v)

OpenDX visualization (of Cookbook 1)

Use in $install/visual to prepare data files for OpenDX from CitcomS output files (Manual section 5.2):

  • (Changed remote_shell to ssh from the default rsh in

  • Fixed output_optional_EL issue in

  • I couldn’t get it to work to collect datafiles from multiple machines: machinefile pidfile timestep
  • It worked to copy output files to one location ($runs/Cookbook1/visual), modify the pid file and run: localhost pidfile timestep
  • Creates files PREFIX.capXX.TIMESTEP and PREFIX.capXX.TIMESTEP.general

Use OpenDX script in $install/visual to plot temperature isosurface (Manual section 5.5). If you need to modify things substantially, have fun* tweaking the OpenDX script... (*sarcasm)

OpenDX visualization if output_format=ascii-gz

Straightforward but a bit tedious trick: Copy and modify selected output files to the format one would get with ascii output:

  • copy coord.* files and *.TIMESTEP files to a directory

  • prepend a prefix to filenames:

    $ for f in *.56000 ; do mv "$f" "m8b.$f" ; done
    $ for f in coord.* ; do mv "$f" "m8b.$f" ; done
  • and modify pid* file accordingly

  • follow steps for ascii visualization

Input file

CitcomS input files is an ascii file with lines of format parameter=value. Any text following # is a comment. Most parameters have default vaules set by the code, so a minimal input file may be used for some problems. An alternative practice is to use an input file with all parameter explicitly set. One can start with a pid* file generated by a test run and modify it into a desired input.

Few notes:

  • Some cookbook example input files suggest the line solver=full for full spherical shell models. This is unnecessary as nproc_surf=12 takes care of it. (Not to be confused with Solver=cgrad or multigrid for the momentum solver outer loop.)
  • The length of calculation is set by maxstep=1000. Setting of minstep=1 and maxtotstep=1000000 only plays for continued runs (as long as maxtotstep >= maxstep), I think.
  • Mesh input parameter dependence: \(nodex = 1 + nprocx \times mgunitx \times 2^{levels-1}\) and similarly for \(nodey\) and \(nodez\)
  • Mesh
    • coor=0 ... uniform mesh in lat, lon, r
    • coor=1 ... coordinates read from coor_file="coor.dat"
    • coor=2 ... uniform in lat, lon but refined in boundary layers, e.g., coor_refine=0.1,0.15,0.1,0.2 (see Manual, appendix A.1.4)
  • Default boundary conditions are impermeable, free-slip, uniform temperature.
  • Viscosity option in Manual, appendix A.1.11. Temperature-, pressure- and composition-dependent viscosity. Also stress dependence, pseudo-plastic rheology.
  • Set write_q_files=1 to output top and bottom heat flux time series files qt.dat and qb.dat.

To start a “new” calculation but read temperature from an “old” one:

zero_elapsed_time=on        # time counts from 0
tic_method=-1               # -1 .. read from *.velo.* files

To start a calculation from a supplied radial temperature profile (+perturbations):

Surprisingly, this initial condition on temperature was not implemented in the CIG version. I tweaked a function from my Boulder CitcomS version and included this option. You can check the source code in lib/Initial_temperature.c in my CitcomS installation in /nfsz1/sramek/CitcomS-3.3.1-work, search for osr which labels my modifications/additions (also little things is some other source and header files).

To use this initial condition, use this in input file:


T,P-dependent viscosity

The Arrhenius-type viscosity relation, in non-dimensional form and with \(\eta_\text{cmb}=1\) constraint, is

\[\eta = \eta_r(r) \exp \left[ \frac{E+V(1-r)}{T_s+T} - \frac{E+V(1-r_c)}{T_s+1} \right]\]
\(E=\frac{E^*}{R\Delta T}\) is non-dimensional activation energy,
\(V=\frac{\rho_0g_0R_0V^*}{R\Delta T}\) non-dimensional activation volume,
\(T_s=\frac{T_\text{surf}}{\Delta T}\),
and \(\eta_r(r_c)=1\).

Mini-script (in /nfs1/pc/install/citcoms) calculates the non-dimensional parameters.

A bit on data structure

struct All_variables *E; defined in global_defs.h contains all global data and is passed between functions.

For example E->monitor.solution_cycles is the current time step (integer).

  • E->fp ... a FILE pointer to *.log output
  • E->fptime ... *.time
  • E->fp_out ... *.info.* (if verbose=1)
  • E->fpqt ... *.qt.dat
  • E->fpqb ... *.qb.dat
  • stderr ... output to screen

Thermal convection in Martian mantle

Input file marstherm8, input parameters for case TC1 from Šrámek & Zhong 2010 JGR doi:10.1029/2010JE003597

Parallel efficiency

Testing 10000 steps of Cookbook 1 case on geofz1 and geofz2.

Number of cores 12 24
CPU time per velocity step [sec] 0.219 0.135
Wall clock time [sec] 2155 1370
Speed up relative to 12 cores 1 1.63/1.57


When using CIG CitcomS, asked to acknowledge CIG and to cite these papers (the third one when using tracers):

  • Zhong, S., M. T. Zuber, L. Moresi, and M. Gurnis, Role of temperature-dependent viscosity and surface plates in spherical shell models of mantle convection, J. Geophys. Res., 105(B5), 11,063–11,082, doi:10.1029/2000JB900003, 2000.
  • Tan, E., E. Choi, P. Thoutireddy, M. Gurnis, and M. Aivazis (2006), GeoFramework: Coupling multiple models of mantle convection within a computational framework, Geochem., Geophys., Geosyst. 7, Q06001, doi:10.1029/2005GC001155.
  • McNamara, A. K., and S. Zhong, Thermochemical structures within a spherical mantle: Superplumes or piles?, J. Geophys. Res., 109, B07402, doi:10.1029/2003JB002847, 2004.

I believe the presentation in this book inspired the early developers:

CitcomS 3-D spherical shell reference study:

  • Zhong, S., A. McNamara, E. Tan, L. Moresi, and M. Gurnis, A benchmark study on mantle convection in a 3-D spherical shell using CitcomS, Geochem. Geophys. Geosyst., 9, Q10017, doi:10.1029/2008GC002048, 2008.

C programming language:


  • Thermochemical convection calculation.
  • Check speed and parallel efficiency (again).

Notes for 4/8

  • Hughes book
  • Viscosity and
  • Boolean input: on/off vs. 1/0
  • Anselm supercomputer
  • OpenDX plotting

Láďa’s Ubuntu 14.04 update

[email of 8/30/2015]

CitcomS 3.3.1 je na všech geofech s Ubuntu 14.04. Instalace vyžadovala nějaké úpravy pro configure i v pythonovských skriptech, nyní pracuje i mezi stroji. OpenDX vykresluje ascii data, patrně i HDF data; HDF mezi více stroji chodit nemůže, nemáme potřebný “parallel filesystem”. GMT kreslí. Tvoje dokumentace by novou situaci mohla reflektovat v níže shrnutých bodech.

CitcomS na klastru KG

  • CitcomS je předinstalovaný v /opt/citcoms, aktivace cesty k němu: source load_citcoms
  • popis instalace:
    • export CFLAGS="-g -O2 -DH5_USE_16_API"; export CPPFLAGS="-DUSE_GZDIR -DH5_USE_16_API"
    • ./configure; make; make install prefix=/opt/citcoms
    • úprava /opt/citcoms/bin/
      • ř.37: nahradit 'heating' => ''
    • úprava /opt/citcoms/bin/
      • pod ř.83 přidat nový řádek: node=node.replace('\n','')
      • ř.95: nahradit => /opt/citcoms/bin/
      • (ř.93: rsh netřeba měnit na ssh, v Ubuntu je to už déle totéž)
    • úpravy v adresářích /opt/citcoms/share/CitcomS/visual a /opt/citcoms/share/CitcomS/visual/OpenDXMacro:
      • soubory f=*.net přejmenovat na *.net

Uniprocessor example

  • source load_citcoms
  • přepnout do pracovního adresáře
  • cp /opt/citcoms/share/CitcomS/examples/ex* .
  • nastavit v example0: maxstep=100 (nebo jinak)
  • CitcomSRegional example0

Single-machine multi-core example

  • source load_citcoms
  • přepnout do pracovního adresáře
  • cp /opt/citcoms/share/CitcomS/examples/ex* .
  • nastavit v example1: maxstep
  • mpirun -n 4 CitcomSRegional example1

Multi-machine example

  • source load_citcoms
  • ověřit bezheslový přístup mezi stroji
  • přepnout do sdíleného pracovního adresáře
  • cp /opt/citcoms/share/CitcomS/examples/ex* .
  • nastavit hosts: co řádek, to jméno stroje
  • nastavit v example1: maxstep, datadir, případně nprocx, nprocy, output_format
  • (OMPI_* proměnné nastaveny)
  • mpirun -n 4 -hostfile hosts /opt/citcoms/bin/CitcomSRegional example1

OpenDX visualization

  • (uniprocessor example, single-machine example)
  • localhost pidfile step [step ...]
  • př. localhost pid000020535 10
  • (multi-machine examples)
  • hosts pidfile step [step ...]
  • př. hosts pid000020590 10 20 30
  • pro snadnější výběr skriptu v OpenDX: ln -s /opt/citcoms/share/CitcomS/visual/
  • dx podle manuálu sekce 5.4