The CLM3.0 model can be built to run in one of three modes. It can run as a stand alone executable where atmospheric forcing data is periodically read in (e.g., using the data in NCEPDATA). This will be referred to as offline mode. It can also be run as part of the Community Atmosphere Model (CAM) where communication between the atmospheric and land models occurs via subroutine calls. This will be referred to as cam mode. Finally, it can be run as a component in a system of geophysical models (CCSM). In this mode, the atmosphere, land, ocean and sea-ice models are run as separate executables that communicate with each other via the CCSM flux coupler. This will be referred to as ccsm mode.
CLM3.0 may be run serially (i.e., on a single processor), in parallel using the Message Passing Interface (MPI) for distributed memory tasks, in parallel using the Open Multi-Processing (OpenMP) directives for shared memory tasks, or finally in parallel using both MPI and OpenMP (hybrid parallelism). As an example, the IBM SP consists of distributed memory nodes interconnected by a high performance network connection, and each node contains multiple shared memory processors. When run on the IBM SP, CLM3.0 uses OpenMP directives for parallelism on processors within a shared memory node and MPI routines for parallelism across distributed memory nodes to take full advantage of the capabilities of the hardware. The supported architectures and associated compilers for running CLM3.0 in offline mode are shown in Table 3. Note that when DGVM is defined (see Table 6), bit-for-bit restarts are not supported on the CRAY X1 when processor configuration is changed mid-run.
| Hardware | Architecture | OS | Compiler |
| IBM SP | RS600 | AIX | IBM XL |
| SGI | MIPS | IRIX64 | MIPS |
| Intel | Intel X86 | Linux | pgf90, lf95 |
| CRAY X1 | Cray X1 | Unicos/mp | ftn |
| NEC SX6 | NEC SX6 | Super-UX | f90 |
| NEC SX6 | NEC SX6 | Super-UX | sxf90 (cross compiler) |
The method of building and running CLM3.0 depends on both the target architecture and whether the model will be run serially, in pure MPI mode (distributed memory), in pure OpenMP mode (shared memory) or in hybrid mode. A general discussion of the various aspects of building and running CLM3.0 follows.
Several cautionary notes have to be made concerning Linux systems. For Linux operating systems, CLM3.0 is supported for both lf95 and pgf90 compilers. We have noted problems in running CLM3.0 in hybrid mode using pgf90 compilers. Consequently, OpenMP threading has been turned off by default for Linux systems. (The user may turn it back on by setting SMP to TRUE in jobscript.csh (see below). We have also noted problems in bounds checking with pgf90 when running in debug mode (DEBUG = TRUE). As a result, the bounds checking in the Makefile has been temporarily disabled.
In order to build and run CLM3.0 in offline mode, a sample script, jobscript.csh, and a corresponding Makefile are provided in the bld/offline directory. In addition, two perl scripts mkDepends (used to generate dependencies in a form suitable for inclusion into a Makefile), and mkSrcfiles (used to make a list of files containing source code) are also included. The user need not worry about these perl scripts.
The script, jobscript.csh, creates a model executable at T42 model resolution with RTM (river transport model) activated and CLM-DGVM and VOCs deactivated, determines the necessary input datasets, constructs the input model namelist and runs the model for one day. Users must edit this script appropriately in order to build and run the executable for their particular requirements and in their particular environment. This script is provided only as an example to aid the novice user in getting CLM3.0 up and running as quickly as possible.
The script can be run with minimal user modification, assuming the user resets several environment variables at the top of the script. In particular, the user must set the following:
The script can be divided into five functional sections: 1) specification of script environment variables; 2) creation of the model input namelist; 3) creation of two header files (misc.h and preproc.h) and a directory search path file (Filepath) needed to build the model executable; 4) creation of the model executable; and 5) execution of the model. Jobscript.csh is set up so that the user will normally only have to modify sections 1) and 2) in order to obtain a model executable and associated namelist. Each of these functional sections is discussed in what follows.
Table 4 lists the user modifiable script environment variables. Some of these variables are used by the Makefile to build the model executable. Although the script provides tentative settings for all these variables, the provided values will generally have to be modified by the user.
| Script Variable | Description |
| LSMLON | Number of model grid longitudes. |
| LSMLAT | Number of model grid latitudes. |
| ROOTDIR | Full pathname for the root source code directory. |
| MODEL_EXEDIR | Full pathname for the directory where the model executable will reside. |
| Object files will be built in the directory $MODEL_EXEDIR/obj. | |
| CSMDATA | Full pathname of root input datasets directory. |
| SMP | Flag that turns on OpenMP (valid values are TRUE or FALSE) |
| If SMP is TRUE, OpenMp is enabled and NTHREADS is the number of OpenMP threads. If SMP is FALSE, OpenMp is disabled and NTHREADS is ignored. Currently, SMP will be set to FALSE for Linux platforms. | |
| NTHREADS | Number of OpenMP multitasking threads. |
| NTHREADS should not exceed the number of physical CPUs (ie, processors) on a shared memory machine and should not exceed the number of CPUs in a node on a distributed memory machine. This setting is ignored if SMP is set to FALSE. | |
| SPMD | Flag that turns on MPI (valid values are TRUE or FALSE). |
| If SPMD is TRUE, MPI is enabled and NTASKS is the number of MPI tasks. If SPMD is FALSE, MPI is disabled and NTASKS is ignored. | |
| NTASKS | Number of MPI tasks. |
| Setting is ignored if SPMD is set to FALSE. NTASKS should not exceed the number of physical CPUs (ie, processors) on a machine. | |
| LIB_NETCDF | Full pathname of directory containing the netCDF library. |
| The setting depends on user's target machine. | |
| INC_NETCDF | Full pathname of directory containing netCDF include files. |
| The setting depends on user's target machine. | |
| LIB_MPI | Full pathname of directory containing the MPI library. |
| The setting depends on user's target machine. Only needed if SPMD is set to TRUE. Not needed on IBM SP, Cray X1, NEC SX6 machines, where the MPI library and include files are obtained directly from the compiler command. | |
| INC_MPI | Full pathname for directory containing the MPI include files. |
| The setting depends on user's target machine. Only needed if SPMD is set to TRUE. Not needed on IBM SP, Cray X1, NEC SX6 machines. | |
| linux_fort_compiler | Determines which linux fortran compiler is used. |
| Valid values are pgf90 or lf95. Only used on linux platforms. | |
| linux_machine | Name of linux machine. |
| Currently NCAR-CGD linux machine "anchorage" is supported in the scripts. The user will need to tailor this for their own platform. Only used on linux platforms. | |
| linux_mpirun_cmnd | The full pathname of mpirun to use if $SPMD is set to TRUE. |
| Multiple fortran compilers often exist on a single linux system. Each compiler can have a unique path where the mpich mpirun binary is installed for that compiler. Only used on linux platforms. | |
| linux_batch | Determines if run will be in batch or interactive mode. |
| Only used on linux platforms. | |
| DEBUG | Flag that turns on debugging in Makefile. Valid values are TRUE or FALSE. |
It is assumed that the user has set up the environment for their particular compiler choice (e.g. LD_LIBRARY_PATH is set correctly in the user's environment).
Before building and running the model with jobscript.csh, the user must specify model namelist variables via the CLM3 namelist, clmexp, found in jobscript.csh. Without modification, jobscript.csh generates a default namelist file, lnd.stdin, which results in a one day model run using the provided datasets. Namelist variables can be divided into several categories: run definitions, datasets, history and restart file settings and land model physics settings. A full discussion of model namelist variables is given in section 4.
The user will generally not need to modify the section of jobscript.csh that creates the header and directory search path files. The script creates three files in the directory $MODEL_EXEDIR/obj: the header files misc.h and preproc.h and the directory search path file, Filepath. To modify these files, the user should edit the file contents from within the script rather than attempt to edit the files directly, since the script will overwrite the files upon execution. The use of these files by gnumake is discussed below. Each of these files is summarized below.
The file, misc.h, contains resolution- and model-independent C-language pre-processor (cpp) tokens.
| misc.h cpp token | Description |
| SPMD | If defined, enables distributed memory, SPMD (single program multiple data), implementation. Automatically defined by script if environment variable SPMD is set to TRUE. See section 3.1.1. |
| PERGRO | If defined, enables modifications that test reasonable perturbation error growth. Not supported at this time. |
The file preproc.h contains resolution-dependent and model-dependent C-language cpp tokens.
| preproc.h cpp token | Description |
| OFFLINE | If defined, offline mode is invoked |
| COUP_CSM | If defined, ccsm mode is invoked |
| COUP_CAM | If defined, cam mode is invoked |
| SCAM | If defined, cam single column mode is invoked |
| This is only applicable if COUP_CAM is ALSO defined | |
| LSMLON | Number of model longitudes |
| LSMLAT | Number of model latitudes |
| RTM | If defined, RTM river routing is invoked |
| VOC | If defined, voc emission is computed |
| DGVM | If defined, dynamic vegetation model is activated |
| NOCOMPETE | If defined, competition for water is turned off (each pft has its own column) |
| This mode is no longer officially supported |
C-preprocessor directives of the form #include, #if defined, etc., are used in the model source code to enhance code portability and allow for the implementation of distinct blocks of functionality (such as incorporation of different modes) within a single file. Header files, such as misc.h and preproc.h, are included with #include statements within the source code. When gnumake is invoked, the C preprocessor includes or excludes blocks of code depending on which cpp tokens have been defined. C-preprocessor directives are also used to perform textual substitution for resolution-specific parameters in the code. The format of these tokens follows standard cpp protocol in that they are all uppercase versions of the Fortran variables, which they define. Thus, a code statement like
Filepath contains a list of directories used by Makefile to resolve the location of source files and to determine dependencies. The search begins in the current directory and proceeds to each directory appearing in Filepath, in the order in which they are specified. All files appearing in these directories will be used unless duplicate files are found. For the case of duplicate files, the first file found will be used by gnumake to create the object file. If user-modified code is introduced, Filepath should contain, as the first entry, the directory containing such modified code.
Users can add new search directories by editing jobscript.csh under ``build Filepath''. The default Filepath directory hierarchy for CLM3.0 is as follows:
| Source Directories | Functionality |
| $MODEL_SRCDIR/main | control routines (history, restart, etc) |
| $MODEL_SRCDIR/biogeophys | biogeophysics routines |
| $MODEL_SRCDIR/biogeochem | ecosystem and biogeochemistry routines |
| $MODEL_SRCDIR/riverroute | river routing routines |
| $MODEL_SRCDIR/csm_share/shr | code shared by all CCSM geophysical model components |
| $MODEL_SRCDIR/utils/timing | timing routines |
| $MODEL_SRCDIR/mksrfdata | generation of surface dataset routines |
The user will generally not need to modify the section of jobscript.csh that builds the model executable. Jobscript.csh invokes gnumake to generate the model executable. The file, Makefile, located in the bld/offline directory, contains commands used by gnumake on each of the supported target architectures. Successful invocation of gnumake results in an executable, "clm", along with a log file, "compile_log.clm", documenting the build procedure. Any problems encountered during the build procedure will be documented in this log file. A parallel gnumake is achieved in the script by invoking gnumake with the -j option, which specifies the number of job commands to run in parallel.
The user will generally not need to modify the section of jobscript.csh that runs the model executable. Jobscript.csh will execute the commands required to run the model under the supported target architectures. Most MPI implementations provide a startup script which accepts the MPI executable as a command line argument. Additional command line arguments allow the user to specify details such as the various machine architectures or number of processes to use for the run. Once MPI has created the specified number of processes, model execution will begin. The collection of active tasks will then compute locally and exchange messages with each other to integrate the model.
Upon successful completion of the model run, several files will be generated in MODEL_EXEDIR. These include history, restart, and initialization files (see section 4.3) as well as log files documenting the model execution. These log files will have names of clm.log.YYMMDD-HHMMSS, where YY is the last two digits of the current model year, MM is the month, DD is the day of the month, HH is the hour, MM is the minutes, and SS is the seconds of the start of the model run. Timing files, (e.g. ``timing.0''), containing model performance statistics are also generated in the executable directory.
When running the model as part of the CAM executable, CAM build and run scripts must be utilized. The user should refer to User's Guide ( to the NCAR Community Atmosphere Model 3.0 (CAM 3.0) at
www.ccsm.ucar.edu/models/ccsm3.0/cam/
for specific details on building and running the CAM executable. We will only discuss some essential points of the CAM build and run scripts here.
The header files, preproc.h and misc.h, as well as the directory search path file, Filepath, are needed for the CAM build procedure in an analogous manner to the CLM3.0 build procedure. The user should keep in mind that the CLM3.0 directory hierarchy MUST appear after the CAM directory hierarchy in Filepath. CLM3.0 contains several files that have the same name as the corresponding CAM files (e.g. time_manager.F90). When running in CAM mode, the corresponding CAM file must be used. The CAM build and run scripts ensure this occurs.
The CLM3.0 namelist, clmexp, must also be specified. By default, RTM river routing is not enabled in cam mode (i.e. the cpp variable, RTM, is not defined). Furthermore, CLM3.0 does not permit the user to independently set several namelist variables (in particular, those dealing with history file logic and run control logic) when running in cam mode. CLM3.0 will override any user input for these variables with the corresponding values used by the CAM model. This is discussed in more detail in section 4.6.
CCSM3.0 will contain CLM3.0. In ccsm mode RTM is defined by default, because this provides the fresh water flux from the land to the ocean model. We refer the reader to the CCSM3.0 User's Guide for further details on running in ccsm mode.