Restart Datasets

CLM3.0 produces ``restart'' files containing data necessary to describe the exact state of the model run when it was halted. Restart files allow CLM3.0 to be continued or branched to produce exactly the same bit-for-bit answers as if it had never stopped. Restart files contain instantaneous binary data for a set of time-dependent variables. The data is stored in binary format to ensure bit-for-bit agreement with a terminated run which was subsequently restarted. Restart files should be used to extend previously-started simulations or to branch from one simulation to another in which history file namelist variables have been modified. For a branch run, the user must change the simulation's case name unless the namelist variable BRNCH_RETAIN_CASENAME is set to .true.. A restart file also contains a version number to ensure that the code being used is compatible with the restart file being read. If the user modifies the contents of the restart file, the version number should be incremented.

The criterion for a ``successful restart'' is that the model has available exactly the same information upon restart that it would have had if it had not stopped. Restart files may need to be modified during model development to include new time-dependent variables. A simple rule is that if a run produces even slightly different answers when restarted compared to when left uninterrupted, then certain variables are missing from the restart file. Similarly, a restart should match a non-restarted run even if the number of MPI tasks and/or the number of OpenMP threads is changed upon restarting the run (as long as both runs are run on the same machine).

The module restFileMod (Section A.83) controls restart file logic. Restart files are written and read in routine restart (Section A.83.1) (in module restFileMod). When routine restart is called with the argument flag set to 'read' the model reads the restart dataset, whereas if the argument flag set to 'write', restart datasets are created. The actual reading and writing of restart files are carried out in the following routines:

• restart_setup (in restFileMod (Section A.83)) sets up restart files read/write and perform necessary consistency checks
• restart_time (in restFileMod (Section A.83)) reads/write time manager information to/from restart file
• restart_biogeophysics (in restFileMod (Section A.83)): reads/writes biogeophysics related restart data
• restart_coupler (in clm_csmMod) (Section A.42) reads/writes CLM restart data needed in csm mode (if CPP variable COUP_CSM is defined)
• restart_rtm (in RtmMod (Section A.26)) reads/writes RTM related restart data if CPP variable RTM is defined
• restart_dgvm (in DGVMRestMod) reads/writes DGVM related restart data if CPP variable DGVM is defined
• restart_accum (in accumulMod (Section A.38)) reads/write accumulation field restart data

If a user modifies the code such that it becomes necessary to add a new variable to the restart file, the user must first determine which of the above routines restart information must be added to. The user should then consult that routine for explicit examples of to how to add restart data.

We briefly summarize key points of writing and reading restart data using biogeophysics related restart variables as examples. The following example, taken out of restart_biogeophys illustrates how a single-level biogeophysics related variable defined at the PFT-level in clmtype (Section A.47) is written to and read from the restart file:

         ! PFT type physical state variable - fwet

         if (flag == 'read') call readin (nio, rbuf1dp, clmlevel=namep)
     !dir$ concurrent
     !cdir nodep
         do p = begp,endp
            if (flag == 'read' ) clm3%g%l%c%p%pps%fwet(p) = rbuf1dp(p)
            if (flag == 'write') rbuf1dp(p) = clm3%g%l%c%p%pps%fwet(p)
         end do
         if (flag == 'write') call wrtout (nio, rbuf1dp, clmlevel=namep)

Subroutines readin and wrtout reside in module iobinary (Section A.64) and correspond to overloaded module interface routines which read in and write out binary data. The array, rbuf1dp, is a temporary array allocated in restart_biogeophys (Section A.83.4) and is dimensioned as the total number of model PFTs. Analogous temporary arrays, rbuf1dg, rbuf1dl and rbuf1dc are also defined and dimensioned as the total number of model gridcells, landunits and columns, respectively. Finally, the argument corresponding to clmlevel can take on the values namep, namec, namel and nameg, (see module clmtype). The following provides a summary of calls to binary writes for PFT-level, column-level, landunit-level, and gridcell-level arrays:

         call wrtout (nio, rbuf1dp, clmlevel=namep)
         call wrtout (nio, rbuf1dc, clmlevel=namec)
         call wrtout (nio, rbuf1dl, clmlevel=namel)
         call wrtout (nio, rbuf1dg, clmlevel=nameg)

Similar calls also pertain to binary read calls.

Multi-level fields can also be written to and read from the restart files as illustrated below:

         ! Column physical state - z (snow)

         allocate (rbuf2dc(-nlevsno+1:0,numc))
         if (flag == 'read') call readin (nio, rbuf2dc, clmlevel=namec)
         do j = -nlevsno+1,0
     !dir$ concurrent
     !cdir nodep
            do c = begc,endc
               if (flag == 'read' ) clm3%g%l%c%cps%z(c,j) = rbuf2dc(j,c)
               if (flag == 'write') rbuf2dc(j,c) = clm3%g%l%c%cps%z(c,j)
            end do
         end do
         if (flag == 'write') call wrtout (nio, rbuf2dc, clmlevel=namec)
         deallocate(rbuf2dc)

It is important to note that when two-dimensional arrays are passed to routines readin or wrtout in iobinary (Section A.64), arrays must be dimensioned as array(level, subgrid_type).