Run Specifications

General Information

   Purpose of Run: ensemble member 5 of rcp6.0 runs with extra output (MOAR)

   Scientific Lead: Jim Hurrell

   Software Engineering Lead: Mariana Vertenstein

   Assigned to: Andy
   Date: 2011-04-13

   Run Length: 96 years
Case Creation Information (all fields are required)

   CCSM tag:   cesm1_0_beta10

   Case Name:  b40.rcp6_0.1deg.006

   Machine:    bluefire

   Compset:    B_RCP6.0_CN
   Resolution: f09_g16

Pre-Configuration Information

   env_conf.xml mods 
   xmlchange -file env_conf.xml -id RUN_TYPE -val 'hybrid'
   xmlchange -file env_conf.xml -id RUN_STARTDATE -val '2005-01-01'
   xmlchange -file env_conf.xml -id RUN_REFCASE -val 'b40.20th.track1.1deg.012'
   xmlchange -file env_conf.xml -id RUN_REFDATE -val '2005-01-01'
   xmlchange -file env_conf.xml -id CLM_CONFIG_OPTS -val '-bgc cn -snicar_frc on'

   env_mach_pes.xml mods
   component       comp_pes    root_pe   tasks  x threads (stride)
   ---------        ------     -------   ------   ------   ------  
   cpl = cpl        320         0        320    x 1       (1     )
   glc = sglc       1           0        1      x 1       (1     )
   lnd = clm        128         320      128    x 1       (1     )
   ice = cice       320         0        320    x 1       (1     )
   atm = cam        448         0        448    x 1       (1     )
   ocn = pop2       64          448      64     x 1       (1     )

Post-Configuration Information

   * Dave Lawrence / Brian Kauffman (cpl): 
       Turn on 3 hour auxiiary history files fields needed to as input to datm to drive clm.

       In addition to a one-line mod to the coupler main program (ccsm_driver.F90, see below).
       we need to change the value of one nml variable in cpl.buildnml.csh 
       This variable turns on 3 hour auxiiary history files...

       histaux_a2x3hr  = .true.

   * CAM (eaton) 
     - Data for offline radiation calculations and for budget analysis
       **for entire run** 
       The following needs to be added to cam.buildnml.csh

         rad_data_output=.true. rad_data_histfile_num=1

     - Higher frequency data
      **For entire 96 years (2005-2100)**
       This includes:
       - CMIP5 fields
       - TEM diagnostics requested by Yaga
       - 6-hrly instantaneous data requested by Mike Wehner

       The following variables need to be added to the existing cam_inparm
       namelist group in cam.buildnml.csh:

          'U','UV2d','UW2d','U2d','V','VTH2d', 'V2d','W2d','Z3'

       Note that this MOAR RCP case is using COSP and not ISCCP, unlike previous
       MOAR RCP cases.

     - Detailed budget diagnostic data requested by AMP
       **Only from 2075 to 2100**

       The following variables need to be added to the existing cam_inparm
       namelist group in cam.buildnml.csh:

        fincl6='CME', 'CMEICE', 'CMELIQ', 'CMFDICE', 'CMFDLIQ',
          'PTEQ', 'PTTEND',  'QRL', 'QRS', 'REPARTICE', 'REPARTLIQ',
          'VDCLDLIQ', 'ZMDICE', 'ZMDLF', 'ZMDLIQ', 'ZMDQ', 'ZMDT',
          'CLDICE:I', 'CLDLIQ:I','Q:I', 'T:I', 'CLDICEAP:I', 'CLDLIQAP:I','QAP:I', 'TAP:I'

   * Gary (clm)

     lnd daily fields    : SOILWATER_10CM[1] QRUNOFF[2] SNOWDP FSNO TSOI_10CM

     [1] - CLM says they can reduce SOILICE and SOILLIQ to the needed top
           0.1m integral, to avoid saving the full depth of these two.
     [2] - CLM says that they can save QRUNOFF (total runoff) instead of the three
           terms (QDRAI + QRGWL + QOVER) that are summed to create total runoff

   * Gary (cice)   

     ice daily fields[3] : aice hi uvel vvel

     [3] - If and only if the sea ice model can dump out daily data - as I
           understand it, it now can.

     More details are at



     * pop2 (questions? contact njn01)
        Set the ocean environment variables OCN_TAVG_TRACER_BUDGET 
        and OCN_TAVG_HIFREQ to TRUE to activate tracer budget terms
        and high-frequency ocean output. These variables are located
        at the end of the env_run.xml file:

             "OCN_TAVG_TRACER_BUDGET"   value="TRUE"  
             "OCN_TAVG_HIFREQ"          value="TRUE"  

   * Buildconf/cice.buildexe.csh        [add three defines to the cicedefs at the end of line 34]
set cicedefs = "... -DAEROFRC -DCCSM3FRC -DPONDFRC"
   * Buildconf/cice.buildnml.csh        [change 1st x to d in histfreq;
                                         replace icefields from /glade/data01/CMIP5/tmp/icefields_nml]
   * Buildconf/cpl.buildnml.csh         [set histaux_a2x3hr = .true. at line 110]
   * Buildconf/cam.buildnml.csh         [add the following four lines after line 121]
 history_budget = .true.                [add to phys_ctl_nl for whole run]
                                        [add the following to cam_inparm]
   'U','UV2d','UW2d','U2d','V','VTH2d', 'V2d','W2d','Z3'

   * Buildconf/clm.buildnml.csh         [replace 29,30 with the following lines]
 hist_mfilt             = 1,365
 hist_nhtfrq            = 0,-24
 hist_fincl2            = 'SOILWATER_10CM','QRUNOFF','SNOWDP','FSNO','TSOI_10CM'
                                        [check that clmdefs contains

  % xmlchange -file env_build.xml -id EXEROOT -val '/glade/data01/CMIP5/tmp/$CASE'
  % xmlchange -file env_run.xml -id DOUT_S_ROOT -val '/glade/data01/CMIP5/tmp/archive/$CASE'
  % xmlchange -file env_run.xml -id DOUT_L_MSROOT -val '/CCSM/csm/$CASE'
  % xmlchange -file env_run.xml -id DOUT_L_MS -val 'TRUE'
  % xmlchange -file env_run.xml -id DOUT_L_HTAR -val 'FALSE'
  % xmlchange -file env_run.xml -id REST_N -val '12'
  % xmlchange -file env_run.xml -id STOP_N -val '12'
  % xmlchange -file env_run.xml -id STOP_OPTION -val 'nmonths'
  % xmlchange -file env_run.xml -id OCN_TAVG_TRACER_BUDGET -val 'TRUE'
  % xmlchange -file env_run.xml -id OCN_TAVG_HIFREQ -val 'TRUE'

SourceMods Information

  * Dave Lawrence / Brian Kauffman (cpl): 
    Turn on 3 hour auxiliary history files fields needed to as input to datm to drive clm.

    In addition to changing the value of a nml variable (see above),
    we need a one-line source mod to the coupler main program (ccsm_driver.F90).
    This change changes the list of fields that go into the history file.

    Unfortunately, the list of fields for 3 hour history output is
    hardcoded to be atm states sent to the coupler, but no fluxes
    (because code was designed to address some other use-case).

    To get the desired fields, states AND fluxes, at the 3hr time interval 
    one character string must be altered in ccsm_driver.F90

character(CL) :: hist_a2x3hr_flds  = 'Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_dens:Sa_pbot:Sa_pslv:Faxa_lwdn'


character(CL) :: hist_a2x3hr_flds  = 'Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_dens:Sa_pbot:Sa_pslv:Faxa_lwdn:&

  * Nancy Norton (ocn): 
    The following modified source files are required:

  * Dave Bailey (ice): 
    A patched version of ice_history.F90 to read the expanded list of output variables is required:

    Also need to add "-DAEROFRC -DCCSM3FRC -DPONDFRC" to cicedefs in Buildconf/cice.buildexe.csh

  % cd SourceMods/src.drv
  % cp /glade/proj2/fis/cgd/cseg/csm/collections/ccsm4_0_beta53/models/drv/driver/ccsm_driver.F90 .
  % vi ccsm_driver.F90                   [change line 253 to the following]
   character(CL) :: hist_a2x3hr_flds  = 'Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_dens:Sa_pbot:Sa_pslv:Faxa_lwdn:&
  % cd SourceMods/src.cice
  % cp /blhome/dbailey/ice_history.F90 . [deals with new variables on the icefields_nml in Buildconf/cice.buildnml.csh]

  % vi Tools/ccsm_l_archive.csh

  rm lines 37-47
  rm three msmkdir calls
  replace line 217 with:
	hsi -q -a 93300606 "cd ${lsmdir} ; put -d $file"

  if (${dirl1} == "cpl" && ${dirl2} == "hist") then

  if (${dirl1} == "ice" && ${dirl2} == "hist") then

  if (${dirl1} == "ocn" && ${dirl2} == "hist") then

  /blhome/mai/bin/ccsm_save_hist.csh $dirl1 $dirl2

Performance/Cost Estimates

  * 8.9 SYPD / 103 kGAUs

Special Instructions
  * Run on GLADE:

  xmlchange -file env_build.xml -id EXEROOT -val '/glade/data01/CMIP5/tmp/$CASE'

  * Have the short-term archiver link output to CAVES:

  xmlchange -file env_run.xml -id DOUT_S_ROOT -val '/glade/data01/CMIP5/tmp/archive/$CASE'

  * Both the cpl and ice daily output is one day per file. To have these packaged
    up into monthly files, add calls to the following two scripts:


    to the long-term archiver.

  * To save the final history files in a directory parallel to the archive directory,
    add a call to the following script near the end of the long-term archiver:


Pre-Run Instructions

  * Run create_production_test
  * Run debug smoke test

  * Add NCAR Software Levels info to checklist 

Run Instructions

  Run Length: 96 years

  Account key:  93300606

  Priority/Targeted queue: regular

Diagnostics Plan

  * vs b40.20th.track1.1deg.012 (1986-2005) at 2041-2060 and 2081-2100

Additional Information

  * [put any additional information here]

Return to Top

Run Checklist

Complete the following checklist prior to beginning the production run:

1.  Update status file: /web/web-data/cseg/ccsm4_0_runs/b40.rcp8_5.1deg.007/status.html:

2.  Document NCAR software levels at beginning of run (use the spinfo command on bluefire)
NCAR SOFTWARE LEVELS: Thu Apr 14 07:51:03 MDT 2011.
AIX:                  bos.mp    
CSM:                  csm.core  
LoadLeveler:          LoadL.full
GPFS:                 gpfs.base 
VSD:                  rsct.vsd.vsdd
POE:                  ppe.poe   
PESSL:                pessl.rte.smp
ESSL:                 essl.rte.smp
FORTRAN:              xlfrte    
PERL:                 perl.rte  
C:                    xlC.rte   

3.  Complete the following table, as necessary, showing
    the component liaison's name and the date the setup
    was approved.

   Component         Liaison/                     Date Approved

      atm            rneale                   2011-04-25 14:26:10

      cpl            [kauff,mvertens,tcraig,other]    ----

      ice            dbailey                  2011-04-13 15:58:46

      lnd            slevis                   2011-04-13 16:23:10
                     erik                     2011-04-14 10:44:02

      ocn            njn01                    2011-04-13 16:24:28
                     bates                    2011-04-14 15:23:17

      env_ file      [mvertens,other]                 ----

      data           strand                   2011-04-13 16:46:09

4.  Create_production_test completed   mai    2011-04-14 10:17:31

5.  Debug smoke test completed         mai    2011-04-14 19:49:24

6.  Performance review completed [who,when]

Return to Top


On 8 May 2011 the run died during model date Oct 2100 (less than three
months from the finish of the run) with the following error from the
lnd.log.110508-143926 file:

 clm2: completed timestep  1678961
(shr_tInterp_getFactors)  ERROR illegal linear times:             -NaNQ       0.00000000            -NaNQ
(shr_sys_abort) ERROR: (shr_tInterp_getFactors)  illegal itimes
(shr_sys_abort) WARNING: calling shr_mpi_abort() and stopping

This is the same problem that plagued cases b40.rcp6_0.1deg.001 and 002 and
occurred again in this case run because one of the source mods was left out:

  % cd SourceMods/src.share
  % cp /glade/proj2/fis/cgd/cseg/csm/runs/cesm1_0/b40.rcp6_0.1deg.002/SourceMods/src.share/shr_tInterp_mod.F90 .

The attempt to re-build with this included (as was done in the 001,002 cases)
failed. Hence, the case and run directories were save as:

Case dir : /glade/proj3/cseg/runs/cesm1_0/b40.rcp6_0.1deg.006_2075-2099
Run  dir : /glade/data01/CMIP5/tmp/b40.rcp6_0.1deg.006_2075-2099

and the case was re-built from scratch. This succeeded in finishing the last
year, model year 2100.

Return to Top