Run Specifications

General Information

   Purpose of Run: ensemble member 5 of rcp2.6 runs with extra output (MOAR)

   Scientific Lead: Jim Hurrell

   Software Engineering Lead: Mariana Vertenstein

   Assigned to: Andy
   Date: 2010-11-23

   Run Length: 96 years
Case Creation Information (all fields are required)

   CCSM tag:   ccsm4_0_beta53

   Case Name:  b40.rcp2_6.1deg.006

   Machine:    bluefire

   Compset:    B_RCP2.6_CN
   Resolution: f09_g16

Pre-Configuration Information

   env_conf.xml mods 
   xmlchange -file env_conf.xml -id RUN_TYPE -val 'hybrid'
   xmlchange -file env_conf.xml -id RUN_STARTDATE -val '2005-01-01'
   xmlchange -file env_conf.xml -id RUN_REFCASE -val 'b40.20th.track1.1deg.011'
   xmlchange -file env_conf.xml -id RUN_REFDATE -val '2005-01-01'
   xmlchange -file env_conf.xml -id CLM_CONFIG_OPTS -val '-bgc cn -snicar_frc on'

   env_mach_pes.xml mods
   component       comp_pes    root_pe   tasks  x threads (stride)
   ---------        ------     -------   ------   ------   ------  
   cpl = cpl        320         0        320    x 1       (1     )
   glc = sglc       1           0        1      x 1       (1     )
   lnd = clm        128         320      128    x 1       (1     )
   ice = cice       320         0        320    x 1       (1     )
   atm = cam        448         0        448    x 1       (1     )
   ocn = pop2       64          448      64     x 1       (1     )

Post-Configuration Information

   * Dave Lawrence / Brian Kauffman (cpl): 
       Turn on 3 hour auxiliary history files fields needed to as input to datm to drive clm.

       In addition to a one-line mod to the coupler main program (ccsm_driver.F90, see below).
       we need to change the value of one nml variable in cpl.buildnml.csh 
       This variable turns on 3 hour auxiiary history files...

       histaux_a2x3hr  = .true.

   * CAM (eaton) 
     - Data for offline radiation calculations and for budget analysis
       **for entire run** 
       The following needs to be added to cam.buildnml.csh

         rad_data_output=.true. rad_data_histfile_num=1

     - Higher frequency data
      **For entire 96 years (2005-2100)**
       This includes:
       - CMIP5 fields
       - TEM diagnostics requested by Yaga
       - 6-hrly instantaneous data requested by Mike Wehner

       The following variables need to be added to the existing cam_inparm
       namelist group in cam.buildnml.csh:

          'U','UV2d','UW2d','U2d','V','VTH2d', 'V2d','W2d','Z3'

     - Detailed budget diagnostic data requested by AMP
       **Only from 2075 to 2100**

       The following variables need to be added to the existing cam_inparm
       namelist group in cam.buildnml.csh:

        fincl6='CME', 'CMEICE', 'CMELIQ', 'CMFDICE', 'CMFDLIQ',
          'PTEQ', 'PTTEND',  'QRL', 'QRS', 'REPARTICE', 'REPARTLIQ',
          'VDCLDLIQ', 'ZMDICE', 'ZMDLF', 'ZMDLIQ', 'ZMDQ', 'ZMDT',
          'CLDICE:I', 'CLDLIQ:I','Q:I', 'T:I', 'CLDICEAP:I', 'CLDLIQAP:I','QAP:I', 'TAP:I'

   * Gary (clm)

     lnd daily fields    : SOILWATER_10CM[1] QRUNOFF[2] SNOWDP FSNO TSOI_10CM

     [1] - CLM says they can reduce SOILICE and SOILLIQ to the needed top
           0.1m integral, to avoid saving the full depth of these two.
     [2] - CLM says that they can save QRUNOFF (total runoff) instead of the three
           terms (QDRAI + QRGWL + QOVER) that are summed to create total runoff

   * Gary (cice)   

     ice daily fields[3] : aice hi uvel vvel

     [3] - If and only if the sea ice model can dump out daily data - as I
           understand it, it now can.

     More details are at



     * pop2 (questions? contact njn01)
        Set the ocean environment variables OCN_TAVG_TRACER_BUDGET 
        and OCN_TAVG_HIFREQ to TRUE to activate tracer budget terms
        and high-frequency ocean output. These variables are located
        at the end of the env_run.xml file:

             "OCN_TAVG_TRACER_BUDGET"   value="TRUE"  
             "OCN_TAVG_HIFREQ"          value="TRUE"  

   * Buildconf/cice.buildexe.csh        [add three defines to the cicedefs at the end of line 34]
set cicedefs = "... -DAEROFRC -DCCSM3FRC -DPONDFRC"
   * Buildconf/cice.buildnml.csh        [change 1st x to d in histfreq; replace icefields from /glade/data01/CMIP5/tmp/icefields_nml]
   * Buildconf/cpl.buildnml.csh         [set histaux_a2x3hr = .true. at line 108]
   * Buildconf/cam.buildnml.csh         [add the following four lines after line 126]
 history_budget = .true.                [add to phys_ctl_nl for whole run]
                                        [add the following to cam_inparm]
   'U','UV2d','UW2d','U2d','V','VTH2d', 'V2d','W2d','Z3'

   * Buildconf/clm.buildnml.csh         [replace 29,30 with the following lines]
 hist_mfilt             = 1,365
 hist_nhtfrq            = 0,-24
 hist_fincl2            = 'SOILWATER_10CM','QRUNOFF','SNOWDP','FSNO','TSOI_10CM'
                                        [check that clmdefs contains

  % xmlchange -file env_build.xml -id EXEROOT -val '/glade/data01/CMIP5/tmp/$CASE'
  % xmlchange -file env_run.xml -id DOUT_S_ROOT -val '/glade/data01/CMIP5/tmp/archive/$CASE'
  % xmlchange -file env_run.xml -id DOUT_L_MSROOT -val '/CCSM/csm/$CASE'
  % xmlchange -file env_run.xml -id DOUT_L_MS -val 'TRUE'
  % xmlchange -file env_run.xml -id DOUT_L_HTAR -val 'FALSE'
  % xmlchange -file env_run.xml -id REST_N -val '12'
  % xmlchange -file env_run.xml -id STOP_N -val '12'
  % xmlchange -file env_run.xml -id STOP_OPTION -val 'nmonths'
  % xmlchange -file env_run.xml -id OCN_TAVG_TRACER_BUDGET -val 'TRUE'
  % xmlchange -file env_run.xml -id OCN_TAVG_HIFREQ -val 'TRUE'

SourceMods Information

  * Dave Lawrence / Brian Kauffman (cpl): 
    Turn on 3 hour auxiliary history files fields needed to as input to datm to drive clm.

    In addition to changing the value of a nml variable (see above),
    we need a one-line source mod to the coupler main program (ccsm_driver.F90).
    This change changes the list of fields that go into the history file.

    Unfortunately, the list of fields for 3 hour history output is
    hardcoded to be atm states sent to the coupler, but no fluxes
    (because code was designed to address some other use-case).

    To get the desired fields, states AND fluxes, at the 3hr time interval 
    one character string must be altered in ccsm_driver.F90

character(CL) :: hist_a2x3hr_flds  = 'Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_dens:Sa_pbot:Sa_pslv:Faxa_lwdn'


character(CL) :: hist_a2x3hr_flds  = 'Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_dens:Sa_pbot:Sa_pslv:Faxa_lwdn:&

  * Keith Lindsay / Nancy Norton (ocn): 
    A patched version of mix_submeso.F90 to pass 5d DEBUG smoke test is required:

  * Jerry Olson (atm): 
    Change 'I' to 'A' in two source files:
  % cd /glade/proj2/fis/cgd/cseg/csm/runs/ccsm4_0/b40.rcp2_6.1deg.006/SourceMods/src.cam
  % cp /glade/proj2/fis/cgd/cseg/csm/collections/ccsm4_0_beta53/models/atm/cam/src/physics/cam/cam_diagnostics.F90 .
  % cp /glade/proj2/fis/cgd/cseg/csm/collections/ccsm4_0_beta53/models/atm/cam/src/physics/cam/stratiform.F90 .
  % vi cam_diagnostics.F90                
  % vi stratiform.F90                     
    This corrects the "instantaneous CAM output" problem that we experienced in the first two MOAR runs.

  * Dave Bailey (ice): 
    A patched version of ice_history.F90 to read the expanded list of output variables is required:

    Also need to add "-DAEROFRC -DCCSM3FRC -DPONDFRC" to cicedefs in Buildconf/cice.buildexe.csh

  % cd SourceMods/src.drv
  % cp /gpfs/proj2/fis/cgd/cseg/csm/collections/ccsm4_0_beta53/models/drv/driver/ccsm_driver.F90 .
  % vi ccsm_driver.F90                   [change line 253 to the following]
   character(CL) :: hist_a2x3hr_flds  = 'Sa_z:Sa_u:Sa_v:Sa_tbot:Sa_ptem:Sa_shum:Sa_dens:Sa_pbot:Sa_pslv:Faxa_lwdn:&
  % cd SourceMods/src.cice
  % cp /blhome/dbailey/ice_history.F90 . [deals with new variables on the icefields_nml in Buildconf/cice.buildnml.csh]
  % cp /ptmp/klindsay/ccsm4_0_beta54+/models/ocn/pop2/source/mix_submeso.F90 SourceMods/src.pop2

  * Mat Rothstein (l_archive):

  % vi b40.rcp2_6.1deg.006.bluefire.l_archive 
foreach dirl1 (`/bin/ls -1`)
    if (-d $DOUT_S_ROOT/${dirl1}) then
       $CASETOOLS/ccsm_l_archive.csh ${dirl1} &

  % cp /gpfs/proj2/fis/cgd/cseg/people/mvr/cesm1_0_beta06_dev/scripts/ccsm_utils/Tools/ccsm_l_archive.csh Tools
  % vi Tools/ccsm_l_archive.csh               

  if (${dirl1} == "cpl" && ${dirl2} == "hist") then

  if (${dirl1} == "ice" && ${dirl2} == "hist") then

  if (${dirl1} == "ocn" && ${dirl2} == "hist") then

  /blhome/mai/bin/ccsm_save_hist.csh $dirl1 $dirl2

Performance/Cost Estimates

  * 8.9 SYPD which is equivalent to
    24*8*32*1.4/8.9 = 966 bluefire GAUs per year which implies that the
    total cost for the run is ~92780 bluefire GAUs not including roughly
    6200 MSS GAUs. The estimate for MSS charges derived from Aug CISL GAU
    report for 93300471. Both of these estimates are based on first 45
    years of the run only. The last 26 years of the run had a sixth atm
    output stream. Hence, these estimates are too low.

Special Instructions
  * Run on GLADE:

  xmlchange -file env_build.xml -id EXEROOT -val '/glade/data01/CMIP5/tmp/$CASE'

  * Have the short-term archiver link output to CAVES:

  xmlchange -file env_run.xml -id DOUT_S_ROOT -val '/glade/data01/CMIP5/tmp/archive/$CASE'

  * Both the cpl and ice daily output is one day per file. To have these packaged
    up into monthly files, add calls to the following two scripts:


    to the long-term archiver.

  * To save the final history files in a directory parallel to the archive directory,
    add a call to the following script near the end of the long-term archiver:


Pre-Run Instructions

  * Run create_production_test
  * Run debug smoke test

  * Add NCAR Software Levels info to checklist 

Run Instructions

  Run Length: 96 years

  Account key: 93300415

  Priority/Targeted queue: regular

Diagnostics Plan

  * vs b40.20th.track1.1deg.011 (1986-2005) at 2041-2060 and 2081-2100

Additional Information

  * [put any additional information here]

Return to Top

Run Checklist

Complete the following checklist prior to beginning the production run:

1.  Update status file: /web/web-data/cseg/ccsm4_0_runs/b40.rcp8_5.1deg.006/status.html:

2.  Document NCAR software levels at beginning of run (use the spinfo command on bluefire)
NCAR SOFTWARE LEVELS: Tue Nov 23 15:07:17 MST 2010.
AIX:                  bos.mp    
CSM:                  csm.core  
LoadLeveler:          LoadL.full
GPFS:                 gpfs.base 
VSD:                  rsct.vsd.vsdd
POE:                  ppe.poe   
PESSL:                pessl.rte.smp
ESSL:                 essl.rte.smp
FORTRAN:              xlfrte    
PERL:                 perl.rte  
C:                    xlC.rte   

3.  Complete the following table, as necessary, showing
    the component liaison's name and the date the setup
    was approved.

   Component         Liaison/                     Date Approved

      atm            olson (bugfix only)      2010-11-23 15:01

      cpl            [kauff,mvertens,tcraig,other]    ----

      ice            dbailey                  2010-11-

      lnd            slevis                   2010-11-23 13:33:32

      ocn            njn01                    2010-11-23 14:02:28

      env_ file      [mvertens,other]                 ----

      data           strand                   2010-11-23 13:48:49

4.  Create_production_test completed   mai    2010-11-23 16:59:36

5.  Debug smoke test completed         mai    2010-11-23 18:58:55

6.  Performance review completed [who,when]

Return to Top


After this case run was completed, it was noticed that one of the ocean monthly
mean history files was corrupt (b40.rcp2_6.1deg.006.pop.h.2098-03.nc). Hence, we
needed to re-run model year 2098 with env_run.xml varibles DOUT_L_MS and

After the re-run of model year 2098 the following commands were done on bluefire:

  % cd /glade/proj3/cseg/runs/ccsm4_0/b40.rcp2_6.1deg.006
  % source ./Tools/ccsm_getenv
  % cd /glade/data01/CMIP5/tmp/archive/b40.rcp2_6.1deg.006/ocn/hist
  % /blhome/klindsay/bin/pop_hist_postprocess.csh

to get the post-run history files that we really wanted. Only
b40.rcp2_6.1deg.006.pop.h.2098-03.nc was written to the MSS, since
everything else was already correct. Commands to place the file in storage:

  % ln b40.rcp2_6.1deg.006.pop.h.2098-03.nc /glade/data01/CMIP5/csm/b40.rcp2_6.1deg.006/ocn/hist
  % msrcp -srcdelete -pe 3650 -wpwd MAI -class rel=ec -pr 93300471\

were issued from /glade/data01/CMIP5/tmp/archive/b40.rcp2_6.1deg.006/ocn/hist.

Return to Top