Plan for Reducing CCSM Data Storage Charges
http://www.cgd.ucar.edu/cseg/data/management/plan.html 15 Dec 2005
CCSM Software Engineering Group              [an error occurred while processing this directive] Up One Level | CSEG Internal Home | CSEG Public Home | CCSM Home

Plan for Reducing CCSM Data Storage Charges

Charges for storage of CCSM data files on the NCAR Mass Storage System (MSS) have reached a significant level and need to be reduced. Presently, storage charges for the data files in the directory /CCSM/csm are nearly 6800 GAUs/month, and charges for the b30.* cases total about 5000 GAUs/month.

When a request is made to write a data file to the MSS, by default two copies are made and stored. Last year, the Scientific Computing Division introduced an option called "Class of Service" which allows users to specify that only one copy be created. This option can be specified at the time the data file is created, or applied retroactively to any MSS data file. So a very obvious means of reducing the CCSM data-storage charges would be to eliminate duplicate copies.

Duplicate copies of MSS files are made for reliability reasons. Data files can become corrupted, and tapes can be broken or otherwise become physically damaged. SCD does not keep detailed records of how often a single data file becomes unusable, but anecdotal reports suggest that the loss rate is small. For further reading, see the SCD document Class of Service -- Guidelines for Users.

Another obvious means of reducing data-file storage charges is to eliminate the data file entirely, and in some cases, this may actually make sense. Several experimental cases, run in the "tuning" phase prior to the CCSM3 production runs, contained errors or were otherwise deemed unusable. The identification of these cases, however, is a labor-intensive process and may or may not be worth the effort.

The following is a prioritized list of specific proposals intended to significantly reduce the cost of MSS charges. These proposals have been approved by the SSC Chairperson.

  1. Modify CCSM3 scripts to request, as a default option, the storage of a single copy of all data files
  2. Approved Action: To reduce the MSS storage charges incurred by future CCSM runs, the single-copy Class of Service option should be made the default in CCSM scripts and models.

  3. Retain only a single copy of all data files created from Earth-Simulator runs, because backups exist off-site, at CRIEPI.
  4. Savings: The elimination of duplicate copies of the T85_gx1v3 Earth- Simulator data files will save an estimated 900 GAUs/month, based upon charges of 1800 GAUs/month in October 2005.

    Approved Action: Eliminate duplicate copies of all files in /CCSM/csm/$case, where $case is:

                    b30.020.ES02
                    b30.045b.ES01
                    b30.044b.ES01
                    b30.045f.ES01
                    b30.045g.ES01
                    b30.044g.ES01
                    b30.044f.ES01
                    b30.025b.ES01
                    b30.025a.ES01
                    b30.047b.ES01
                    b30.030b.ES01
                    b30.020.ES01
                    b30.030f.ES01
                    b30.030g.ES01
                    b30.026.ES01
                    b30.040b.ES01
                    b30.025.ES01
                    b30.026a.ES01
                    b30.030h.ES01
                    b30.041f.ES01
                    b30.041g.ES01
                    b30.041b.ES01
                    b30.040g.ES01
                    b30.040f.ES01
                    b30.100.ES01*
                    b30.047g.ES01
                    b30.046f.ES01
                    b30.046g.ES01
                    b30.046b.ES01
                    b30.047f.ES01
                    b30.009.ES03
                    b30.004a.ES01*
                    b30.004.ES01
                    b30.004b.ES01*
                    b30.036b.ES01
                    b30.036g.ES01
                    b30.036f.ES01
    
            
    * These cases have no backups in Japan, but the owners have confirmed that it is ok to keep only single copies at NCAR.

  5. Retain only a single copy of all data files from the /CCSM/csm/b30.* cases, except keep duplicates of the (non-ES) IPCC control runs
  6. Many of the CCSM/csm/b30.* cases were configured at the T85/gx1v3 resolution and were often run for hundreds of model years, generating a large volume of data. Because of the very large amounts of data, and the correspondingly large data-storage charges, these cases are prime candidates for single-copy storage.

    However, several of the b30.* cases are central to the CCSM IPCC contribution, and special consideration needs to be given to the issue of file retention for these cases, in particular: the T85/gx1v3 cases b30.009 (1990 control) and b30.026b (4x CO2); the T42/gx1v3 cases b30.004 (1990 control) and b30.043 (1870 control); and the T32/gx3v5 cases b30.031, b30.032, b30.032a, b30.032b, and b30.048.

    Data-storage reduction options considered are:

    Savings: A rough estimate of the number of GAUs/month the elimination of duplicate copies of all of the non-Earth Simulator b30.* files will save is based upon the following:

         
             ES b30.* files = 36% of all b30.* files  
         Non-ES b30.* files = 64% of all b30.* files
         
             ES b30.* files = cost ~1800 GAUs/month ==> all b30.* files cost ~5000GAUs/mo
         Non-ES b30.* files = cost ~3200 GAUs/month
    

    Eliminating duplicate copies of all non-Earth-Simulator b30.* files would save approximately 1600 GAUs/month; however, because the exact ratio of history/total files is variable among cases, an accurate estimate of the savings gained by eliminating duplicate copies of all but history files is not easily made at this time.

    At 6% of the total b30.* cases, b30.030e is the largest. This run generated a large volume of 6-hour atm history files; its data volume breaks down as follows (source: msdu -k)

              9496440743 /CCSM/csm/b30.030e           (total)
              7772500871 /CCSM/csm/b30.030e/atm
               105993369 /CCSM/csm/b30.030e/atm/rest
              4263725166 /CCSM/csm/b30.030e/atm/hist
                18311092 /CCSM/csm/b30.030e/cpl
                18306953 /CCSM/csm/b30.030e/cpl/rest
               152075533 /CCSM/csm/b30.030e/ice
               687904544 /CCSM/csm/b30.030e/lnd
               839827294 /CCSM/csm/b30.030e/ocn
               603027418 /CCSM/csm/b30.030e/ocn/hist
               236657116 /CCSM/csm/b30.030e/ocn/rest
    

    In this example, the atm history files account for 45% of the total data volume. If a similar percentage of the data volume from the remaining b30.* cases is due to history files, then our estimated savings will fall far short of 1600 GAUs/month, unless we decide to eliminate all duplicate files, including history files.

    Approved Action: Keep duplicate copies of IPCC experiments (b30.004, b30.009, b30.026b, b30.031, b30.032, b30.032a, b30.032b, b30.043, and b30.048) for one more year, then reconsider. Eliminate duplicate copies of all other files in /CCSM/csm/$case, where case is:

         (Sorted in descending order by size)
                    b30.030e
                    b30.036e
                    b30.040e
                    b30.101
                    b30.020
                    b30.030c
                    b30.100.02
                    b30.030a
                    b30.013
                    b30.021
                    b30.036d
                    b30.041b
                    b30.044d
                    b30.045b
                    b30.044e
                    b30.044c
                    b30.025
                    b30.040d
                    b30.036a
                    b30.040b
                    b30.044b
                    b30.040c
                    b30.036b
                    b30.036c
                    b30.051d
                    b30.051e
                    b30.051b
                    b30.120b
                    b30.042d
                    b30.002
                    b30.026
                    b30.010
                    b30.120c
                    b30.051a
                    b30.040w
                    b30.014
                    b30.018
                    b30.120a
                    b30.011
                    b30.001
                    b30.100
                    b30.003
                    b30.012
                    b30.006
                    b30.017
                    b30.105
                    b30.025a
                    b30.015
                    b30.005
                    b30.042c
                    b30.024
                    b30.020.scl
                    b30.009a
                    b30.020a
                    b30.042b
                    b30.051a.slevis
                    b30.009b
                    b30.014b
                    b30.042a
                    b30.033b
                    b30.033c
                    b30.034b
                    b30.042e
                    b30.033d
                    b30.030b
                    b30.120a_nosolar
                    b30.023
                    b30.061
                    b30.040g
                    b30.025b
                    b30.038b
                    b30.037d
                    b30.040a
                    b30.007
                    b30.031yr310
                    b30.019
                    b30.057
                    b30.058
                    b30.059
                    b30.018a
                    b30.030d
                    b30.1870o3
                    b30.037z
                    b30.049
                    b30.022
                    b30.041e
                    b30.041d
                    b30.041a
                    b30.037b
                    b30.037c
                    b30.041g
                    b30.041c
                    b30.020.rad
                    b30.004.volc
                    b30.120
                    b30.040z
                    b30.104
                    b30.008
                    b30.013.test
                    b30.012.0
                    b30.051c
                    b30.104w
                    b30.130
                    b30.045c
                    b30.045a
                    b30.044a
                    b30.009.y63
                    b30.041z
                    b30.030z
                    b30.020.beta14
                    b30.016
                    b30.100.01
    

  7. Investigate the need to retain files from the CCSM3 "b21" runs
  8. Do we need to retain any of the following b21.* data files, which were generated with the ccsm2_1_beta10 through ccsm2_1_beta18 tags? Cases are documented on the CCSM tags page.

    Savings: The elimination of all copies of the b21.* files would save an an estimated 50 GAUs/month; elimination of duplicate copies would save 25 GAUs/month.

    Approved Action: Delete all of the data files in the cases /CCSM/csm/$case, where $case is:

                    b21.001
                    b21.002
                    b21.003
                    b21.004
                    b21.005
                    b21.006
                    b21.007
                    b21.008
                    b21.009
                    b21.010
    

  9. Investigate the need to retain files from the CCSM3 "b22" runs
  10. All of the b22.005 cases are ocean-sensitivity/tuning runs that were set up and run by Nancy Norton on behalf of the oceanography working group.

    Savings: The elimination of all copies of the b22.005* files would save an an estimated 30 GAUs/month.

    Approved Action: Delete all of the data files in the cases /CCSM/csm/$case, where $case is

                    b22.005  
                    b22.005a
                    b22.005b
                    b22.005c
                    b22.005d
                    b22.005e
                    b22.005f
                    b22.005fs
                    b22.005g
                    b22.005h
                    b22.005i
                    b22.005j
                    b22.005k
                    b22.005m
                    b22.005n
                    b22.005o
    

    Do we need to keep any of the data files from the b22.* cases, which were generated with the ccsm2_2_beta01 through ccsm2_2_beta08 tags? Cases are documented on the CCSM tags page.

    Savings: The elimination of all copies of the remaining b22.* files, excluding the b22.005* files, would save an estimated 180 GAUs/month, whereas eliminating duplicate copies would save approximately 90 GAUs/month.

    Approved Action: Delete all of the data files in the cases /CCSM/csm/$case, where $case is:

                    b22.001
                    b22.002
                    b22.003
                    b22.004
                    b22.004a
                    b22.006
                    b22.007
                    b22.008
                    b22.009
                    b22.011
                    b22.012
                    b22.013
                    b22.014
                    b22.015
                    b22.016
                    b22.017