CGD Research: CSEG Overview

CCSM Software Engineering Group

CCSM software infrastructure

CSEG infrastructure improvement efforts have focused on producing a more robust, easily extensible and portable model system. Efforts to improve CCSM robustness were concentrated in the areas of source code maintenance and testing infrastructure. A major improvement to the CCSM code base resulted from the migration of the CCSM source code from CVS to the Subversion revision control system. Over 100 users were affected by this transition and the general consensus has been that the migration resulted in greater flexibility and fluency in the development work cycle. The CCSM testing framework was also significantly improved in order to ensure system robustness as new science is incorporated across model components. As part of this project, an automated test suite capability was implemented which can be trivially extended to include new component configurations. The new testing framework has become an integral part of CCSM code development.

Porting both the release and development versions of CCSM to new platforms continues to be an important priority for the CSEG group. Successful porting efforts, including the production of validated climates, were achieved for the Cray X1E and the NCAR Linux Cluster. Current work is underway to port to the Cray XT3. The CCSM Software Engineering Group has worked closely with our partners at the DOE Oak Ridge National Lab and Cray to carry out this effort. Portability efforts have also focused on the creation of a single-executable CCSM. Until now, all versions of CCSM have run as a multiple-executable (MPMD) concurrent systems. CSEG has produced a new single-executable (SPMD) version of the MPMD concurrent model. The single executable implementation will lead to enhanced portability and ease of debugging compared to the MPMD version and will be able to run on machines that do not currently support MPMD implementations, such as the NCAR IBM BlueGene/L.

An automated performance utility was also created in order to provide users with the capability of easily determining optimal load-balanced configurations for their production runs. This tool has dramatically decreased the time and effort required to assess CCSM load balance, has provided CCSM with a source for easily documenting CCSM performance, and has made it much easier for users in the community to understand and assess CCSM performance.

CSEG has also created a new experimental run database, containing a web interface for both the input and query of model information, in order to provide both science and software developers the ability to easily duplicate and archive details of CCSM experiments. In the past, meta-data detailing various model runs were stored in multiple places, thereby making it difficult to easily determine details of past runs. A central, searchable database will provide the necessary information to easily reproduce a particular run and will also serve as a single placeholder for pointers to diagnostics and other model output.

Creation of Next Generation CCSM Data Models

The CCSM data models provide scientists with the capability to perform experiments where selected components are forced with prescribed data, thereby eliminating feedback. This mode of running CCSM is often used to examine new science within one component before it is coupled to the fully active system. It was desirable for all CCSM data models to have the ability to leverage code-reusability in order to perform operations such as regridding from the input data to the model grid and time cycling of input forcing data. The entire data model suite was completely rewritten to ensure uniform functionality and to also make it much easier to incorporate new science in any given data model. As part of this process, new science functionality was incorporated into several of the data models, including the creation of slab ocean functionality in the data ocean component. The new data model suite is proving to be an integral part of the spin up runs being performed by the Biogeochemistry Working group and those being carried out as part of the Carbon Land Model inter-comparison Project (C-LAMP).

CAM Development

CAM development efforts focused on the need to more easily experiment with alternative dynamical cores. Two projects were targeted as part of this effort. The first project merged the versions of the finite volume (FV) dynamical core utilized in CAM and in NASA's GEOS5. As part of this process, a generic interface to the FV dynamical core was implemented. The goal of the second project was to integrate the HOMME dynamical core into CAM. A primary motivation for this work is the need to produce a version of CAM that will run efficiently on tens of thousands of processors. The HOMME dynamical core utilizes a cubed sphere grid and gives CAM this capability due to its implementation using highly scalable spectral element methods. This project has advanced to the stage of early validation of the HOMME dynamical core driven by idealized physical packages.

Significant infrastructure improvements have also been made to CAM over the last year. A new, comprehensive, automated testing framework was introduced to facilitate the ability for atmospheric modelers to quickly and thoroughly test code modifications prior to having these changes incorporated into the CAM source code base. This testing framework has proven to be an invaluable part of the CAM development effort by identifying countless problems before they entered the model.

POP Development

The main focus of POP development was the integration of the LANL POP2 ocean model into CCSM. This work involved the extensive modification of the POP2 source code to support CCSM requirements and to include existing CCSM ocean-model physics. In addition, major new physical parameterizations were also incorporated. POP2 is now completely integrated into the CCSM code base and scripts and will be utilized in new CCSM integrations.

CLM Development

Up to this point, CLM has been constrained to run on the CAM grid when coupled to CAM. CLM software development efforts have focused on providing CLM with the functionality to run on its own high resolution, fine-mesh grid, independent of the CAM grid and couple to CAM via physically relevant downscaling and upscaling parameterizations. Improvement of memory usage and memory scaling has been fundamental to this work and will continue to be important as the requirement for memory and performance portable routines becomes integral to the CCSM system.

CICE Development

CICE software engineering efforts have focused on the incorporation of the Los Alamos National Laboratory sea ice model (CICE) as the standard CCSM sea ice component. The first step in this process involved the evaluation of the differences between CICE version 3.1 and the Community Sea Ice Model (CSIM) version 5. The second stage of the process was the incorporation of CICE version 4.0 into the development CCSM code base. Tests are currently underway to evaluate the impact of the new code within the coupled model framework.

In addition to incorporating CICE into the coupled model, other software engineering efforts involved providing assistance in efforts to couple CICE Version 4.0 to the current proto-type sequential CCSM and the investigation of running CICE at 0.1-degree globally on petascale architectures.

Creation of Single-Executable Sequential CCSM

Projected peta-scale architectures will require the ability for each CCSM component to run on much higher resolution configurations utilizing processor counts that are one to two orders of magnitude greater than present settings. A sequential CCSM, where all components run on the same processor set, might provide a better architecture in which to ensure the type of quasi-local communication necessary to scale the system up to these projected processor utilizations. The goal of this project is to create a sequential system that contains backwards compatibility with the current concurrent CCSM system (where each component runs on a unique set or processors, not utilized by any other component), provides "plug and play" capability of data and active components, and produces the same climate as the current concurrent system.

The stand-alone CAM model (already a sequential system) was chosen as the starting point for the creation of a CCSM sequential system. To accomplish the first stage of this project, a significant advance was implemented in CAM to permit the transition from a CAM standalone model, where all the surface components were implemented as sub-components of CAM that made explicit use of CAM's data structures, to a version of CAM implemented as a sequential CCSM where the surface components are independent of CAM and communicate with CAM only utilizing framework dependent data structures. The design of these framework-dependent coupling interfaces will make it straightforward to add other CCSM components (e.g., CICE4 and POP2) in the next stage of this project.

Both ESMF and MCT (the current CCSM coupling framework) are being examined as coupling frameworks for the sequential system. Examination of ESMF as a coupling framework is occurring via the Stage I Evaluation project which is due to be completed later this fall.

Additional Information

Print version (.pdf)

CGD Sectional Narratives

CAS | CCR | CDP | CMS | CSEG | OS | TSS |