February 1, 2008

CGD/IS began tracking major data stores in January, 2006.
At that time there was about 25 TB of storage available. That
number has more than doubled to 80 TB as of January, 2008, and
will continue to grow due primarily to increased model output
and analysis data sets.


(Historical data indicates the division started with 1 TB of
data in January, 2003.)

This growth has begun to strain our current infrastructure.
Over the next few months, CGD/IS will be over-hauling significant
portions of the infrastructure to support continued growth and
at the same time maintain stability and reliability. Current
storage allocations will also be increased, some significantly.

The planned increases in space will also require a new implementation
of backup software and hardware, which will add cost and time.
However, the backup policies will remain in effect, and possibly
be expanded.

Note: Backups are held for 3 months, only.

For the most part, users will only notice (and enjoy) the increased
space. However, there are a few areas where division practices will
be changed significantly. Please read below.

This plan remains
flexible, but is now under way. Please feel free to comment or
ask questions. Progress and details of the changes will be in
the CGD/IS SysWrk notifications.

/home (H:)
    Backup: daily to tape, snapshots at 0:00, 6:00, 12:00, 18:00
    Quota:  3 GB per person
    Size:   1 TB

    This is the most critical space within the division, and
    is served by a highly reliable system. It is also the
    single most expensive piece of equipment, and the most
    difficult to upgrade. The unit is at its maximum capacity,
    and will have to remain in service for at least another year,
    and possibly longer.

    The eventual goal is to increase the standard quota to 10 GB
    per user once new hardware is acquired. Until then, CGD/IS will be
    shaping the quotas to address users' individual immediate needs.

----------------------- Backup: daily to tape Quota: None. Allocation manage by section. Current Size: Varies by section: acacia 1.1G cam 7.0G cas 3.2G ccr 50G cdp 8.4G cms 63G cseg 2.2G csm 32G oce 12G pcm 1.1G systems 1.1G tss 58G vemap 23G Historical note: Circa Jan, 2002, /web/web-data was 17 GB of common space and shared by the entire div. This resulted in the ubiquitous e-mails pleading for users to clean up their space. The space was increased several times, and filled several times over the succeeding years. Finally, in 2006, each section was given its own partition to make usage management a local issue, not division wide. The space allocated was based on existing usage plus 10%. This resulted in lop-sided allocations based on the each section's use of the web, but it has served rather well until now. Note: A few soft links exist from the current /web/web-data directories into /project/
. These links will be broken and the data (as of today) migrated to /web/web-data. External mounts (i.e., /project/
) will no longer be available from the web server. This is for stability reasons as well as security. /project/
------------------ Backup: None. Quota: None. Allocation managed by section. Size: 1 TB per section. The /project/
partitions will be increased initially by 500GB to 1.5 TB each, with an eventual goal of 2 TB per section by the beginning of summer, 2008. The /project/
partitions seem to be the most successful allocation/use of data storage in the division and will continue to be grown as long they are useful. /fs/cgd ------- Backup: Daily. Quota: None. Allocation unmanaged. Size: /csm 126 GB /data0 369 GB /home0 29 GB The space for each of these will be increased after consultation with CSEG and Dennis Shea (the traditional keeper of the space). A group quota scheme based on sections will be implmented on /home0 and /data0 in an attempt to avoid the partitions from being filled (and the resulting e-mails to the division to please clean up). If group quotas fail, individual quotas will be implemented. Ownership on static data files in /csm will be changed to a common owner to ease management issues for CSEG. /var/mail --------- Backup: daily to tape, snapshot at 6:00, 12:00, 18:00 Quota: 1 GB per person Size: 100 GB Unlike the other partitions listed here, /var/mail is adequately sized and the quotas are reasonable. However, access time via Thunderbird and other mail clients has become intolerable when performing any mail operations (reading, sending, deleting messages). This is due to the increase e-mail traffic load, which increases the disk I/O. To hopefully fix this problem, /var/mail will be moved to new hardware, placed into a new highly available cluster configuration. NOTE: Additionally, /var/mail will no longer be mounted on all systems within the division. This will affect users who still access e-mail with command-line interfaces such as mutt or pine. These e-mail clients can be re-configured to use the IMAP protocols. (Details later.)