-----------------------
| CCSM case b40.009 |
-----------------------
% qstat -a | grep amai
469389.nid03588 amai batch b40.009 1995 1 1184 -- 01:30 R 00:12
469390.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469391.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469392.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469393.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469394.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469395.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469396.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469397.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469398.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469399.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469400.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469401.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
469402.nid03588 amai batch b40.009 -- 1 1184 -- 01:30 H --
% ls b40.009.o4*
b40.009.o467231 b40.009.o467555 b40.009.o467840 b40.009.o468264 b40.009.o468457 b40.009.o469150 b40.009.o469294
b40.009.o467232 b40.009.o467556 b40.009.o467999 b40.009.o468265 b40.009.o468458 b40.009.o469151 b40.009.o469295
b40.009.o467233 b40.009.o467557 b40.009.o468062 b40.009.o468266 b40.009.o468459 b40.009.o469152 b40.009.o469296
b40.009.o467234 b40.009.o467558 b40.009.o468063 b40.009.o468267 b40.009.o468460 b40.009.o469153 b40.009.o469297
b40.009.o467235 b40.009.o467559 b40.009.o468064 b40.009.o468268 b40.009.o468874 b40.009.o469154 b40.009.o469298
b40.009.o467236 b40.009.o467560 b40.009.o468065 b40.009.o468269 b40.009.o468875 b40.009.o469155 b40.009.o469299
b40.009.o467237 b40.009.o467561 b40.009.o468066 b40.009.o468270 b40.009.o468876 b40.009.o469156 b40.009.o469382
b40.009.o467238 b40.009.o467648 b40.009.o468067 b40.009.o468271 b40.009.o468877 b40.009.o469157 b40.009.o469383
b40.009.o467239 b40.009.o467649 b40.009.o468068 b40.009.o468451 b40.009.o468878 b40.009.o469158 b40.009.o469384
b40.009.o467240 b40.009.o467650 b40.009.o468069 b40.009.o468452 b40.009.o468879 b40.009.o469159 b40.009.o469385
b40.009.o467241 b40.009.o467651 b40.009.o468070 b40.009.o468453 b40.009.o468880 b40.009.o469290 b40.009.o469386
b40.009.o467552 b40.009.o467652 b40.009.o468071 b40.009.o468454 b40.009.o468881 b40.009.o469291 b40.009.o469387
b40.009.o467553 b40.009.o467837 b40.009.o468262 b40.009.o468455 b40.009.o468882 b40.009.o469292 b40.009.o469388
b40.009.o467554 b40.009.o467839 b40.009.o468263 b40.009.o468456 b40.009.o468883 b40.009.o469293
% ls b40.009.o4* | wc -l
97
>>> Successful completion
% ls -l b40.009.o4* | egrep '(\ 6293\ |\ 6292\ |\ 6285\ |\ 6284\ )'
-rw-rw-r-- 1 amai cli017 6284 Nov 6 00:12 b40.009.o467232
-rw-rw-r-- 1 amai cli017 6284 Nov 6 01:28 b40.009.o467233
-rw-rw-r-- 1 amai cli017 6284 Nov 6 02:44 b40.009.o467234
-rw-rw-r-- 1 amai cli017 6284 Nov 6 04:00 b40.009.o467235
-rw-rw-r-- 1 amai cli017 6284 Nov 6 05:15 b40.009.o467236
-rw-rw-r-- 1 amai cli017 6285 Nov 6 06:31 b40.009.o467237
-rw-rw-r-- 1 amai cli017 6285 Nov 6 07:46 b40.009.o467238
-rw-rw-r-- 1 amai cli017 6285 Nov 6 20:53 b40.009.o468062
-rw-rw-r-- 1 amai cli017 6285 Nov 6 23:27 b40.009.o468068
-rw-rw-r-- 1 amai cli017 6285 Nov 7 07:49 b40.009.o468071
-rw-rw-r-- 1 amai cli017 6285 Nov 7 11:10 b40.009.o468265
-rw-rw-r-- 1 amai cli017 6285 Nov 7 12:35 b40.009.o468267
-rw-rw-r-- 1 amai cli017 6285 Nov 7 13:50 b40.009.o468268
-rw-rw-r-- 1 amai cli017 6285 Nov 7 15:04 b40.009.o468269
-rw-rw-r-- 1 amai cli017 6285 Nov 7 16:45 b40.009.o468271
-rw-rw-r-- 1 amai cli017 6285 Nov 7 17:59 b40.009.o468451
-rw-rw-r-- 1 amai cli017 6285 Nov 7 19:13 b40.009.o468452
-rw-rw-r-- 1 amai cli017 6285 Nov 7 20:27 b40.009.o468453
-rw-rw-r-- 1 amai cli017 6285 Nov 7 22:50 b40.009.o468456
-rw-rw-r-- 1 amai cli017 6285 Nov 8 00:04 b40.009.o468457
-rw-rw-r-- 1 amai cli017 6285 Nov 8 04:37 b40.009.o468458
-rw-rw-r-- 1 amai cli017 6285 Nov 8 05:51 b40.009.o468459
-rw-rw-r-- 1 amai cli017 6285 Nov 8 13:52 b40.009.o468881
-rw-rw-r-- 1 amai cli017 6285 Nov 8 20:58 b40.009.o468883
-rw-rw-r-- 1 amai cli017 6285 Nov 9 04:44 b40.009.o469150
-rw-rw-r-- 1 amai cli017 6285 Nov 9 21:54 b40.009.o469382
-rw-rw-r-- 1 amai cli017 6293 Nov 10 03:39 b40.009.o469383
-rw-rw-r-- 1 amai cli017 6293 Nov 10 04:53 b40.009.o469384
-rw-rw-r-- 1 amai cli017 6293 Nov 10 06:26 b40.009.o469386
-rw-rw-r-- 1 amai cli017 6293 Nov 10 07:39 b40.009.o469387
-rw-rw-r-- 1 amai cli017 6293 Nov 10 09:00 b40.009.o469388
% ls -l b40.009.o4* | egrep '(\ 6293\ |\ 6292\ |\ 6285\ |\ 6284\ )' | wc -l
31
>>> Failures
Failure mode 1: Job goes into execution state without doing anything *
% ls -l b40.009.o4* | egrep '(\ 5369\ |\ \ \ 84\ )'
-rw-rw-r-- 1 amai cli017 84 Nov 6 11:35 b40.009.o467649
-rw-rw-r-- 1 amai cli017 84 Nov 6 12:29 b40.009.o467652
-rw-rw-r-- 1 amai cli017 5369 Nov 6 12:52 b40.009.o467837
-rw-rw-r-- 1 amai cli017 5369 Nov 6 15:01 b40.009.o467839
-rw-rw-r-- 1 amai cli017 5369 Nov 6 15:05 b40.009.o467840
-rw-rw-r-- 1 amai cli017 84 Nov 7 09:08 b40.009.o468264
-rw-rw-r-- 1 amai cli017 84 Nov 7 15:04 b40.009.o468270
-rw-rw-r-- 1 amai cli017 84 Nov 7 21:03 b40.009.o468455
-rw-rw-r-- 1 amai cli017 84 Nov 8 10:26 b40.009.o468877
-rw-rw-r-- 1 amai cli017 84 Nov 8 10:42 b40.009.o468880
-rw-rw-r-- 1 amai cli017 84 Nov 8 18:19 b40.009.o468882
% ls -l b40.009.o4* | egrep '(\ 5369\ |\ \ \ 84\ )' | wc -l
11
Failure mode 2: restart files for cpl and atm are not written **
% ls -l b40.009.o4* | egrep '(\ 5498\ |\ 5499\ )'
-rw-rw-r-- 1 amai cli017 5498 Nov 5 22:57 b40.009.o467231
-rw-rw-r-- 1 amai cli017 5499 Nov 6 07:53 b40.009.o467239
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:12 b40.009.o467240
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:30 b40.009.o467241
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:31 b40.009.o467552
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:32 b40.009.o467553
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:34 b40.009.o467554
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:35 b40.009.o467555
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:36 b40.009.o467556
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:37 b40.009.o467557
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:38 b40.009.o467558
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:39 b40.009.o467559
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:40 b40.009.o467560
-rw-rw-r-- 1 amai cli017 5499 Nov 6 09:41 b40.009.o467561
-rw-rw-r-- 1 amai cli017 5499 Nov 6 11:34 b40.009.o467648
-rw-rw-r-- 1 amai cli017 5499 Nov 6 12:27 b40.009.o467650
-rw-rw-r-- 1 amai cli017 5499 Nov 6 12:29 b40.009.o467651
-rw-rw-r-- 1 amai cli017 5499 Nov 6 17:13 b40.009.o467999
-rw-rw-r-- 1 amai cli017 5499 Nov 6 22:08 b40.009.o468063
-rw-rw-r-- 1 amai cli017 5499 Nov 6 22:09 b40.009.o468064
-rw-rw-r-- 1 amai cli017 5499 Nov 6 22:10 b40.009.o468065
-rw-rw-r-- 1 amai cli017 5499 Nov 6 22:11 b40.009.o468066
-rw-rw-r-- 1 amai cli017 5499 Nov 6 22:12 b40.009.o468067
-rw-rw-r-- 1 amai cli017 5499 Nov 7 00:40 b40.009.o468069
-rw-rw-r-- 1 amai cli017 5499 Nov 7 01:41 b40.009.o468070
-rw-rw-r-- 1 amai cli017 5499 Nov 7 09:07 b40.009.o468262
-rw-rw-r-- 1 amai cli017 5499 Nov 7 09:08 b40.009.o468263
-rw-rw-r-- 1 amai cli017 5499 Nov 7 11:22 b40.009.o468266
-rw-rw-r-- 1 amai cli017 5499 Nov 7 21:03 b40.009.o468454
-rw-rw-r-- 1 amai cli017 5499 Nov 8 06:18 b40.009.o468460
-rw-rw-r-- 1 amai cli017 5499 Nov 8 10:24 b40.009.o468874
-rw-rw-r-- 1 amai cli017 5499 Nov 8 10:25 b40.009.o468875
-rw-rw-r-- 1 amai cli017 5499 Nov 8 10:26 b40.009.o468876
-rw-rw-r-- 1 amai cli017 5499 Nov 8 10:41 b40.009.o468878
-rw-rw-r-- 1 amai cli017 5499 Nov 8 10:42 b40.009.o468879
-rw-rw-r-- 1 amai cli017 5499 Nov 9 05:04 b40.009.o469151
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:19 b40.009.o469152
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:20 b40.009.o469153
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:21 b40.009.o469154
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:22 b40.009.o469155
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:23 b40.009.o469156
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:24 b40.009.o469157
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:25 b40.009.o469158
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:26 b40.009.o469159
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:27 b40.009.o469290
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:29 b40.009.o469291
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:30 b40.009.o469292
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:31 b40.009.o469293
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:32 b40.009.o469294
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:33 b40.009.o469295
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:34 b40.009.o469296
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:35 b40.009.o469297
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:36 b40.009.o469298
-rw-rw-r-- 1 amai cli017 5499 Nov 9 06:37 b40.009.o469299
% ls -l b40.009.o4* | egrep '(\ 5498\ |\ 5499\ )' | wc -l
54
% ls -l b40.009.o469385
-rw-rw-r-- 1 amai cli017 5507 Nov 10 05:12 b40.009.o469385
% tail -1 b40.009.o469385
Model did not complete - see /tmp/work/amai/b40.009/ccsm_se/cpl.log.081110-045409
% ls /tmp/work/amai/b40.009/ccsm_se/cpl.log.081110-045409
ls: /tmp/work/amai/b40.009/ccsm_se/cpl.log.081110-045409: No such file or directory
% ls logs/*081110-045409*
ls: No match.
NOTES
-----
1. The initial submission was done at approximately 16:30 (14:30 MST) of 05 Nov 2008.
2. * In this failure mode we are charged for the use of 1184 processors for 1:30
without writing a single byte of output anywhere (it does write a small amount
of text to the b40.009.o46???? file -- see NOTE 4 below). The H jobs remain.
3. **This is the worst failure mode. The job runs for a whole year using 1184
processors for wc 1:16. It then leaves the restart files in such a state that
restarts are impossible. Henceforth, all H jobs are cleaned out of the queue.
After an arbitrary amount of time I re-submit the job, which is now at the
end of the queue. So we have a long wait in the queue only for the year to be
re-done from the beginning. (But see NOTE 5 below.)
4. Failure mode 1 comes in two flavors:
% cat b40.009.o467649
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
% tail -5 b40.009.o467839
-------------------------------------------------------------------------
- CCSM BUILD HAS FINISHED SUCCESSFULLY
-------------------------------------------------------------------------
Thu Nov 6 14:52:14 EST 2008 -- CSM EXECUTION BEGINS HERE
Terminated
5. Two of the failures labeled failure mode 2 actually occurred during the
middle part of the run (could not write a history file), not with the
writing of the re-start files. Two others occurred when the machine
crashed unexpectedly.