Diagnostics and Archiving

Unified Model
1

Hi,
I am running job u-db898. I have reduced the diagnostic set enormously as I only want minimal monthly pp (well I’d like netcdf) output. I am also getting a lot of “dumps”. So I have three questions.

  1. I am getting some other pp data files beyond the monthly-mean I am asking for:
    tetts@ln03:/work/n02/n02/tetts/cylc-run/u-db898/share/data/History_Data> ls -lh jan.pp
    -rw-r–r-- 1 tetts n02 419M Dec 7 05:41 db898a.pa1982jan.pp
    -rw-r–r-- 1 tetts n02 445M Dec 7 05:42 db898a.pd1982jan.pp
    -rw-r–r-- 1 tetts n02 412M Dec 7 05:43 db898a.pe1982jan.pp
    -rw-r–r-- 1 tetts n02 309M Dec 7 05:43 db898a.pk1982jan.pp
    -rw-r–r-- 1 tetts n02 43M Dec 12 06:05 db898a.pm1982jan.pp

But as far as I can tell there is no upa, upd, upe or upk diagnostics turned on. I’ve turned off all packages except Opt COSP, Opt Monthly Mean Diagnostics and UKCA Coupling Macro. The first two packages should be producing monthly-mean diagnostics and the latter does not seem to be responsible for the unwanted output – PV is one of the diagnostics requested in that package, and it is not in any of the .pp files. So how do I find out what is producing the unwanted pp data and stop it?

  1. I get a shedload of unwanted “dump” files (36 Gbytes in size) of which 16 Gbytes look to be dumps at the start of the month. (see below). Is there a way of auto-deleting all these dump files except what is needed? As each simulation is only for a month at a time, deleting everything unneeded at the end of a month would be fine.

tetts@ln03:/work/n02/n02/tetts/cylc-run/u-db898/share/data/History_Data> ls -lh | grep -v ‘.pp$’
total 36G
-rw-r–r-- 1 tetts n02 8.0G Dec 12 05:07 db898a.da19820201_00
-rw-r–r-- 1 tetts n02 8.0G Dec 12 05:56 db898a.da19820301_00
-rw-r–r-- 1 tetts n02 15M Dec 12 05:56 db898a.p41982feb
-rw-r–r-- 1 tetts n02 15M Dec 12 05:22 db898a.p41982jan
-rw-r–r-- 1 tetts n02 15M Dec 12 05:56 db898a.p51982feb
-rw-r–r-- 1 tetts n02 15M Dec 12 05:22 db898a.p51982jan
-rw-r–r-- 1 tetts n02 30M Dec 12 04:43 db898a.p619820101
-rw-r–r-- 1 tetts n02 30M Dec 12 04:54 db898a.p619820111
-rw-r–r-- 1 tetts n02 30M Dec 12 05:06 db898a.p619820121
-rw-r–r-- 1 tetts n02 30M Dec 12 05:34 db898a.p619820131
-rw-r–r-- 1 tetts n02 30M Dec 12 05:45 db898a.p619820210
-rw-r–r-- 1 tetts n02 30M Dec 12 05:56 db898a.p619820220
-rw-r–r-- 1 tetts n02 49M Dec 12 04:43 db898a.p719820101
-rw-r–r-- 1 tetts n02 49M Dec 12 04:54 db898a.p719820111
-rw-r–r-- 1 tetts n02 49M Dec 12 05:06 db898a.p719820121
-rw-r–r-- 1 tetts n02 49M Dec 12 05:34 db898a.p719820131
-rw-r–r-- 1 tetts n02 49M Dec 12 05:45 db898a.p719820210
-rw-r–r-- 1 tetts n02 49M Dec 12 05:56 db898a.p719820220
-rw-r–r-- 1 tetts n02 420M Dec 12 04:43 db898a.p819820101
-rw-r–r-- 1 tetts n02 420M Dec 12 04:54 db898a.p819820111
-rw-r–r-- 1 tetts n02 420M Dec 12 05:06 db898a.p819820121
-rw-r–r-- 1 tetts n02 420M Dec 12 05:34 db898a.p819820131
-rw-r–r-- 1 tetts n02 420M Dec 12 05:45 db898a.p819820210
-rw-r–r-- 1 tetts n02 420M Dec 12 05:56 db898a.p819820220
-rw-r–r-- 1 tetts n02 49M Dec 12 04:43 db898a.p919820101
-rw-r–r-- 1 tetts n02 49M Dec 12 04:54 db898a.p919820111
-rw-r–r-- 1 tetts n02 49M Dec 12 05:06 db898a.p919820121
-rw-r–r-- 1 tetts n02 49M Dec 12 05:34 db898a.p919820131
-rw-r–r-- 1 tetts n02 49M Dec 12 05:45 db898a.p919820210
-rw-r–r-- 1 tetts n02 49M Dec 12 05:56 db898a.p919820220
-rw-r–r-- 1 tetts n02 729M Dec 12 05:56 db898a.pa1982feb
-rw-r–r-- 1 tetts n02 7.9M Dec 12 05:22 db898a.pa1982jan
-rw-r–r-- 1 tetts n02 323M Dec 12 05:56 db898a.pb1982jan
-rw-r–r-- 1 tetts n02 1.2G Dec 12 05:56 db898a.pc1982jan
-rw-r–r-- 1 tetts n02 537M Dec 12 05:56 db898a.pd1982feb
-rw-r–r-- 1 tetts n02 7.9M Dec 12 05:22 db898a.pd1982jan
-rw-r–r-- 1 tetts n02 644M Dec 12 05:56 db898a.pe1982feb
-rw-r–r-- 1 tetts n02 7.9M Dec 12 05:22 db898a.pe1982jan
-rw-r–r-- 1 tetts n02 84M Dec 12 05:56 db898a.pf1982jan
-rw-r–r-- 1 tetts n02 303M Dec 12 05:56 db898a.pg1982jan
-rw-r–r-- 1 tetts n02 8.8M Dec 12 05:56 db898a.ph1982feb
-rw-r–r-- 1 tetts n02 8.8M Dec 12 05:22 db898a.ph1982jan
-rw-r–r-- 1 tetts n02 22M Dec 12 05:56 db898a.pi1982jan
-rw-r–r-- 1 tetts n02 64M Dec 12 05:56 db898a.pj1982jan
-rw-r–r-- 1 tetts n02 478M Dec 12 05:56 db898a.pk1982feb
-rw-r–r-- 1 tetts n02 7.9M Dec 12 05:22 db898a.pk1982jan
-rw-r–r-- 1 tetts n02 4.7G Dec 12 05:56 db898a.pm1982feb
-rw-r–r-- 1 tetts n02 4.8G Dec 12 05:22 db898a.pm1982jan
-rw-r–r-- 1 tetts n02 196M Dec 12 04:43 db898a.pt19820101
-rw-r–r-- 1 tetts n02 196M Dec 12 04:54 db898a.pt19820111
-rw-r–r-- 1 tetts n02 196M Dec 12 05:06 db898a.pt19820121
-rw-r–r-- 1 tetts n02 196M Dec 12 05:34 db898a.pt19820131
-rw-r–r-- 1 tetts n02 196M Dec 12 05:45 db898a.pt19820210
-rw-r–r-- 1 tetts n02 196M Dec 12 05:56 db898a.pt19820220
-rw-r–r-- 1 tetts n02 49M Dec 12 05:56 db898a.pu1982feb
-rw-r–r-- 1 tetts n02 49M Dec 12 05:22 db898a.pu1982jan
-rw-r–r-- 1 tetts n02 15M Dec 12 05:56 db898a.pv1982feb
-rw-r–r-- 1 tetts n02 15M Dec 12 05:22 db898a.pv1982jan
-rw-r–r-- 1 tetts n02 0 Dec 12 05:22 db898.stash
-rw-r–r-- 1 tetts n02 11K Dec 12 05:56 db898.xhist
-rw-r–r-- 1 tetts n02 455 Dec 12 06:05 fort.9
drwxr-sr-x 2 tetts n02 4.0K Dec 12 06:00 mean_archflags
drwxr-sr-x 2 tetts n02 4.0K Dec 7 00:17 seedfiles

  1. I don’t want to archive data to JASMIN (the only stuff I care about is 40 Mbytes a month) but I may want to move the wanted data somewhere else. I think I’ve got archiving turned off (though it was pointing to Luke’s archive dir on ARCHER2) but the pptransfer is failing Error message is:
    [WARN] file:atmospp.nl: skip missing optional source: namelist:archer_arch
    [WARN] file:pptransfer.nl: skip missing optional source: namelist:archer_arch
    [WARN] file:atmospp.nl: skip missing optional source: namelist:moose_arch
    [WARN] file:atmospp.nl: skip missing optional source: namelist:script_arch
    [WARN] file:pptransfer.nl: skip missing optional source: namelist:pptransfer
    [FAIL] Transfer: Failed to load &archer_arch namelist from namelist file: pptransfer.nl
    [FAIL] Terminating PostProc…
    [FAIL] transfer.py <<‘STDIN’
    [FAIL]
    [FAIL] ‘STDIN’ # return-code=1
    2023-12-12T08:37:56Z CRITICAL - failed/EXIT

So my question is – can I just have the postproc job copy/move things to my archive dir on ARCHER2 but with no transfer to JASMIN? If not,how do I stop the suite submitting the pptransfer job?

many thanks
Simon

and a ps/4th question.

I’ve modified the simulation so it starts 1st Oct 2010 and runs for 3 months. It proceeds as above except it does not generate PP data. How do I fix that?

2

Hi SImon

The extra files just contain headers for the usage profiles that are defined but not written to. You may need to disable both unused usage profile and its associated model output stream (I can’t exactly remember!) to stop the empty files.

I can’t follow the dump files issue (your suite is running as I’m looking at it) - but it seems to be writing monthly dumps as requested. Postproc should delete superseded files.

but I may want to move the wanted data somewhere else — off ARCHER?

I don’t understand q4?

Grenville

3

@grenville thanks a lot. Have gone and ignored the ppNNN which crashed the model. So also ignored the unused usage namelists. (They can always be switched back on later if someone wants them). I’ve still got some output, but it is a lot less so I can live with it.

The 4th question was because I did not see any PP data but that gets generated at the end of the run!
Simon