Problem with postproc_atmos

Unified Model
1

Hi,
I’ve run into a problem with one of my simulations (u-cu100) running on ARCHER2.
It was running fine but crashed (due to a disk quota issue) while creating a seasonal mean atmospheric file 1909 mam.
I restarted it and it ran forward a few months but crashed trying to create the annual mean file as the 1909 mam file was corrupt,
I followed the advice here, to restart from 1908.
https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#Restartingfromarchivedrestarts
It has now crashed again in the 19080101 postproc-atmos task with:
[ERROR] Annual mean for year ending November 1909 not possible as only got 0 file(s):
Which is a similar error as before (although before it crashed due to a corrupt file rather than the absence of any files).
Is there anyway to reset it so that it stops thinking the run has got to the end of 1909?
Thanks
Andrew

2

Andrew

Strange message in job.err, but in job.out:

Seasonal mean for Winter, ending 1908 not possible as only got 2 file(s):
        /work/n02/n02/schn02/cylc-run/u-cu100/share/data/History_Data/cu100a.pm1908feb, /work/n02/n02/schn02/cylc-run/u-cu100/share/data/History_Data/cu100a.pm1908jan

It can’t find the Dec 1907 file - that file appears to have been converted to pp in /work/n02/n02/schn02/archive/u-cu100/19080101T0000Z/cu100a.pm1907dec.pp

Have you changed the model start date? You may need to back up to the beginning of a climate meaning period to get in sync with the meaning.

Grenville

3

Hi Grenville,
The model failed at the end of 1909. So I backed up to the restart files in 1908 to restart (with an NRUN), and thus changed the start date. I only have restart files for January so that was what I used.
Can you advise when I should be restarting from?
Thanks
Andrew

4

Andrew

For the climate meaning to work, it’ll need cu100a.pm1907dec file - please hang on little, we may have a way to convert back to a fields file.

Grenville

5

Hi Grenville,
Thanks for looking into this.
This is where the pp file is
/work/n02/n02/schn02/archive/u-cu100/19080101T0000Z/cu100a.pm1907dec.pp
Does that help?
Andrew

6

Hi Grenville,
I think my new run should be bit compatible with respect to the existing run. So would skipping the postproc_atmos task be an option? Given that I already have the output saved?
Thanks
Andrew

7

Andrew

Looks like you have switched off climate meaning - in that case you should be OK to continue the run, but I’m not exactly sure what state postproc will be in.

The code we were hoping to find is not readily acessible.

Grenville

8

Hi Grenville,
So I restarted in 1908 January.
The postproc-atmos task failed for every cycle but I ignored this (marked as successful) for all of 1908 hoping that by the time it got to the first cycle of 1909 it would be OK.
But the cycle 190901 is failing with error message
[ERROR] Annual mean for year ending November 1909 not possible as only got 0 file(s):
Why does it think that it should be calculating the annual mean for 1909 when its only in January?
Is there anyway to stop this?
Thanks
Andrew

9

Hi Andrew

I’ve been a bit slow! The climate meaning is happening in postproc (create_means=true) (not in the model as I’d somehow thought).

Your meaning periods are djf, mam,jja, son – I think the model originally started in Dec, or March or June or Sept - resetting the start data to Jan causes confusion.

Setting create_means=false will let postproc succeed but you will need to crate means from the monthly output by some other method.

Grenville