I have just ran an AMIP UKESM1.1 simulation at version 13.8 – u-dr157/run7. The run has ran from 1979-01-01 to 2014-12-30 (Well to the first timestep of 2015). I have monthly output and it is all there except for the December 2014 data. How do I get the model to produce that and transfer it to Jasmin?
I’ve looked in /mnt/lustre/a2fs-nvme/work/n02/n02/tetts/cylc-run/u-dr157/run7/share/cycle/20141001T0000Z/ and there are files there for Sep/Oct/Nov but no December. Model output says its last timestep was 2015-01-01.
I think this might be ‘by design’, see app/postproc/rose-app.conf [env]ARCHIVE_FINAL=false
If memory serves me right, the postproc scripts require a specific setting (something related to time period or end date) from Rose/Cylc when this is set to true, but that setting is no longer available after the move to Cylc-8. So, this was a workaround (setting to false) till postproc is upgraded to correspond to cylc-8. However, I am not aware if this has been fixed yet.
In this case the workflow may have to be set to run for an extra month so that the required files are processed and archived.
I think it will have to be atleast a month extra. The CMIP6 ancillaries should have extra months before and after the target period.
For a run that has stopped, changing the RUNLEN (or equivalent ) and ‘cylc vr suite/runX’ should work.
I tried running for an extra month and the model crashed. Both the SST and Sea-ice ancillaries go up to 2016-12 (and I am running 2015-01-01). I could imagine that the UKCA emission ancillaries stop in 2014 but can’t work out the filepaths so can’t inspect them.
Simon
Error message:
???
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 1
? Error from routine: EG_BICGSTAB
? Error message: NaNs in error term in BiCGstab after 1 iterations
? This is a common point for the model to fail if it
? has ingested or developed NaNs or infinities
? elsewhere in the code.
? See the following URL for more information:
? https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints
? Error from processor: 279
? Error number: 71
If the ancillaries or emissions run out of data, the model should fail cleanly with a relevant message.
I suspect this is due to the time varying GHG concentrations used by Radiation ([namelist:clmchfcg]). These are annual values (specified upto 2015 here) and the model uses these by interpolating between current and ‘next’ year. If the model is running for Jan-2015 the code will try to interpolate between 2015 and 2016, which does not exist so will use random values for these gases in radiation.
You can try after copying 2015 values as 2016 in the above namelist, although there are a lot of entries to change.
Mohit
There is a convpp command-line utility from Mule , but not sure if this will result in files that are identical to those from the postproc step, or if additional processing is needed.
There would be no harm in changing ARCHIVE_FINAL to true, reloading with a cylc vr u-dr157 and re-triggering postproc task from the 20141001 cycle. I’ve just tried it in a test run of your suite and it seems to work for me. If it doesn’t work, then you’ve not lost anything and can still to do it manually.
The version of postproc your suite is using checks if ARCHIVE_FINAL is set otherwise tries to figure out if the current cycle is the last one, which as Mohit says, won’t work because of changes associated with cylc 8.
many thanks for that suggestion. Worked and I’ve just triggered the archive which seems to have worked too. Only thing is that there are more files – 4 a.pm. rather than 3. I assume that is a side-effect of it being the last archive or I added an extra month to the run.
One follow up question. Is it OK to have ARCHIVE_FINAL set true for ‘normal’ running? Or will that cause problems? I could imagine that it would stop the annual average files being generated.
And I am trying to do the same thing to run u-dr873 which has, I think, ran to the end of 2024 but I did not set it up to run an extra day or more. Using the gui ‘trigger’ is greyed out for the postproc action. So I can’t run it from the suite.
No you can’t set ARCHIVE_FINAL=true for normal running. As soon as you set ARCHIVE_FINAL=true I think it will start archiving stuff that is needed for the next cycle to continue. You’ll need to run the suite with ARCHIVE_FINAL=false and switch it to True for the final cycle. This is why the recommended method currently is to run one cycle more than you need.
Not sure why trigger would be greyed out for postproc unless the whole suite has shutdown. Try retriggering it from the command liine: cylc trigger u-dr873//20241001T0000Z/postproc
tetts@puma2 ~]$ cylc trigger u-dr873//20241001T0000Z/postproc
WorkflowStopped: u-dr873 is not running
If I do cylc play u-dr873 and then look at the gui it is showing as stopped. I increased the run by 1 D to 36Y1D then did cylc vr u-dr873 and that gives me a running job. I can then trigger the postproc and that works. I then manually triggered pptransfer and that has put data on jasmin.
housekeeping did not run and I it is not showing in the cylc gui.
Would a better approach to run all my workflows for 1 more day than I need, giving a last cycle that is 1 day long. That might leave a bunch of data in share/data/History_Data/ – u-dr873 has left a bunch of files in that directory – most are December 2014, some look to be October 2014.
If I extend by 1D the default UKESM1.1 AMIP configuration will probably crash when it runs out of GHG drivers but I don’t care about that.
[