Lib-4171 : UNRECOVERABLE library error continued

Unified Model
1

Continuing the discussion from Lib-4171 : UNRECOVERABLE library error:

Hi CSM team,

Could I request further support on this suite please? I have encountered similar problem a few times in the last few weeks but I have managed to resolve the issues. In this last thread, the model failed at 10 months. I have now almost reached 5 years. So I think the model is stable enough to think this is not a science issue. I have had some issued with not being able to write to file because of disk quota on archer2. I have removed data from my work space so this should no longer be the issue. I have perturbed the restarts but this did not help. Much like in the previous discussion, I did the following:

  1. [env]PRINT_STATUS=PrStatus_Diag (Note: for debug only, revert back for actual run)
  2. [namelist:prnt_control]print_writers=1 (so output from all PEs is available).

The above settings caused the model to crash at start of timestep, but the model longer without them, so I reverted them.

I have since run it with PrStatus_Normal and [namelist:prnt_control]print_force_flush=.true. but this did not help either. Could I have some help trying to figure out what went wrong please?

Penny

2

I have run into the same ‘negative q at the start of convection’ error in the run about half way through the 202012 month. I have perturbed the restart a few times but that did not help. I have looked at the data output from the month before and things looks okay (i.e. the SSTs look sensible). So my guess is that perturbing theta should have worked.

I perturbed the restart from the month before and tried to rerun it (202011) so that I can get a new restart file for 202012. My thinking being that perhaps a small perturbation to the restart is not enough. I perturbed the 202011 file and triggered 202011 in the GUI. But the model failed with the following error in ocean.output.

===>>> : E R R O R

===>>>> : problem with nittrc000 for the restart
verify the restart file or rerun with nn_rsttr = 0 (namelist)

Are there any additional settings changes I should make to rerun a month which had previouly completed?

Penny

3

Hi Penny,
Yes, restarting a coupled model from a previous state is more complicated, see instructions : https://code.metoffice.gov.uk/trac/moci/wiki/tips_CRgeneral#Restartingfromarchivedrestarts

Apart from theta there can be other ways of perturbing the model: e.g.

Convection sub-stepping- Panel: um → namelist → UM Science Settings → section05 Convection → n_conv_calls : 2 → 3

Or model timestep itself u-dt414/rose-suite.conf: ATMOS_TIMESTEPS_PER_DAY=72 (increase this making sure the length i.e. 86400/N is a factor of 3600 secs)

Both these are supposed to be temporary and should be reverted back after a few months. This can be done without pausing the run: revert to original in ~/roses/u-dt414 and then execute cylc reinstall u-dt414 followed by cylc reload u-dt414, so the setting will be picked up by the next coupled task.

Mohit

4

That is extremely helpful. Thank you for all the info.