Free running mode with vn12 nesting suite

[shakka]

Hi CMS,

I’m trying to submit a vn 12.0 suite in free-running mode (i.e. FREE_RUN = true), with large-scale forcing from ERA5. However, the suite won’t submit and I get Jinja2 errors related to errors in suite-runtime/lams.rc at submission time:

[FAIL] File “/home/shakka/cylc-run/u-cq701/suite-runtime/lams.rc”, line 466, in top-level template code
[FAIL] DATADIR = {{mod[“ics”][“dir”]}}
[FAIL] UndefinedError: ‘dict object’ has no attribute ‘prev_ics’

Looks like the code that allows each cycle to take initial conditions from the previous LAM forecast isn’t complete. Am I missing a branch/code update or revision that will allow this suite to run like this in free-running mode? Or have I hit upon a combo of incompatible settings?

Thanks in advance,
Ella

[grenville]

Hi Ella

Some changes were made 8 weeks ago to u-by395 in the fix_ics branch - see https://code.metoffice.gov.uk/trac/rmed/ticket/300 – I’ve not looked hard at what’s been done…

Grenville

[shakka]

Grenville, you genius. I never would have found that tidbit of info - thanks! I’ve made a copy from the correct revision and it seems to be running fine.

Just a note though that the latest revisions of u-by395 don’t have the option to run on archer2 (only ncas-cray-xc30) so I had to update the following files to get it to work:

bin/setup_metadata
meta/rose-meta.conf
opt/rose-suite-ncas.conf

and made the folder /site/ncas-cray-ex/ with the suite-adds.rc and suite-vars.rc files.

I’ve committed these changes in my new suite u-cq759 so it should run okay at revision 236230.

Cheers!
Ella

[grenville]

Ella - I think not! Glad it’s working.

We are working to get our ARCHER2 additions into the trunk.

Grenville

[shakka]

Hmm. Have gotten to the first ec_recon step and encountering some more failures - nothing particularly useful in the pe_output or job.out files, even with extra diagnostic messages switched on in RCF_PRINTSTATUS.

Any idea how to extract a little more detail from this error message? The input ancillaries are the same that are working without hitch in my re-initialised run and it’s the first cycle point, so I am struggling to figure out what the input list in question could be.

???
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 4001
? Error from routine: CHECK_IOSTAT
? Error message:
? Error reading namelist EXCEPTIONS
? IoMsg: A READ operation tried to read past the end-of-file.
? Please check input list against code.
? Error from processor: 0
? Error number: 0
???

[grenville]

Ella

I suspect it’s read past the end of SHARED

FILE_MANAGER: Assigned : SHARED (see
FILE_MANAGER:          : Unit :  11 (fortran)

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 4001
?  Error from routine: CHECK_IOSTAT
?  Error message:
?        Error reading namelist EXCEPTIONS
?        IoMsg: A READ operation tried to read past the end-of-file.
?        Please check input list against code.
?  Error from processor: 0
?  Error number: 0
????????????????????????????????????????????????????????????????????????????????

which probably means there’s a mismatch between the namelist(s) created by Rose and those expected by the model. This suite is UM vn12.2 – u-co447 was UM vn12.0 (did that one work?)

Grenville

[shakka]

ahaaa. u-co433 is the one that’s running (I used u-co447 to create my n2560 ancils). Currently having some issues with the Arctic suite u-cq635. But that’s by the by.

cq758 and cq759 (Arctic/Antarctic free-running suites, respectively) are both 12.2 whereas the re-initialised suites are 12.0.

How can I make the namelists match (or is there no quick fix for that?)

[shakka]

Got to the bottom of it:

there are two extra entries in the SHARED namelist within glm_um/rose-app.conf@L70 at vn12.2

[file:SHARED]
source=(namelist:nlcfiles) namelist:nlstcall namelist:ancilcta namelist:temp_fixes namelist:jules_temp_fixes namelist:carbon_options namelist:coupling_control namelist:model_domain namelist:planet_constants namelist:run_dust namelist:run_glomap_aeroclim namelist:run_ukca namelist:run_gwd namelist:run_vera namelist:run_murk namelist:run_convection (namelist:run_comorph) namelist:run_bl namelist:run_precip namelist:run_radiation namelist:run_cloud namelist:run_aerosol namelist:lam_config namelist:run_ozone namelist:run_free_tracers namelist:run_eng_corr namelist:gen_phys_inputs namelist:run_dyn namelist:run_dyntest namelist:easyaerosol namelist:jules_model_environment namelist:jules_surface_types namelist:jules_surface namelist:jules_radiation namelist:jules_hydrology namelist:jules_rivers (namelist:jules_overbank) namelist:jules_water_resources namelist:jules_sea_seaice namelist:jules_soil namelist:jules_vegetation namelist:jules_irrig namelist:jules_soil_biogeochem namelist:jules_snow (namelist:jules_urban) namelist:run_stochastic namelist:run_electric namelist:ancilcta (namelist:items(:)) namelist:exceptions

Have amended this and it is now running. Looks like there are a few differences between vn12.0/12.2 app/um/rose-app.conf namelists too, but I’ll tackle those when I get there.

Thanks for your help! E

[bassett]

Hi Grenville / Ella, we’re also experiencing issues when setting free running to True. Receive a mod is undefined error.

Ella - is it ok to take a look at your suite to see the changes you made to get it to work?

Grenville - when will the archer2 fix for this be released?

Much appreciated

Richard

[shakka]

Hi Richard,

Of course! I’m still working through a couple of fixes to the namelists, but I’ll pop a comment here to let you know when I’m done and commit the changes for you.

Cheers,
Ella

[bassett]

Amazing thank you!

[shakka]

Hi Richard,
I’ve just committed a version of suite u-cq759 which is running at vn12.2 on archer2 in free-running mode. There were a lot of finnicky namelist changes to get it to run so hopefully this saves you some effort!
Best wishes,
Ella

[bassett]

Brilliant this is really great and will certainly save us considerable time!

I had a quick test and it looks to be missing the /site/ncas-cray-ex/ folder, wondering if its not synced across? Can’t see it in https://code.metoffice.gov.uk/trac/roses-u/browser/c/q/7/5/9/trunk (apologies if I missed it)

Thanks
Richard

[shakka]

Woops, well spotted - forgot to fcm add it! Will correct that now…
E

[shakka]

Hmm. Unfortunately after 30 mins I get an error in the fcst like this:

Error from routine: set_thermodynamic
Error message: A total of 25 points had negative mass in set_thermodynamic. This indicates the pressure fields are inconsistent between different levels and the model is about to fail.

@Grenville, have I missed something when copying stuff over from u-by395 do you think? I’ve clearly done something!

Cheers
Ella

[grenville]

Hi Ella

I don’t think this is a problem with something missing from the suite – more some numerical instability in the model. There are several queries on our old helpdesk relating to set_thermodynamics errors, see for example #3136 (negative mass in set_thermodynamic) – Old CMS Website, where the problem seemed to be resolved by shortening the time step. Perhaps try that.

Grenville