Dear NCAS-CMS Helpdesk folks,
This is just to post email-exchange that flags some “gotcha” glitches that people may encounter
when converting the standard MONSOON v13.9 UKESM1.1 AMIP suite (u-ds836) to 2015+
It’s in the form of a series of email exchanges, among the ULeeds SEE folks with Mohit Dalvi.
But this should also help provide some useful info for the UKESM community as well.
Best regards,
Cheers
Graham
Dr. Graham Mann
Lecturer in Atmospheric Science (Univ. Leeds)
–
From: Graham Mann G.W.Mann@leeds.ac.uk
Date: Tuesday, 20 January 2026 at 13:12
To: Ross Herbert R.J.Herbert@leeds.ac.uk, Mohit Dalvi mohit.dalvi@metoffice.gov.uk, Daniel Grosvenor D.Grosvenor@leeds.ac.uk
Cc: Masaru Yoshioka M.Yoshioka@leeds.ac.uk, Saffron Heddell eesgh@leeds.ac.uk
Subject: Re: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Hi Ross,
Thanks for this. And Mohit’s reply indicates this non-gregorian ancil issue can remain hidden in initial months.
(Or only become terminal for the run later)
Re: the NCAS-CMS helpdesk — yes, I use this, and did already refer to this.
But I tend to use it only when running the UM on ARCHER2.
I guess some things will be common across systems, but I was assuming since it ran for several months it was not a general issue.
I’ll add this email-exchange as a post for MONSOON3 and check again if there are other MONSOON3 posts there.
Cheers
Graham
—-
From: Mohit Dalvi mohit.dalvi@metoffice.gov.uk
Date: Tuesday, 20 January 2026 at 10:56
To: Ross Herbert R.J.Herbert@leeds.ac.uk, Graham Mann G.W.Mann@leeds.ac.uk, Daniel Grosvenor D.Grosvenor@leeds.ac.uk
Cc: Masaru Yoshioka M.Yoshioka@leeds.ac.uk, Saffron Heddell eesgh@leeds.ac.uk
Subject: RE: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
OFFICIAL
Thanks Ross,
Yes, this is due to using timeseries ancils in 360-day calendar (which is how all CMIP6 ones have been created by default) in a Gregorian calendar run. The calendar indicator in these files is set to ‘undefined’ so the UM does not immediately complain about using these ancils, but it is likely the time stamps read in are slightly misinterpreted / displaced right from the start, leading to the time interpolation error further on.
From the pe_output it looks like the model was reading data for STASHcode 96, which is Chl-a as Ross has mentioned and you might encounter this for DMS, land_cover, veg ancils as well.
See if NCAS helpdesk has any solutions, or others have created 365-day versions of these, otherwise you will have to switch to climatological (cyclic) versions for near-future runs, e.g. the /projects/ukesm/Jeremy.walton.mon/ancil/MEDUSA and other u-by791 files used in standard UKESM1.1 Atmos runs.
Cheers
--
Mohit C. Dalvi (he/him)
Earth System and Mitigation Science Team
Met Office Hadley Centre
Fitzroy Road, Exeter
EX1 3PB, U.K.
--
From: Ross Herbert R.J.Herbert@leeds.ac.uk
Date: Tuesday, 20 January 2026 at 10:42
To: Graham Mann G.W.Mann@leeds.ac.uk, Mohit Dalvi mohit.dalvi@metoffice.gov.uk, Daniel Grosvenor D.Grosvenor@leeds.ac.uk
Cc: Masaru Yoshioka M.Yoshioka@leeds.ac.uk, Saffron Heddell eesgh@leeds.ac.uk
Subject: RE: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Graham,
I’ve had that error in the past when I was using non-gregorian ancils – in my case these were for DMS and CHL.
Ps – do you use https://cms-helpdesk.ncas.ac.uk/? You might find your questions have already been answered!
Ross
Ross Herbert
Post Doctoral Research Assistant
Institute for Climate and Atmospheric Science
University of Leeds
R.J.Herbert@leeds.ac.uk
From: Graham Mann G.W.Mann@leeds.ac.uk
Sent: 20 January 2026 10:35
To: Mohit Dalvi mohit.dalvi@metoffice.gov.uk; Daniel Grosvenor D.Grosvenor@leeds.ac.uk
Cc: Masaru Yoshioka M.Yoshioka@leeds.ac.uk; Ross Herbert R.J.Herbert@leeds.ac.uk; Saffron Heddell eesgh@leeds.ac.uk
Subject: Re: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Hi Mohit,
Thanks for this.
I realised I had missed-out to change the cmip6_stratos strat-aerosol optical properties files
(I had re-set the SAD files to future, but not the aerosol optical properties)
There was also a typo in the filename for the SAD, which I then fixed.
And also the usual issue with re-setting the number of months and start-year for the SAD.
But once I’d remedied those, the v13.9 ERA5-nudged transient-AMIP UKESM1.1 suite then ran OK (u-dw217) (for the specified 3 months).
Thanks a lot for your help with this.
One last thing, and I don’t know if this might be connecting with the MASS outage, but when I set a copy of this suite to run-on for a longer period (2-years in this case).
This “longer-run copy" of u-dw217 is suite u-dw274, and it gave an error in Aug 2017 after running 7-8 months:
(see error message below)
I guess it’s reasonably likely others may well have also encountered the same “later ANCIL error" below?
(Is there some other anvil-error that only “becomes a problem” after completing 6-7 months of a run — or in Aug/Sep?)
I’m aware also there is a MASS outage at the moment, and not sure if this may be part of the problem?
(only Jan to May 2017 seem to have been archived, the jun17, jul17 monthly-mean files have not been archived yet, still at /data/users/graham.mann.ext/cylc-run/u dw274/run1/share/data/History_Data)
Mohit — Do you know if is this is another “gotcha” caused by some glitch with STASH requests/settings or issue?
(I haven’t yet configured the run for “real-month” STASH and this might be the issue I guess).
Note that I cribbed a bit off Masaru’s v13.9 UKESM1.1 AMIP nudged run, which has some differences in boundary conditions (and doesn’t use the time-varying greenhouse gas settings).
Masaru (or Ross/Dan) — did you encounter this or similar issue when running your v13.9 nudged suite on for longer?
Best regards,
Cheers
Graham
Error msg is in the log-file, during August 2017:
/data/users/graham.mann.ext/cylc-run/u-dw274/run1/log/job/20170801T0000Z/atmos_main/01
???
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 507
? Error from routine: UP_ANCIL
? Error message: REPLANCA: TIME INTERPOLATION ERROR
? Error from processor: 239
? Error number: 81
???
[239] exceptions: An non-exception application exit occured.
[239] exceptions: whilst in a serial region
[239] exceptions: Task had pid=68948 on host nidd1392
[239] exceptions: Program is “/home/users/graham.mann.ext/cylc-run/u-dw274/run1/share/fcm_make_um/build-atmos/bin/um-atmos.exe”
Warning in umPrintMgr: umPrintExceptionHandler : Handler Invoked
gc_abort (Processor 239): um_abort called
From: Mohit Dalvi <mohit.dalvi@metoffice.gov.uk>
Date: Friday, 16 January 2026 at 17:30
To: Graham Mann <G.W.Mann@leeds.ac.uk>, Daniel Grosvenor <D.Grosvenor@leeds.ac.uk>
Cc: Masaru Yoshioka <M.Yoshioka@leeds.ac.uk>, Ross Herbert <R.J.Herbert@leeds.ac.uk>, Saffron Heddell <eesgh@leeds.ac.uk>
Subject: RE: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Hi Graham,
Apart from UKCA, the Easyaerosol module also uses NetCDF files for stratospheric aerosol properties.
Since the error is seen before any messages from UKCA it can be assumed that it is the easyaerosol input files causing this. The [namelist:easyaerosol]easyaeroosol_dir needs to be set to ‘$ANCILDIR/ atmos/GC5/n96e/easyaerosol/cmip6_stratos/future_2014-2100/v1’ for running a 2015+ period.
Best wishes
--
Mohit C. Dalvi (he/him)
Earth System and Mitigation Science Team
Met Office Hadley Centre
Fitzroy Road, Exeter
EX1 3PB, U.K.
From: Graham Mann <G.W.Mann@leeds.ac.uk>
Sent: Friday, 16 January 2026 16:37
To: Daniel Grosvenor <d.grosvenor@leeds.ac.uk>; Mohit Dalvi <mohit.dalvi@metoffice.gov.uk>
Cc: Masaru Yoshioka <m.yoshioka@leeds.ac.uk>; Ross Herbert <r.j.herbert@leeds.ac.uk>; Saffron Heddell <eesgh@leeds.ac.uk>
Subject: Re: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Hi Mohit,
cc: Dan et al.
I promise I won’t keep this thread going, but thought good idea still for this reply here.
That did fix the issue there with the UP_ANCIL number of headers.
It now gives similar error message re: the netCDF files exceeding the number of fields or so:
???
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 1983
? Error from routine: UKCA_NC_EMISS_MOD:GET_EMFILE_REC
? Error message: Invalid time record 2, End-of-file reached ?
? Error from processor: 0
? Error number: 73
???
I can see this is subroutine “get_emfile_rec” from within UKCA_NC_EMISS_MOD.
And I noticed there, the error message says “Error code: 1983”
I checked the log file "dw217.fort6.pe000” at the /data/scratch directory
/data/scratch/graham.mann.ext/cylc-run/u-dw217/run1/work/20170101T0000Z/atmos_main/pe_output/
And then it seems to have proceeded through the full set of UM ancillary files OK, but then is failing here within the UKCA netCDF emissions code part-way through the 1st timestep.
The “1983” might just be coincidence, but I’m wondering if maybe this indicates the emissions could still somehow be thinking the year is 1983?
Could there be another switch (for UKCA) that I’ve missed when changing the start-date from 1982 to 2017?
Or is there an over-write from some other file in Rose for standard configs
(e.g. that was set for the standard UKCA nudged run?)
I noticed your reply mentioned the UKCA netCDF files:
“This is only relevant for files read in via the UM ancillary system ([namelist:items])
and not the NetCDF ones as used for UKCA or Easyaerosol.”
And then not sure if you were indicating there might yet be 1 or 2 other similar settings issues to resolve?
I did check for any subsequent commit-points in Dan G’s Rose repository entry/branch for that suite dr373 (that you sent the change-set link for), but that was the latest commit-point from that suite.
Is there another settings-modification here, for the multi-decadal UKCA emissions ancils ?
Thanks for any help or pointers you can give here.
Best regards,
Cheers
Graham
** Dr. Graham Mann, Lecturer in Atmospheric Science **
** Institute for Climate & Atmospheric Science T: +44 0113 3431660 **
** Room 10.108, School of Earth & Environment F: +44 0113 3435259 **
** University of Leeds, Leeds, LS2 9JT, U.K. E: G.W.Mann@leeds.ac.uk **
From: Daniel Grosvenor <D.Grosvenor@leeds.ac.uk>
Date: Friday, 16 January 2026 at 13:59
To: Graham Mann <G.W.Mann@leeds.ac.uk>, Mohit Dalvi <mohit.dalvi@metoffice.gov.uk>
Cc: Masaru Yoshioka <M.Yoshioka@leeds.ac.uk>, Ross Herbert <R.J.Herbert@leeds.ac.uk>, Saffron Heddell <eesgh@leeds.ac.uk>
Subject: RE: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Thanks Mohit. Yes, hopefully that should work Graham – good luck!
Cheers,
Dan.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Daniel Grosvenor,
Research Fellow,
Institute for Climate and Atmospheric Science (ICAS),
University of Leeds,
Tel: +44 (0)113 343 8668
From: Graham Mann <G.W.Mann@leeds.ac.uk>
Sent: 16 January 2026 12:54
To: Mohit Dalvi <mohit.dalvi@metoffice.gov.uk>
Cc: Daniel Grosvenor <D.Grosvenor@leeds.ac.uk>; Masaru Yoshioka <M.Yoshioka@leeds.ac.uk>; Ross Herbert <R.J.Herbert@leeds.ac.uk>; Saffron Heddell <eesgh@leeds.ac.uk>
Subject: Re: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Hi Mohit,
Thanks so much for this. And ah, OK.
I can see from the diffs there in the link you sent, from Dan’s u-dr373 suite,
That he increased nancil_lookupsa from the same 50,000 value in the suite there.
And I’ve just increased this by the same factor-20 as Dan did there (to 1,000,000)
Thanks again for your help with this,
Best regards,
Cheers
Graham
[graham.mann.ext@cazccylc1 u-dw217]$ fcm diff
Index: app/um/rose-app.conf
— app/um/rose-app.conf (revision 343935)
+++ app/um/rose-app.conf (working copy)
@@ -114,7 +114,7 @@
l_amipii_ice_processing=.true.
l_snow_assim=.false.
l_sstanom=.false.
-nancil_lookupsa=50000
+nancil_lookupsa=1000000
use_lookup_dates_anc_time_interp=.true.
[namelist:carbon_options]
From: Mohit Dalvi <mohit.dalvi@metoffice.gov.uk>
Date: Friday, 16 January 2026 at 11:48
To: Graham Mann <G.W.Mann@leeds.ac.uk>
Cc: Daniel Grosvenor <D.Grosvenor@leeds.ac.uk>, Masaru Yoshioka <M.Yoshioka@leeds.ac.uk>, Ross Herbert <R.J.Herbert@leeds.ac.uk>, Saffron Heddell <eesgh@leeds.ac.uk>
Subject: RE: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Hi Graham
This error is usually encountered while adding a long timeseries ancillary, especially with multiple fields like that in a land surface ancillary. The parameter to modify is [namelist:ancilcta]nancil_lookupsa, e.g. see
https://code.metoffice.gov.uk/trac/roses-u/changeset/325586
For information the number of LOOKUPS represents the total number of 2-D slices of data (so 85 for a 3-D file, 27 for an UKESM land surface file) x number of time records in all the ancillaries being used.
This is only relevant for files read in via the UM ancillary system ([namelist:items]) and not the NetCDF ones as used for UKCA or Easyaerosol.
Best wishes
--
Mohit C. Dalvi (he/him)
Earth System and Mitigation Science Team
Met Office Hadley Centre
Fitzroy Road, Exeter
EX1 3PB, U.K.
From: Graham Mann <G.W.Mann@leeds.ac.uk>
Sent: Friday, 16 January 2026 11:10
To: Mohit Dalvi <mohit.dalvi@metoffice.gov.uk>
Cc: Daniel Grosvenor <d.grosvenor@leeds.ac.uk>; Masaru Yoshioka <m.yoshioka@leeds.ac.uk>; Ross Herbert <r.j.herbert@leeds.ac.uk>; Saffron Heddell <eesgh@leeds.ac.uk>
Subject: “Insufficient space for LOOKUP headers” error-msg in UP_ANCIL within v13.9 ERA5-nudged AMIP UKESM1.1 run on MONSOON3
Hi Mohit,
Hope all is good with you.
I’m just getting in touch re: some test runs I’ve been doing at UM v13.9 on MONSOON3.
I only started on this on Wednesday, but did a “rosie lookup” on MONSOON3, and saw your runs
included there an ERA5-nudged AMIP UKESM1.1 suite there — your suite u-ds836
I tried doing a “rosie copy” of that, and submitted it as a test run, and it seemed to work fine.
(my suite u-dw163) — it ran the initial 3 months it was set for, within 1982.
When that ran OK, I decided to proceed and configure the run to re-start instead from 1st Jan 2017,
and proceed towards enacting the interim spin-up steps equivalently to how I’d done this at v12.1.
When I tried that yesterday, u-dw195, it failed with the “PP HEADERS ON ANCILLARY FILE DO NOT MATCH”, which I realised was simply that the ancil files were set from the CMIP6 historical ancils up to 2014.
I’d been through this same workflow at v12.1, and had updated the job to re-start instead from the CMIP6 SSP2-4.5 future-scenario ancils (emissions, SSTs, seaice etc.).
And so I thought go ahead, and try that, as it would be expected to solve that glitch there (the ancils then would have the correct year-range etc.)
I did that, and proceeded to update all the ancils to match my v12.1 run, finding where these ancils were on MONSOON3, and adding these in, within the copy of the u-dw195 —> u-dw217.
This “future ancils updated” version of the ERA5 nudged job is u-dw217.
This seemed to then progress further — or at least now gives a different UP_ANCIL error mesage…
But this message I have not encountered before, or not at v12.1. It says:
???
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 14
? Error from routine: INITIAL_4A
? Error message: INANCILA: Insufficient space for LOOKUP headers
? Error from processor: 112
? Error number: 70
???
I remember we did encounter this similar error at v7.3 and v8.4, and usually this simply indicated there were too many STASH items requested — or the domain-profile & time-profile of the requested STASH output led to more memory than had been set by default.
My recollection is that there is an integer variable “NLOOKUP” or similar, for each input/output stream.
And at various points we had to increase NLOOKUP from whatever the default was, to get the model to then output the fields we wanted to write to file etc.
In this case, it looks like this error is being triggered from an "input data-stream” rather than an output data-stream.
But I’m guessing the remedy to the error-message is something similar to this.
Please can you let me know if there’s a straightforward change to one of the rose-app.conf or rose-suite.conf settings that solves this problem.
I’ve cc’d in Dan, Masaru and Ross as well in case they’ve encountered this issue
(or in case they may likely encounter it in the future..).
I’ve jotted down also some background info below
Thanks a lot for your help with this,
Best regards,
Cheers
Graham
Background info
For Saffron Heddell’s PhD (cc’d), we’ve resolved for Saffie to start her UM work from some initial simulations starting from the port of the Hunga HTHH-MOC exp2a runs I did with v12.1 UKESM1.1 AMIP runs, but at v13.9.
As we resolved from when Saffie & visited the MO back in April 2024, the Hunga runs we identified to be as closely aligned to the CCMI UKESM1.1 runs, and we referred to Luke’s MONSOON re-run of the 1st ensemble member of the REF-D2 runs.
Basically I re-started UKESM1.1 from the 1st Oct 2017 dump from that CCMI REF-D2 (u-da771), with da771 being Luke’s re-run of this, at v12.1 UKESM1.1, and on MONSOON2 .