Forcing nesting suite with ERA5

[grenville]

Ella

Finally found the LAM_INCLUSION error in u-co447.

Grenville

[shakka]

Hi Grenville,

Sorry for the slow reply. I managed to get the EC domain to run on its own using u-co447 (which is probably why you can’t see the error in there).

… and I’ve also gotten u-co433 to run without the LAM inclusion errors by re-running. Now I’m just getting WGDOS packing errors about soil moisture contents, which I’m investigating now…

Cheers,
Ella

[shakka]

Update: I turned off the verification stash package and it has fixed the error. I am wondering if the problem was related to the slt ancil that soil fields are read in from having in-filled data (and the fact that Antarctica has no soil…!) that caused the values to inflate.

[shakka]

Hi Grenville,

I managed to get around the above errors by correcting my inputs and produced some data from ERA5 atmospheric forcing. However, so far the simulations don’t include sea ice or SST, which is going to be important for the domains I’m running.

I had originally thought I could regrid sea ice and SSTs onto the ANCIL_0p11 ‘glm’ (actually the EC-cut out) domain in the recon step (or the 12km inner LAM domain - I guess it wouldn’t matter which if I was doing it this way), but this requires the data to match the target grid and the data I have are global, and at a different resolution.

So it seems like there are two options:

• cut out and regrid existing sea ice/SST data to fit the ANCIL_0p11 EC-domain, then reconfigure using the glm_um recon step
• download ERA5 data myself and make sure that it includes the standard forcing variables, plus sea ice and SST so that this is read in right from the outset
• (is there another way to do this?)

It’s a lot of work to convert everything to the right resolution and cut it to the right domain size, so I think it would be easier to do option 2. BUT: does the UM read in all the variables included in the ERA5 grib file, or is it looking for specifics? And if it’s looking for specifics, can I change the code somewhere to make sure it reads in sea ice and SSTs ?

Hope that all made sense. Your advice is always very much appreciated!

Cheers,
Ella

[shakka]

Hi again Grenville, I’ve attempted to download the data myself and am having a lot of issues getting the files data into the right format. I’ve not worked with grib much before and I feel like I’m smashing my head into a brick wall!

Ordinarily I would download the right ERA5 data straight from MARS as a grib but it’s down until October. I’ve found some netcdf data on model levels on CEDA and tried to convert it on jasmin but every method I’ve tried has done something else wacky to the data.

Do you have access to era5 data that you could send my way? I need T/q/u/v on model levels, and sst/sea ice/mslp/geopotential/Ts/lsm at the surface (i.e. standard forcing fields + sea ice/sst).

Alternatively, any other suggestions to get sea ice and sst into the model would be very helpful.

Hope you had a great bank hol. Cheers,
Ella

[grenville]

Hi Ella

I’m trying to get some insight into your problem from other CMS’ers. We’ve done similar things but not for many years, and the knowledge is buried. I used to have a huge amount of grib but not since we moved machines. Let me have a look around. ( How did you eventually crate the mask?)

Grenville

[shakka]

Thanks Grenville, that would be very helpful. I might have found a way to hack the grib metadata into the right format (using cdo setzaxis) but who knows whether that will get past the EC recon.

To get the mask/orography ancillaries I did the following (and wrote it into a document so it’s reproducible!):

1. Set up model to run with global UM analysis (n2560)
   a. Adapt the following files to permit running with n2560:

suite-runtime/dm.rc [insert @L95:

pre-script =“”"
{% if DRV_MOD[“name”] != “ec” %}
{% if DRV_MOD[“resln”] != “n2560” %}
. $ROSE_DATA/etc/ancil_versions_{{ DRV_MOD[“config”] }}
{% else %}
. $ROSE_DATA/etc/ancil_versions_n2560
{% endif %}
{% else %}
cd {{DRV_MOD[“ec_lam_ancil_dir”]}} && . ./ancil_versions && cd -
{% endif %}
“”"

• bin/setup_metadata [insert:

fp.write(‘values=“n96”,“n216”,“n320”,“n512”,“n768”,“n1280”,“n2560”\n’)]

• /site/ncas-cray-ex/suite-adds.rc [insert @L255:

{% if DRV_MOD[“resln”] != “n2560” %}
{% set MPI_TASKS_PER_NODE = (NCPU_PER_NODE * HYPERTHREADS / (2OMP_NUM_THREADS))|int %}
{% set TASKS_PER_NUMA = (MPI_TASKS_PER_NODE /2)|int %}
{% else %}
{% set MPI_TASKS_PER_NODE = (NCPU_PER_NODE * HYPERTHREADS / (24OMP_NUM_THREADS))|int %}
{% set TASKS_PER_NUMA = 1 %}
{% endif %}

• /site/ncas-cray-ex/suite-adds.rc [change @L280:

–time = 00:40:00

• [change @L352-353:

–{% set RCF_NPROCY = 6 %}
–{% set RCF_NPROCX = 5 %}

• [change @ L319:

–time = 03:00:00

• rose-suite.conf [insert:

dm_resln=”n2560”

(can also change in GUI after updating config files)

• app/install_cold/opt/rose-app-ncas-cray-ex.conf [insert @L77:

[file:$ROSE_DATA/etc/ancil_versions_n2560]
mode=symlink
source=/work/n02/n02/shakka/n2560e/ancil_versions

b. Add n2560 grid file app/glm_um/opt/rose-app-n2560.conf with contents:

[namelist:nlsizes]
global_row_length=4096
global_rows=3200
land_field=104103

[namelist:nlst_mpp]
extended_halo_size_ew=10
extended_halo_size_ns=10

[namelist:nlstcgen]
steps_per_periodim=720

c. Make sure the UM builds with Grenville’s local changes to his branch

/home/grenville/branches/vn12.0_all_ntp_fix

2. Set up to run with UM analysis. Check recon is expecting a UM file (not grib)

3. Make sure glm recon is set to take all input data (aerosol climatologies, lsm, orog etc.) from the startdump by selecting option 1 under each ancillary namelist entry.

(This might not work for all of them, e.g. albedo, dust climatology might have to be read in from ancil file or reset to a default value. This shouldn’t matter too much because in the first instance we are just creating the orog/lsm outputs that we can then turn into ancils. i.e. the science doesn’t have to make sense.)

4. Set up STASH outputs so that the um (LAM) outputs include all the relevant sub-grid fields you need for the orography ancillary (5,6,7,17,18,33,34,35,36,37), LSM (30) and the land fraction (505).

5. Some variables aren’t included in the startdump but are needed for the LAM UM recon to run (e.g. unfiltered orography, stash=7). Set this up to set this to zero (i.e. source = 3) in the UM recon.

6. Allow glm recon to run, and then LAM recon to run.

7. Locate reconfigured file cylc-run//share/cycle/// /<rg01_rs01_name>/ics/<rg01_rs01_name>_astart. Open with xconv and convert relevant lsm and orography variables into one netcdf for orog/lsm.

8. Use xancil to turn these into ancillaries.

N.B. You’ll need to debug errors along the way, e.g. updating LAND_FIELD to match expected values, adding / subtracting stash fields

Hopefully this is helpful for someone else along the way.

Cheers,
Ella

[grenville]

Ella

Doug Lowe has some ERA5 data in /work/n02/n02/lowe/CIDAR_Data/

Any good?

Grenville

[shakka]

Hi Grenville,

Definitely good, but unfortunately still no sea ice or SSTs included…!

Ella

[shakka]

Hi Grenville,

I’ve now got myself some grib data that contains sea ice and SSTs. I’ve now got two problems:

1. It looks like the recon doesn’t recognise the SST field (“Parameter 34 in GRIB file not found in lookup table”) although the skin temperature variable reads in okay so hopefully there should be some data for the ocean. Do you know where the lookup table for the grib conversion lives? I am thinking that I could update this to get around this error.

2. I am getting the same “LAM inclusion error” in the ec_um_recon as above (although interestingly with different longitude values now). I have double checked again for my own sanity and the ancil domain linked in dm_ec_lam_ancil_dir definitely contains ancils on a larger grid than the LAM ancils linked in rg01_r201_ancil_versions. So I am a bit baffled about where this is coming from. Could you help me get my head around this?

FYI the suite is u-co433.

Cheers,
Ella

edit: I’ve also checked out the edits in the ticket here which appears to show a related problem.

Looks like maybe somewhere during the rotation an extra 360 degrees is added (somewhere between L192 - 204)?

edit 2: I also tried increasing the size of the EC cut-out domain and am getting the same error. “Eastern boundary at 576.0313172 not within source LAM with 539.5000000”

[jeff]

Hi Ella

Could you run this again and this time set RCF_PRINTSTATUS to Extra diagnostic messages in um->env->Runtime Controls->Reconfiguration only.

Thanks.

Jeff.

[shakka]

Hi Jeff, Grenville,

I’ve managed to get to the bottom of this.

Seems like the error results from the domain of the ERA5 data that I have downloaded. I wonder if this might be due to differences in the convention of longitude naming between ECMWF and UM (i.e. +/- 180 vs 0-360 degrees).

Either way, the fix was to download global ERA5 data rather than data from a limited domain. This results in larger data volumes but it does at least run.

[shakka]

For anyone trying to do this after me, here is the method I have used:

Set-up to run with ERA5 data @vn12

1. Download forcing data from ECMWF

Data specification:

• Global grid
• Grib format
• Containing the following variables:
  • T, u, v, q on model levels (N.B. NOT pressure levels) – these are not available via Climate Data Store
  • LSM, orography, skin temp, surface pressure, sea ice fraction and SST on the surface
• Single file for each initialisation/cycle point containing all necessary variables

Downloading using python is fairly straightforward using cdsapi (check the details here)

During your retrieval you can either download directly into a single file if all the data you need are on the same tape archive (i.e. most recent years) or otherwise you can concatenate model level and surface fields into single grib files (one for each cycle).

2. Follow procedure here to produce ancils for a larger cut-out domain that the ERA5 data will be interpolated onto

3. FOR DOMAINS CONTAINING THE POLES: set glm recon to take SST and SIC from the ERA dump (i.e. set source = 1)

(note that all other fields should come from the ancil files)

[grenville]

Hi Ella

Great that you got this going and many thanks for the detailed notes - I’m sure they’ll be indispensable in the future!

Grenville