UKESM1.1 coupled suite running extremely slowly on ARCHER2 after Cylc 8 migration

Unified Model
1

Dear CMS team,

I am experiencing a severe performance issue with a UKESM1.1 fully coupled pre-industrial workflow on ARCHER2.

The workflow details are:

Parent suite: u-cj511;

Current workflow: u-eb425;

Model: UKESM1.1 coupled;

UM version: vn12.1;

Platform: PUMA2 and ARCHER2;

Workflow version: Cylc 8.

The workflow was copied from the official UKESM1.1 pre-industrial suite u-cj511 and migrated from Cylc 7 to Cylc 8 using the NCAS migration guidance.

Following the migration, I also had to update several obsolete ARCHER2 environment settings. The original configuration produced errors because cce/12.0.3 and GC3-PrgEnv/2.0/94430 were no longer available. Thus, I am currently using cce/15.0.0 and GC3-PrgEnv/v1.

The workflow (Cylc 8 update was validated) now completes the build and reconfiguration tasks successfully, and the coupled model starts without an immediate scientific or numerical error. However, a 130-minute coupled job (a 10-day simulation) advances only about 27 model hours before being terminated with:

CANCELLED DUE TO TIME LIMIT.

Could you please help me to check whether the current cce/15.0.0, GC3-PrgEnv/v1, and MPI/network environment is appropriate for UKESM1.1 vn12.1 on ARCHER2, and advise on how to solve the current issue?

Relevant runs and job IDs include:

u-eb425/run4: 14776906, 14833040

u-eb425/run5: 14857282

u-eb425/run6: 14869675

I am sorry for providing several runs with the same issue, as I have tried to debug it myself.

Please let me know if you need any further information.

Kind regards,

Lucky

2

Hi Lucky

Could you please commit any local changes you have to u-eb425. The most recent commits on MOSRS still reflect Cylc 7 syntax.

Also you mention that the workflow is running slowly. Do you have a benchmark run speed in mind and if so where from?

Thanks

Jonny

3

Hi again Lucky.

I’ve discussed this with a colleague and there are actually much more recent, Cylc 8, workflows at this webpage :right_arrow: https://code.metoffice.gov.uk/trac/UKESM/wiki/UKESM1.1StandardJobs

I’ve checked and the UM version 13.6 workflows onwards should be already compatible with the present ARCHER2 setup.

All the best.

Jonny

4

Hi Jonny,

Thanks for pointing it out. I have used the fcm commit to upload my current copy to MOSRS. which is the Cylc8 version of u-eb425, which committed revision 365125.

Regarding the benchmark, I was referring to the NCAS page titled Performance of Standard UM Configurations: https://cms.ncas.ac.uk/unified-model/performance/.

It reports that the UKESM1 pre-industrial coupled suite u-bz764, using 32 × 18 × 2 atmosphere processes, 12 × 9 ocean processes, and an allocation of 9 + 1 + 1 nodes, completed 90 model days in approximately 1 hour and 41 minutes.

I understand that this is a UKESM1 vn11.2 benchmark rather than an exact UKESM1.1 vn12.1 comparison. However, u-eb425 advances only about 27 model hours in 130 minutes using a similar decomposition and node allocation.

Could you please advise what run rate should normally be expected for this UKESM1.1 configuration?

Best wishes,

Lucky

5

PS: Since you have offered me the recent suites, would you recommend that I use one of these newer workflows as the basis for my experiment, rather than continuing to debug the current UKESM1.1 vn12.1 workflow?

6

Hi Lucky.

yes, that would be a lot easier for you I am sure! :smiley:

Cheers

Jonny

7

It’s difficult to give a meaningful value run rate estimation since there are many variables involved. The workflow is N96ORCA1 which is (I think) the lowest supported resolution for the UKESM and it is set up to run on about 700 processors (ATM_PROCX * ATM_PROCY). At order of magnitude level I would expect order ~1 year/day, which matches the values in the link you pasted.

I would still encourage you to use the more recent model versions however since a lot can change in a few years, even on the same platform. :smiley:

All the best.

Jonny