Nested ukca suite on archer2

Unified Model
,
1

Hello
I have a nested ukca suite that runs fine on monsoon (u-cl515). And i would like that to run on archer2 for the polarres project.

David Case has produced a nesting suite version of u-by395_archer2 that works on archer without ukca.

I have copied across all of the ancil and start files from monsoon to archer for use with ukca in the nesting suite.

I have made a new suite (u-co330) from u-by395 at revision 224502 because later revisions have a problem with ukca free running.
I modified u-co330 following David Case’s changes for u-by395_archer2 and pointed it to the ukca files. That suite compiles and makes the ancillaries required for nested ukca.

Unfortunately it has a segmentation fault on the first timestep of the glm_um run. If i use the same um exe in a copy of u-by395_archer2 that runs fine. And when i’ve tried the exe from u-by385_archer2 that has crashed with a segfault too.

So this suggests that there is a problem with the suite.
Is there any special measures that are taken for the global amip/cmip setups to deal with extra memory per node for dealing with ukca?

I’d appreciate any ideas that could point me in the right direction.

thanks.

2

Hi Paul

As a complete shot in the dark, try running on more processors 8x10 is pretty small for n216. We’re running n216 on 32x32

Grenville

3

Hello Grenville

Thanks – but no luck with 32x32.

If I underpopulate the tasks on a node will I get more memory per task?

As it seems to be using slurm is there any directives to make it use more memory per task?

Is there a 12.0+ amip suite that I could look at to see if I can see anything?

Cheers

Paul

4

Running on 32x32 gave 8x the memory cf 8x10.

To under populate increase the number of OMP threads - that’s in site/ncas-cray-ex/suite-adds.rc

and/or set NCPU_PER_NODE to something less than 128 (but I’ve not actually tried any of this in this suite)

{% set NCPU_PER_NODE = 128 %}
…

    [[{{DRV_MOD["name"]}}_um]]
        init-script = """ 
                         export HOME=$DATADIR
                         module load cray-netcdf/4.4.1.1
                         module load cray-hdf5/1.10.0.1
                         module list 2>&1             
                      """

{% set HYPERTHREADS = 1 %}
{% set OMP_NUM_THREADS = 1 %}
{% set MPI_TASKS_PER_NODE = (NCPU_PER_NODE * HYPERTHREADS / (OMP_NUM_THREADS))|int %}
{% set TASKS_PER_NUMA = (MPI_TASKS_PER_NODE / 8)|int %}
{% set RCF_NPROCY = 4 %}
{% set RCF_NPROCX = 3 %}
{% set TOTAL_MPI_TASKS = (RCF_NPROCX * RCF_NPROCY)|int %}

5

Hello Grenville

Unfortunately still no luck.

Changing omp_num_thread=4 didn’t fix it.

I will have to start with prints to track it….

~WRD0000.jpg

6

hmm does sound like a bug - no core file, nor ATP trace! – i’d like to have a look too.

7

Thanks – u-co330 is committed

~WRD0000.jpg

8

Paul

Rebuild and include fcm:um.xm/branches/grenvillelister/vn12.0_all_ntp_fix

The compiler doesn’t like all_ntp not being dimensioned explicitly in ukca_main1-ukca_main1.F90

Grenville

9

Hello Grenville

Thanks – I was just tracking it down in ukca_main1 with prints….

I’ll give it a go!

Cheers

Paul

~WRD0000.jpg

10

argh - I mistyped the url - it needs a dev

fcm:um.xm/branches/dev/grenvillelister/vn12.0_all_ntp_fix

11

Hello Grenville

No problem.

It looks like its working for global and regional.

I seem to have a path issue for the regional – so when I’ve fixed that I can make sure its working across the scales and then its hopefully done!

Thanks again!

Cheers

Paul

~WRD0000.jpg

12

Hello Grenville

Just to confirm – got my version of the nested suite with ukca and different microphysics all running now.

Thanks to both you and David for getting it set up!

Cheers

Paul

13

Hi Paul

Thanks for letting us know.
Please remind me where your aero and ukca ancils are - I’ll copy them centrally.

Grenville

14

Hi Both,

Thanks for tracking down this issue! I’ll add this to the UKCA bugs list

https://www.ukca.ac.uk/wiki/index.php/Bugfixes_(new_approach)

Bugs found in UKCA can also be emailed to ukca_bugs@metoffice.gov.uk .

Best wishes,
Luke

15

Hello Grenville

These are the additional paths I added

source=/work/n02/n02/frfp/ancils_for_archer/ancil_master_aero/NS/RA2_AERO_ANCILS/aero_clims/GA6.0_antie/L90_40km_20YR/v2

source=/work/n02/n02/frfp/ancils_for_archer/ancil_ukca_aero/hadzm/ancils/n512e_mon_gcal/ncdf_ukca

source=/work/n02/n02/frfp/ancils_for_archer/ancil_ukca_glm1/hamgo/ancils/ukca-v10p3/n216e_mon_gcal/ncdf_ukca

so just everything in /work/n02/n02/frfp/ancils_for_archer

cheers

Paul

~WRD0000.jpg