# Nested ukca suite on archer2 **URL:** \ **Category:** Unified Model\ **Tags:** Monsoon2, ARCHER2\ **Created:** [1 June 2022 07:32 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604 "2022-06-01T07:32:46Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [1 June 2022 07:32 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/1 "2022-06-01T07:32:46Z")
Hello I have a nested ukca suite that runs fine on monsoon (u-cl515). And i would like that to run on archer2 for the polarres project. David Case has produced a nesting suite version of u-by395\_archer2 that works on archer without ukca. I have copied across all of the ancil and start files from monsoon to archer for use with ukca in the nesting suite. I have made a new suite (u-co330) from u-by395 at revision 224502 because later revisions have a problem with ukca free running. I modified u-co330 following David Case’s changes for u-by395\_archer2 and pointed it to the ukca files. That suite compiles and makes the ancillaries required for nested ukca. Unfortunately it has a segmentation fault on the first timestep of the glm\_um run. If i use the same um exe in a copy of u-by395\_archer2 that runs fine. And when i’ve tried the exe from u-by385\_archer2 that has crashed with a segfault too. So this suggests that there is a problem with the suite. Is there any special measures that are taken for the global amip/cmip setups to deal with extra memory per node for dealing with ukca? I’d appreciate any ideas that could point me in the right direction. thanks. ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [1 June 2022 08:15 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/2 "2022-06-01T08:15:14Z")
Hi Paul As a complete shot in the dark, try running on more processors 8x10 is pretty small for n216. We’re running n216 on 32x32 Grenville ---
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [1 June 2022 08:41 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/3 "2022-06-01T08:41:58Z")
Hello Grenville Thanks – but no luck with 32x32. If I underpopulate the tasks on a node will I get more memory per task? As it seems to be using slurm is there any directives to make it use more memory per task? Is there a 12.0+ amip suite that I could look at to see if I can see anything? Cheers Paul ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [1 June 2022 09:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/4 "2022-06-01T09:14:43Z")
Running on 32x32 gave 8x the memory cf 8x10. To under populate increase the number of OMP threads - that’s in `site/ncas-cray-ex/suite-adds.rc` and/or set NCPU\_PER\_NODE to something less than 128 (but I’ve not actually tried any of this in this suite) **{% set NCPU\_PER\_NODE = 128 %}** … ```auto [[{{DRV_MOD["name"]}}_um]] init-script = """ export HOME=$DATADIR module load cray-netcdf/4.4.1.1 module load cray-hdf5/1.10.0.1 module list 2>&1 """ ``` {% set HYPERTHREADS = 1 %} **{% set OMP\_NUM\_THREADS = 1 %}** {% set MPI\_TASKS\_PER\_NODE = (NCPU\_PER\_NODE \* HYPERTHREADS / (OMP\_NUM\_THREADS))|int %} {% set TASKS\_PER\_NUMA = (MPI\_TASKS\_PER\_NODE / 8)|int %} {% set RCF\_NPROCY = 4 %} {% set RCF\_NPROCX = 3 %} {% set TOTAL\_MPI\_TASKS = (RCF\_NPROCX \* RCF\_NPROCY)|int %} ```auto ``` ---
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [1 June 2022 12:50 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/5 "2022-06-01T12:50:40Z")
Hello Grenville Unfortunately still no luck. Changing omp\_num\_thread=4 didn’t fix it. I will have to start with prints to track it…. ![~WRD0000.jpg](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/52d5f1c08649212fe6768d2cacdcf8df711810d9.jpeg) ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [1 June 2022 13:08 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/6 "2022-06-01T13:08:36Z")
hmm does sound like a bug - no core file, nor ATP trace! – i’d like to have a look too. ---
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [1 June 2022 13:33 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/7 "2022-06-01T13:33:17Z")
Thanks – u-co330 is committed ![~WRD0000.jpg](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/52d5f1c08649212fe6768d2cacdcf8df711810d9.jpeg) ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [7 June 2022 08:49 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/8 "2022-06-07T08:49:09Z")
Paul Rebuild and include `fcm:um.xm/branches/grenvillelister/vn12.0_all_ntp_fix` The compiler doesn’t like `all_ntp` not being dimensioned explicitly in `ukca_main1-ukca_main1.F90` Grenville ---
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [7 June 2022 09:01 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/9 "2022-06-07T09:01:52Z")
Hello Grenville Thanks – I was just tracking it down in ukca\_main1 with prints…. I’ll give it a go! Cheers Paul ![~WRD0000.jpg](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/52d5f1c08649212fe6768d2cacdcf8df711810d9.jpeg) ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [7 June 2022 09:08 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/10 "2022-06-07T09:08:34Z")
argh - I mistyped the url - it needs a dev fcm:um.xm/branches/dev/grenvillelister/vn12.0\_all\_ntp\_fix ---
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [7 June 2022 10:45 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/11 "2022-06-07T10:45:49Z")
Hello Grenville No problem. It looks like its working for global and regional. I seem to have a path issue for the regional – so when I’ve fixed that I can make sure its working across the scales and then its hopefully done! Thanks again! Cheers Paul ![~WRD0000.jpg](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/52d5f1c08649212fe6768d2cacdcf8df711810d9.jpeg) ---
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [7 June 2022 14:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/12 "2022-06-07T14:55:32Z")
Hello Grenville Just to confirm – got my version of the nested suite with ukca and different microphysics all running now. Thanks to both you and David for getting it set up! Cheers Paul ---
**Author:** ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville)\ **Post date:** [7 June 2022 15:19 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/13 "2022-06-07T15:19:42Z")
Hi Paul Thanks for letting us know. Please remind me where your aero and ukca ancils are - I’ll copy them centrally. Grenville ---
**Author:** ![Luke](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/luke/32/45_2.png) [@Luke](https://cms-helpdesk.ncas.ac.uk/u/Luke)\ **Post date:** [7 June 2022 15:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/14 "2022-06-07T15:44:29Z")
Hi Both, Thanks for tracking down this issue! I’ll add this to the UKCA bugs list [https://www.ukca.ac.uk/wiki/index.php/Bugfixes\_(new\_approach)](https://www.ukca.ac.uk/wiki/index.php/Bugfixes_(new_approach)) Bugs found in UKCA can also be emailed to [ukca\_bugs@metoffice.gov.uk](mailto:ukca_bugs@metoffice.gov.uk) . Best wishes, Luke ---
**Author:** ![frfp](https://avatars.discourse-cdn.com/v4/letter/f/ea5d25/32.png) [@frfp](https://cms-helpdesk.ncas.ac.uk/u/frfp)\ **Post date:** [8 June 2022 06:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/15 "2022-06-08T06:57:10Z")
Hello Grenville These are the additional paths I added source=/work/n02/n02/frfp/ancils\_for\_archer/ancil\_master\_aero/NS/RA2\_AERO\_ANCILS/aero\_clims/GA6.0\_antie/L90\_40km\_20YR/v2 source=/work/n02/n02/frfp/ancils\_for\_archer/ancil\_ukca\_aero/hadzm/ancils/n512e\_mon\_gcal/ncdf\_ukca source=/work/n02/n02/frfp/ancils\_for\_archer/ancil\_ukca\_glm1/hamgo/ancils/ukca-v10p3/n216e\_mon\_gcal/ncdf\_ukca so just everything in /work/n02/n02/frfp/ancils\_for\_archer cheers Paul ![~WRD0000.jpg](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/52d5f1c08649212fe6768d2cacdcf8df711810d9.jpeg) ---
**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/1fd2411499ffcbc299fe756cd5cdf26e44956558.png) [@system](https://cms-helpdesk.ncas.ac.uk/u/system)\ **Post date:** [1 November 2023 08:47 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/16 "2023-11-01T08:47:00Z")
---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [2 November 2023 22:22 UTC](https://cms-helpdesk.ncas.ac.uk/t/nested-ukca-suite-on-archer2/604/17 "2023-11-02T22:22:18Z")