# PiControl HadGEM3-GC3.1 suite **URL:** https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636 **Category:** Uncategorized **Created:** [30 January 2025 14:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636 "2025-01-30T14:14:48Z") **Posts on this page:** 20 **Page:** 1
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [30 January 2025 14:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/1 "2025-01-30T14:14:48Z")
Hello, I am trying to run my piControl HadGem3-HC3.1-LL suite u-dm681 on puma2. My suite is a copy of the standard job u-as037. I have changed the suite to run for 1 month (down from 500yr) and turned off a lot of the stash requests. Any other differences between the suites is simply my attempt to get the suite running (project names, user names etc). When I run the job, all of the fcm\_make\_\* files and recon succeed. But when it launches the coupled step the job failes. The job logs did not reveal much to help me identify the problem. The end of the job.err has the following: ??? ??? WARNING ??? ? Warning code: -100 ? Warning from routine: CHECK\_RUN\_DIFFUSION ? Warning message: ? cldbase\_opt\_sh set to sh\_wstar\_closure since ? Smagorinsky diffusion not chosen. ? Warning from processor: 0 ? Warning number: 9 ??? srun: error: nid003203: tasks 0-1,3-127: Aborted srun: launch/slurm: \_step\_signal: Terminating StepId=8589187.0+0 srun: launch/slurm: \_step\_signal: Terminating StepId=8589187.0+1 srun: launch/slurm: \_step\_signal: Terminating StepId=8589187.0+2 slurmstepd: error: \*\*\* STEP 8589187.0+0 ON nid003199 CANCELLED AT 2025-01-30T13:34:22 \*\*\* slurmstepd: error: \*\*\* STEP 8589187.0+2 ON nid003204 CANCELLED AT 2025-01-30T13:34:22 \*\*\* slurmstepd: error: \*\*\* STEP 8589187.0+1 ON nid003203 CANCELLED AT 2025-01-30T13:34:22 \*\*\* srun: error: nid003204: tasks 0-5: Terminated srun: Force Terminated StepId=8589187.0+2 srun: error: nid003199: tasks 0-49: Terminated srun: error: nid003202: tasks 149-197: Terminated srun: error: nid003201: tasks 100-148: Terminated srun: error: nid003200: tasks 50-99: Terminated srun: Force Terminated StepId=8589187.0+0 srun: error: nid003203: task 2: Aborted (core dumped) srun: Force Terminated StepId=8589187.0+1 [FAIL] run\_model \<\<‘ **STDIN** ’ [FAIL] [FAIL] ‘ **STDIN** ’ # return-code=143 2025-01-30T13:34:24Z CRITICAL - failed/EXIT Above this, there was a stream of errors of the form: MPICH ERROR [Rank 200] [job id 8589187.0] [Thu Jan 30 13:34:21 2025] [nid003203] - Abort(1) (rank 200 in comm 480): application called MPI\_Abort(comm=0x84000003, 1) - process 200 I have not yet successfully run the job yet. It appears to me like it can’t submit the job but I am not sure why. Any ideas on what might be causing this would be much appreciated, Penny ---
### Author: ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville) #### Post date: [30 January 2025 15:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/2 "2025-01-30T15:58:47Z")
please allow us read permission on your home and work spaces on ARCHER (and your space on puma) ```auto chmod -R g+rX /home/n02/n02/ chmod -R g+rX /work/n02/n02/ ``` (revert permissions on your .ssh directory & files) Grenville ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [31 January 2025 09:45 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/3 "2025-01-31T09:45:28Z")
Sorry. Permissions updated. ---
### Author: ![grenville](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/grenville/32/29_2.png) [@grenville](https://cms-helpdesk.ncas.ac.uk/u/grenville) #### Post date: [31 January 2025 16:25 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/4 "2025-01-31T16:25:55Z")
Penny u-as037 has not been run for some time and has a pre puma2 nemo working copy source. It looks like you changed some NEMO settings that caused the problem. We will commit the `nemo_sources` branch (you can use `/home/n02/n02/ros/nemo/branches/dev_r5518_GO6_package` for now) and revert `nemo_path_excl` to `NEMOGCM/NEMO/OPA_SRC/TRD/trdtrc.F90`. My copy of u-as037 with `/home/n02/n02/ros/nemo/branches/dev_r5518_GO6_package` is running okay. Grenville ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [3 February 2025 08:57 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/5 "2025-02-03T08:57:40Z")
Excellent. That was the problem and it is running now. I have decided to attempt to convert the piControl run to an SSP245 run. By comparing suites on trac I can see what changes are needed for the MO supercomputer. Is there a suite ported over to archer2 which has ozone redistribution which you could suggest I look at? I was told this might be tricky so it would be good to see how it is implemented. I would be also interested to hear about how porting it went. For example, was it hardware dependent? Or did the fix work much like on the monsoon? Many thanks, Penny ---
### Author: ![AnnetteOsprey](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/annetteosprey/32/27_2.png) [@AnnetteOsprey](https://cms-helpdesk.ncas.ac.uk/u/AnnetteOsprey) #### Post date: [3 February 2025 12:23 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/6 "2025-02-03T12:23:13Z")
Hi Penny, We have instructions for adding OR to a GC3.1 suite on Archer2: [Ozone redistribution](https://cms.ncas.ac.uk/unified-model/ozone-redistribution/) Have a go with that and let me know if you need any help. Annette ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [3 February 2025 14:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/7 "2025-02-03T14:54:53Z")
Thanks for the link Annette. **In terms of the ozone redistribution:** There are 2 files I don’t have which I was hoping someone might have access to: 1. The ozone file which have the following path name in the suite I am trying to recreate (see [https://code.metoffice.gov.uk/trac/roses-u/browser/b/j/6/1/6/trunk/rose-suite.conf](https://code.metoffice.gov.uk/trac/roses-u/browser/b/j/6/1/6/trunk/rose-suite.conf)) here is the path: n96e/ssp245/Ozone/v1/historic\_interpolated\_3d\_ozone\_n96e\_2015\_2099\_ants.anc 1. Orogorphy files in /work/y07/shared/umshared/hadgem3/ancil/atmos/ there is no N96 resolution. **Other changes (not zone related)** Would it be possible to port over the following file as well please? /data/d01/ukcmip6/ssp585\_N96O1\_ensemble1\_dumps/bg466o\_20150101\_restart\_trc.nc Thank you! Penny ---
### Author: ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher) #### Post date: [3 February 2025 15:24 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/8 "2025-02-03T15:24:09Z")
Hi Penny, The ozone file is in the CMIP6\_ANCILS directory on ARCHER2 here: /work/y07/shared/umshared/cmip6/ancils The other files I should be able to grab for you tomorrow. Regards, Ros ---
### Author: ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher) #### Post date: [4 February 2025 10:43 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/9 "2025-02-04T10:43:22Z")
Hi Penny, Orography files are now in `/work/y07/shared/umshared/hadgem3/ancil/atmos/`. There were a couple of N96 directories; I’ve guessed at `n96e_orca025_go6`. If that is not the correct directory please let me know. After a bit of a hunt I’ve managed to find the start dump directory on the old Met Office XCE/F. The files are on ARCHER2 under `/work/y07/shared/umshared/ukcmip6/ssp585_N96O1_ensemble1_dumps` Regards, Ros. ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [4 February 2025 10:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/10 "2025-02-04T10:44:05Z")
Amazing. Thank you! ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [6 February 2025 09:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/11 "2025-02-06T09:14:44Z")
I am still working on getting the piControl suite working. Lots of my questions have already been asked and answered in other channels. Which has been very handy. The suite it running up until pptransfer. I have followed the steps for setting up the Globus transfer and that went okay. There is limited information in the job.error for why the transfer if failing. It fails not long after starting the run. I have run it twice to make sure it was repeatable. Here is the gist of the error: [WARN] Failed to generate checksums. [ERROR] Checksum generation failed. [FAIL] Command Terminated [FAIL] Terminating PostProc… Full path on archer2 for the error is here /work/n02/n02/penmaher/cylc-run/u-dm681/log.20250204T172957Z/job/18500101T0000Z/pptransfer/03/job.err Thank you! Penny ---
### Author: ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher) #### Post date: [6 February 2025 12:35 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/12 "2025-02-06T12:35:19Z")
Hi Penny, If you are wanting to automatically transfer data over to JASMIN you will need to go through the Globus setup instructions here: [Configuring PPTransfer using Globus](https://cms.ncas.ac.uk/unified-model/pptransfer-globus/) And the suite will also need to be modified as it hasn’t been updated for use with Globus. - In panel _fcm\_make\_pp → Config → pp\_sources_ Change the revision number of the `postproc_2.3_pptransfer_gridftp_nopw` branch from `4557` to `5411`. - In file `app/postproc/rose-app.conf` add the following variables to the `[namelist:pptransfer]` section and set `gridftp=false`: I think that should then work. Regards, Ros. ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [6 February 2025 12:56 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/13 "2025-02-06T12:56:01Z")
Hi Ros, Thanks for the response. I have done these steps already. I am running with postproc\_2.4 as I plan on including ozone redistribution. Penny ---
### Author: ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher) #### Post date: [6 February 2025 15:02 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/14 "2025-02-06T15:02:55Z")
Ok. I’ll take a look at you suite and get back to you. The separate checksumming step is not used for Globus so there is some switch not set properly. But one quick question. When you went from postproc\_2.3 to postproc\_2.4 how did you do the upgrade? Did you run the `rose app-upgrade` command or just change the branch names? Cheers, Ros. ---
### Author: ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher) #### Post date: [6 February 2025 15:17 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/15 "2025-02-06T15:17:21Z")
Hi Penny, Try turning `verify_checksums` off in postproc. That can only be used for rsync transfers. In fcm\_make\_pp set the meta to be: `archive_and_meaning/fcm_make/postproc_2.4` to pick up the fcm\_make\_pp metadata rather than the postproc app metadata. Cheers, Ros. ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [10 February 2025 10:21 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/16 "2025-02-10T10:21:42Z")
Hi Ros, Thanks. The suite is now running. I am now implementing the ozone redistribution changes. Penny ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [10 February 2025 10:37 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/17 "2025-02-10T10:37:36Z")
> [@RosalynHatcher](#): > > n96e\_orca025\_go6 Hi Ros, the files for orography in `n96e_orca025_go6` → n96 is what I was looking for, I assume orca025 mean 0.25 grid. Could you tell me what go6 means please? I am looking to recreate the HadGEM3-GC3.1-N96ORCA1 set up. ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [11 February 2025 11:51 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/18 "2025-02-11T11:51:54Z")
Hi Ros, The orogorphy files are not quite what I am looking for. Would it be possible to get the N96 ORCA1 GO6.0 please? Penny ---
### Author: ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher) #### Post date: [12 February 2025 09:33 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/19 "2025-02-12T09:33:53Z")
Hi Penny, Do you have the full Met Office path to the file please? There isn’t an ORCA1 directory where the other files are. GO6 is Global Ocean 6 Regards, Ros. ---
### Author: ![penmaher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/penmaher/32/437_2.png) [@penmaher](https://cms-helpdesk.ncas.ac.uk/u/penmaher) #### Post date: [12 February 2025 12:11 UTC](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636/20 "2025-02-12T12:11:42Z")
Hi Ros, Thanks for the response. I am trying to port suite u-bj616 which runs at the met office. In that suite the orog files are as follows: OZONE\_SHARE = $CYLC\_SUITE\_SHARE\_DIR/ozone\_redistribution OROGRAPHY\_INPUT=$OZONE\_SHARE/qrparm.orog REMOTE\_PROJECTS\_LINK = /hpc/common\_xcs source=$REMOTE\_PROJECTS\_LINK/um1/ancil/atmos/n96e/orca1/orography/globe30/v6/qrparm.orog I can’t find where CYLC\_SUITE\_SHARE\_DIR is defined but I am guessing the source line, if it still exists is where they are. Penny [Next page](https://cms-helpdesk.ncas.ac.uk/t/picontrol-hadgem3-gc3-1-suite/1636.md?page=2)