# Postproc nemo memory problem with UKESM1
**URL:** https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-memory-problem-with-ukesm1/1242
**Category:** Unified Model
**Tags:** ARCHER2
**Created:** [7 November 2023 08:53 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-memory-problem-with-ukesm1/1242 "2023-11-07T08:53:49Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@mwadley](https://cms-helpdesk.ncas.ac.uk/u/mwadley)
#### Post date: [7 November 2023 08:53 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-memory-problem-with-ukesm1/1242/1 "2023-11-07T08:53:49Z")
Hello Ros,
Perhaps you can help me with this?
I am trying to run a copy of the UKESM1 pre-industrial run (u-bc964, my version u-da655). The model runs fine for 5 quarters, but on the fifth quarter postproc nemo fails with the error (in job.err)
```auto
Lmod is automatically replacing "cce/15.0.0" with "gcc/11.2.0".
Due to MODULEPATH changes, the following have been reloaded:
1) cray-mpich/8.1.23
[WARN] file:atmospp.nl: skip missing optional source: namelist:moose_arch
[WARN] file:nemocicepp.nl: skip missing optional source: namelist:moose_arch
[WARN] file:atmospp.nl: skip missing optional source: namelist:script_arch
[WARN] file:nemocicepp.nl: skip missing optional source: namelist:script_arch
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=4795310.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.
```
It looks to as if the process hasn’t been allocated sufficient memory to create the annual means, but is OK for the seasonal means. I found that ‘sebsteinig’ (see below) had the same error, and increased the memory accordingly. However, I still get the memory error, even with 20G allocated. I found that I needed to insert the extra memory request after the model build, as otherwise the file is overwritten. I also can’t find anywhere to set the memory in the rosie GUI to incorporate it into the build.
Any ideas?
Thanks,
Martin
> [@Postproc\_nemo failure out of memory](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116/6):
>
> UPDATE: problem solved! The ARCHER2 Helpdesk got back to me and said they applied a patch to the Lustre filesystem over the past few days. I tested this with several suites and the postproc now seems to be running fine again on the serial nodes! One additional thing I had to to do was to request a bit more memory for the postproc jobs (the default is just below 2GB). 10 GB works fine for me, but I did not test other values. If anybody runs into the same OOM error, I just added the memory requ…
---
### Author:  [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)
#### Post date: [7 November 2023 08:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-memory-problem-with-ukesm1/1242/2 "2023-11-07T08:54:15Z")
Hi Martin,
We’ve also seen it OOM sometimes, even with the memory increased - it’s nemo\_rebuild. One of my colleagues has been in touch with ARCHER2, but I’m not sure what state that’s at.
Our advice at the moment is to run it on a compute node.
Regards,
Ros.
---
### Author:  [@mwadley](https://cms-helpdesk.ncas.ac.uk/u/mwadley)
#### Post date: [7 November 2023 08:54 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-memory-problem-with-ukesm1/1242/3 "2023-11-07T08:54:26Z")
Hi Ros,
Thanks. I’ve been grepping around looking for any reference to the queue used for postproc nemo, but drawn a blank. Could you point me in the right direction as to where to change the queue please, or where it is documented?
Thanks,
Martin
---
### Author:  [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)
#### Post date: [7 November 2023 08:55 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-memory-problem-with-ukesm1/1242/4 "2023-11-07T08:55:12Z")
Hi Martin,
You have to look at the inheritance for the task to figure these type of things out.
So
postproc tasks inherit form POSTPROC (suite.rc)
POSTPROC inherits from POSTPROC\_RESOURCE (suite.rc)
POSTPROC\_RESOURCE inherits from HPC\_SERIAL (site/archer2.rc)
HPC is the family that sets the queue to standard so you need to change POSTPROC\_RESOURCE to inherit HPC rather than HPC\_SERIAL in site/archer.rc
And remove the `--mem` option
Can you please also give us permission to read your puma2 home directory so it’s easier for us to help in future.
Thanks
Cheers,
Ros.
---
### Author:  [@system](https://cms-helpdesk.ncas.ac.uk/u/system)
#### Post date: [7 December 2023 08:56 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-memory-problem-with-ukesm1/1242/5 "2023-12-07T08:56:11Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.