# Postproc\_nemo failure out of memory

**URL:** <https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116>\
**Category:** Unified Model\
**Tags:** ARCHER2\
**Created:** [1 September 2023 09:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116 "2023-09-01T09:44:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sebsteinig](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sebsteinig](https://cms-helpdesk.ncas.ac.uk/u/sebsteinig)\
**Post date:** [1 September 2023 09:44 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116/1 "2023-09-01T09:44:48Z")

</div>

Hi,

My suite u-cz307 (GC3.1, UM 10.7) has problems with the ‘postproc\_nemo’ task after updating it to the new ARCHER2 OS.

This is the error in the log file:

```auto
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=4356300.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

```

The last line of the output log indicates that the problem occurs when running the `[INFO] command: main_pp.py nemo` script.

The weird thing is that the task sometimes works (about 1 out of 5 times). So I have to re-trigger the task several times (e.g. up to 10 times in this cycle: `/home/n02/n02/ssteinig/work/cylc-run/u-cz307/log/job/24100101T0000Z/postproc_nemo`) until it finally succeeds.

This only started to happen since I adjusted my old suite (u-cu066) to the new ARCHER2 OS (u-cz307) following this thread: [UM vn10.7 on the ARCHER2](https://cms-helpdesk.ncas.ac.uk/t/um-vn10-7-on-the-archer2/1081). All other tasks work fine.

This is the diff between both suites: [https://code.metoffice.gov.uk/trac/roses-u/changeset/HEAD/c/z/3/0/7/trunk?old=HEAD&old\_path=c/u/0/6/6/trunk](https://code.metoffice.gov.uk/trac/roses-u/changeset/HEAD/c/z/3/0/7/trunk?old=HEAD&old_path=c/u/0/6/6/trunk)

Do you have any recommendation on where I could start debugging the problem?

Thank you very much for your help.

Best wishes,  
Seb

---

<div class="post-metadata">

**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\
**Post date:** [21 September 2023 09:12 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116/2 "2023-09-21T09:12:59Z")

</div>

Hi Sebastien,

Did you manage to solve the problem?

We’ve seen OOMs with postproc too since the OS upgrade, at the moment the workaround is to run the task on a compute node rather than the serial node.

Regards,  
Ros.

---

<div class="post-metadata">

**Author:** ![sebsteinig](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sebsteinig](https://cms-helpdesk.ncas.ac.uk/u/sebsteinig)\
**Post date:** [21 September 2023 10:27 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116/3 "2023-09-21T10:27:48Z")

</div>

Hi Ros,

Thanks for your reply.

I contacted the ARCHER2 Helpdesk as I think this is an issue on their end. They acknowledged a lot of I/O problems at the moment and think the problem is caused by “a bug in the Lustre filesystem which causes intermittent read errors”.

They are currently working on a fix and I will copy any update I receive to this thread.

Best wishes,  
Seb

---

<div class="post-metadata">

**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\
**Post date:** [21 September 2023 10:52 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116/5 "2023-09-21T10:52:09Z")

</div>

Hi Seb,

Yes, we’re waiting on update when that fix is going to be applied to see if it does fix the problem. In the meantime using a compute node is the workaround.

Cheers,  
Ros.

---

<div class="post-metadata">

**Author:** ![sebsteinig](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sebsteinig](https://cms-helpdesk.ncas.ac.uk/u/sebsteinig)\
**Post date:** [6 October 2023 13:13 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116/6 "2023-10-06T13:13:49Z")

</div>

UPDATE: problem solved!

The ARCHER2 Helpdesk got back to me and said they applied a patch to the Lustre filesystem over the past few days.

I tested this with several suites and the postproc now seems to be running fine again on the serial nodes! One additional thing I had to to do was to request a bit more memory for the postproc jobs (the default is just below 2GB). 10 GB works fine for me, but I did not test other values.

If anybody runs into the same OOM error, I just added the memory request to the postproc resources in the `archer2.rc` configuration:

```auto
    [[POSTPROC_RESOURCE]]
        inherit = HPC_SERIAL
        pre-script = """
                     module load postproc
                     module list 2>&1
                     ulimit -s unlimited
                     """
        [[[directives]]]
            --mem=10G

```

Best wishes,  
Seb

---

<div class="post-metadata">

**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/1fd2411499ffcbc299fe756cd5cdf26e44956558.png) [@system](https://cms-helpdesk.ncas.ac.uk/u/system)\
**Post date:** [8 October 2023 13:14 UTC](https://cms-helpdesk.ncas.ac.uk/t/postproc-nemo-failure-out-of-memory/1116/7 "2023-10-08T13:14:40Z")

</div>

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.