Out of memory error

JULES
1

Hello

I’m running JULES on JASMIN (suite u-ec531). I’m using a version of JULES 5.2 with the DGW groundwater model in it. The model needed a lot of spin up for the groundwater, so I was running a couple of spin up cycles (1980-2019) and then a main run (again 1980-2019). However, I’m getting an out of memory error during the spin up. I reduced the domain - I was running all of Africa (u-dg545) - and I turned off DGW (groundwater) and river routing, but I’m still getting the same error. I would be hugely grateful if anyone had any suggestions how I might avoid this error? Is there any way I can reduce memory consumption?

srun: error: host1039: task 0: Out Of Memory
[FAIL] rose-jules-run # return-code=255
2026-09-22T15:37:37Z CRITICAL - failed/EXIT

Many thanks in advance

Sarah

2

Hi Sarah.

Have you tried increasing the amount of memory available to the Slurm task using either of the --mem or --mem-per-cpu options to sbatch.

These can (presumably) be specified in the rose suite itself or at runtime in the Cylc web UI.

Cheers

Jonny

3

Thanks for the reply, Jonny.

No I haven’t tried --mem-per-cpu, but I doubled ‘ntasks’ from 20 to 40. Unless I am mistaken that doubled my memory from 20 to 40 GB? It made zero difference, as my model fell over at exactly the same point, which made me suspicious that the problem was with the model rather than the allocation of memory. However, I didn’t find anything obviously amiss in the outputs. Is this memory usage very high for a JULES run the size of mine or is it normal? It also didn’t seem to help that I substantially reduced my model extent from all of Africa to the Niger, which I wasn’t expecting.

image
image896×152 7.02 KB

5

Hi Sarah.

It looks like your latest run of the jules task in u-dg545 has succeeded?..

$ tail -1 /home/users/sarcol/cylc-run/u-dg545/runN/log/job/1/jules/NN/job.out

2026-09-11T17:39:57Z INFO - succeeded

Regarding changing the memory allocation, this isn’t filtering through into your running workflow.

These are submission directives to the same completed task as above…

$ cat /home/users/sarcol/cylc-run/u-dg545/runN/log/job/1/jules/NN/job\
> |grep -i sbatch

#SBATCH --job-name=jules.1.u-dg545/run7
#SBATCH --output=cylc-run/u-dg545/run7/log/job/1/jules/01/job.out
#SBATCH --error=cylc-run/u-dg545/run7/log/job/1/jules/01/job.err
#SBATCH --partition=standard
#SBATCH --account=terrafirma
#SBATCH --qos=high
#SBATCH --time=14:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=40

If your manual memory allocations were filtering through they’d show up in this script as something like #SBATCH --mem-per-cpu=1G.

Cheers

Jonny

6

Ah yes, that’s an older one from 11 September without spin up. One way to solve this problem is to just do the main run, save the final groundwater levels as the new initial groundwater levels and keep rerunning until the model is spun up. But it would be much easier if I could do, for example, NSPIN=3 and just leave it.

I’ll try setting the manual memory allocation now.

Thanks again

Sarah

7

I guess I’m just not really sure how much memory these runs should be using, so I don’t whether I’m not allocating enough or the usage is a sign something is wrong/badly set in the model…