Hi Emma,
It is best to do this in a systematic way, and try each potential optimisation one at a time, otherwise you won’t know whether the changes are working and whether they are improving or degrading performance, since multiple changes may cancel each other out.
So first collect your base runtimes - how long does it take to run at the moment with no changes. If you have times for multiple cycles it is useful to take an average, and look at min/max times as there can be a lot of varaibility in runtimes on Archer2. And I would switch on those timers I mentioned when you are testing the changes.
To help optimise the I/O you need to know how much data you are writing out each cycle. How much STASH data? How many different streams? And how large are the dumps?
Striping
There is a bit about striping on the Archer2 website here: I/O and file systems - ARCHER2 User Documentation
To stripe a directory run the following:
lfs setstripe -c -1 <dir-name>
This means that any new files written to the directory will be striped. It won’t change any files that are already in there. If you want to do that you need to move the files out then copy back in.
But striping will only benefit large files. If there are a mix of small and large files being written to the same directory, you may want to try and split them out somehow. Also as your suite writes output to a new directory every cycle, you will need to stripe the whole cycle directory or build something in to stripe certain directories each cycle. I can help with this but, you need to test whether striping is actually helping.
IO server
There is documentation on the UM IO server here: Met Office (Login)
My suite is u-cf432. Most of the IO server settings are under “IO system settings → IO server” in the GUI. It is best to use the GUI so you can see information about what each setting is doing. You also need to select a number of IO server processes in the rose-suite.conf file. I have 256 which is 4 IOS nodes which is a lot - you could start with fewer than that. I am then using 32 tasks per server. So I have 8 IO servers in total, with each server using half a node. The IO server works by assigning pp streams to each servers, so there is no point in having more servers than streams.
The variable ios_offset is set to 6016 which is the number of ranks I’m using for the atmosphere, and the ios_spacing of 1, means that all my IOS processes are together at the end on their own nodes. This might not be what you want to do, but in my case the IO server seemed to need a lot of memory. The usual case is to space the IO server processes evenly throughout the atmos processes.
Optimising the model and understanding performance is a complicated topic, and you will need to run a few experiments to figure out what is best for your configuration.
Annette