Thanks for adding the extra resource - I think this ticket can be closed now.
For these 101-node 6+ hr UM jobs, around 64 GB of profile data is generated and that’s with summarisation, which fortunately is turned on by default (export PAT_RT_SUMMARY=1).
Along with the “PAT_RT_MPI_THREAD_REQUIRED=3” setting that I mentioned earlier, I also needed to set “PAT_RT_PARALLEL_MAX=10000” in order to improve the likelihood that the instrument UM wouldn’t abort part way through.
I’ve just relaunched the suite using “pat_build -g mpi”, which should profile the MPI routines only.
Cheers,
Michael