Is there a way of changing a suite so that it runs somewhere else than /work/n02/n02/tetts? For example, I’d like to run a whole bunch of related suites in /work/n02/n02/tetts/test/job1, /work/n02/n02/tetts/test/job2 etc. I think I need to do two things:
Modify the --chdir directive in archer2.rc in the [[HPC]] block.
Change where the various ancillaries are copied/linked too.
I have tried 1 but that then causes fcm_make_um2 & fcm_make2_pp to fail with ‘submit-failed’. I think this is because they have relative filenames for their stdout and stderr log files. For example:
As the directories do not exist, slurm submission fails :-(.
So is there a way of changing the cylc-run to /work/n02/n02/tetts/cylc-run? This seems to be created automatically and populated with all its directories etc. Presumable rose expects to look there for log files etc. Or is there a better way?
That would be fine, and I am quite happy for cylc to put its files where ever it wants. And many of the cylc related files are appearing in /work/n02/n02/tetts/cylc-run/suite_name - presumably because $HOME gets redefined. But the UM suite is defining the output log files as cylc-run/suite_name etc [relative file names]. So in the unedited suite they appear relative to /work/n02/n02/tetts (for me) which works and means all cylc stuff is where it is wanted. But if I chdir somewhere else slurm falls over as the directories have not been constructed.
So, how are the filenames for the output logs constructed in fcm_make_um2 & fcm_make2_pp?
If I understand that, I can change em!
Hi Ros,
I am trying to modify it to -chdir=/work/n02/n02/{{ARCHER2_USERNAME}}/test/utest with the idea that the model runs in that directory putting all its output etc there…
Simon
The --output and --error log paths are constructed by cylc.
You can’t use the --chdir to influence where model data ends up.
If you want the model output data (pp files, dumps, etc) saved to a location outside of the ~/cylc-run/suite-id directory structure you will need to move it as part of the workflow. E.g. postproc can be used to “stage/save” the model output data to somewhere else on ARCHER2.
Hi Ros,
OK – that is probably a better way to proceed. I assume I need my own postproc script or can I simply modify the existing one? Either way, how do I do that? Is there a general housekeeping script I can run at the end – that can clean up the ‘crap’ in cylc-run/suite-id so I don’t run out of disk space.
Hi Ros,
That looks like what I need. Is there some documentation on postproc_2.3_archer2 and housekeeping? I am probably going to start afresh with a slightly different and up-to-date AMIP configuration and make it produce netCDF output.
Simon
When you’ve decided on which suite you are going to use let me know as it may be using a different version of postproc which will require different ARCHER2 branches.
The housekeeping app uses standard Rose functionality (rose_prune — Rose Documentation 2019.01.8 documentation). You can see what directories it is pruning (deleting) and on what frequency in the file: /home/n02/n02/tetts/roses/u-db898/app/housekeeping/rose-app.conf
For example in u-bd898: prune-work-at=-P9M is saying delete the work directory (ie. ~/cylc-run/u-bd898/work/<cycle-point>) when they are 9months prior to the current cycle point.
Hi Ros,
thanks a lot. My disk storage is dominated by share. If I add:
prune{share}=-P0D
would that delete everything in share? I assume that the archiving has already run before this step…
Yes, I appreciate it might not be wise – restarting/continuing the run would be tricky!
If I want restartability then maybe prune{share}=-P9M would be sensible. (But how cylc works out which share/data/History_data is more than 9 months old is hard to see…)
Simon
History_data can’t be managed by rose prune as it has no idea what files the UM needs. Postproc will handle deleting of superceded files and when a file is archived it is deleted from the History_data directory. On/off switches for deletion of superseded files can be found e.g. for atmosphere in panel postproc → Atmosphere → archiving. I notice you need to switch archive_switch to True to enable archiving.
You could also put the entire cylc /share and /work directories on to the SCRATCH disk so it doesn’t use your /work quota. To do this add the following 2 lines to the top of the ~/roses/<suiteid>/rose-suite.conf file:
Perfect. And then I archive what I need at the end of the run. pp/netcdf files and most recent dump.
And if on scratch then the scratch cleaner will save my day!
Doesn’t matter when you add it, the important thing is to make sure the rose edit GUI isn’t open at the same time otherwise if you hit save in the GUI again it will overwrite any changes you have made to the files directly in the meantime.