Porting RAS to ARCHER2

I’m working on porting the RAS suite to ARCHER2, making use of the ants singularity container. The working suite for this is u-cq149, and the trac ticket is 338.

For the ancil_topographic_index step I need a topographic_index.nc input file. Is there already a suitable file available on ARCHER2 that I can use, or do I need to get this from the Met Office?

thanks,
Doug

Hi Doug

That file is not on Monsoon as far as I can see.
it may need to come from /data/users/hadzc/RMED/ANTS

Grenville

Doug

Copied topographic_index.nc to /work/y07/shared/umshared/ancil/atmos/master/soil_data/hwsd/v4

Grenville

Doug

Just to let you know - this isn’t the right place to put topographic_index.nc – I’m not sure where is at the moment, so it may not remain here.

Grenville

hi Grenville,

Thanks for this. I’ll try with the file in this location - but will make a note it might move in future.

cheers,
Doug

hi Grenville,

I’m getting this error when I try accessing the topography netcdf file:

OSError: [Errno -101] NetCDF: HDF error: b'/work/y07/shared/umshared/ancil/atmos/master/soil_data/hwsd/v4/topographic_index.nc'
Exception ignored in: <function CFReader.__del__ at 0x2acd8c5db290>
Traceback (most recent call last):
  File "/opt/conda/envs/ants/lib/python3.7/site-packages/iris/fileformats/cf.py", line 1129, in __del__
    self._dataset.close()
AttributeError: 'CFReader' object has no attribute '_dataset'

Could you check the netcdf file? Ncview won’t open the file for me either, so I wonder if it might be corrupted?

thanks,
Doug

Hi

I’m also in need of this file (adding ANTs ancillary generation for RA3 suite driven by ERA5) any luck in finding a working version? I get the same error as Doug for my qrprarm.hydtop generation. I also can’t view in xconv and ncdump returns unable to open file errors

Never mind, I found a working copy in Doug’s file space (thanks Doug!). The last piece of the puzzle is getting the SRTM orography generation to work in my suite now is this working in your suite Doug?

Hi all,

Yes - Claudio Sanchez shared his topography file with me (which I just moved across to ARCHER2 today :wink: ). I’ve asked if he’s okay with this file being put in the shared data space as well, so it’s more accessible for everyone.

He is also asking the data owners of the SRTM orography data if he can share this with me. If we get a positive answer from them, I’ll ask if these also could be put into the shared data space here as well.

cheers,
Doug

1 Like

Apologies Doug – this one slipped through the net – glad you got the file!

1 Like

Ah great, I’ll keep an eye out.

@grenville on shared space note, is there a better ants_python_path to use than my own anaconda ants installation path?

There is a container with ANTS which you can use at:
$UMDIR/bin/python_ants

This may help

1 Like

thanks, I’ll give it a whirl. Makes the suite more shareable if I don’t have pointers to user space in it so I’ll test if everything still works smoothly pointing to that instead!

@douglowe I tried giving your suite a run yesterday - it looks like you’ve done a lot, which is great. Is it finished, or are you still looking for the file: SRTM_concat.nc in Regn1_resn_1_ancil_mean_orog ? I was stuck here yesterday, with just a handful of apps left to run.

I admit that I pinched a few other files, so I mention this in case you or Helen are looking for feedback when making things available to others.
Files: c4_percent_1d.nc , qrparm.orog , topographic_index.nc
Scripts: ancil_lct_postproc_moruses.py
Directories: ancil_master_cap
Executable: central_ancillary.exe (This is available at $UMDIR/CAP9.1/build/bin/central_ancillary.exe if you want to use this centrally installed version)

I don’t know if this information is helpful, but I hope it may be. Thanks again, Dave

hi Dave - I’m still waiting for the SRTM files from Claudio, but other than that I think we’re close to being sorted (for this iteration of the tool set, at least).

Feedback on what user actions are needed to enable the workflow is very helpful - I realise there’s a lot of fudges I’ve made to try to get this to work, a number of which I’ve already forgotten about, so being reminded of these is very helpful :wink:

Regarding the ancil_lct_postproc_moruses.py script. This isn’t in the ANTS container yet - Claudio has said that this (and another script?) aren’t likely to make it into the ANTS release quite yet, but they are being used still in the workflow. Would it be possible for these extra scripts in be included in the next iteration of the ANTS container, or should we download these separately still?

I’ve setup my workflow (suite u-cq149) to use the ANTS container - I modified the python_env script to do this. The one step which I’ve not got working yet is the initial download of the extra scripts from the FCM server.

I would probably suggest not modifying the container, but we could install them somewhere and then provide the full path to the running script (ie replace ancil_lct_postproc_moruses.py with [/work/some location]/ancil_lct_postproc_moruses.py) ?

I will ask my colleagues about shared spaces on Archer, as we aren’t supporting this and it seems like a temporary approach until ANTS gets updated. If you think that this is a better solution than downloading extra scripts in the suite, then let me know.

Having a shared space for these scripts would be a better solution than downloading the scripts from the suite itself I think - as the combined download is near ~1Gb, for the sake of only 1 or 2 scripts. Plus this is the one step in the process I’m not sure how to include at the moment - it needs fcm for the download, which I guess could be run on puma, and then the relevant files copied to archer2 during setup - but adapting the workflow to do this would require a reorganising of the step order (I think?), which would be quite disruptive.

The two sets of extra code that are downloaded are:
ANTS_FCM_URL=fcm:ancil_ants.xm_br/dev/claudiosanchez/r10519_RAS_ants_v0p16p0@10623
CONTRIB_FCM_URL=fcm:ancil_contrib.xm_br/dev/claudiosanchez/r10581_RAS_contrib_v0p16p0@10872

I think we probably only need the CONTRIB pack though.