JDMA failing

Unified Model
1

Hi,

My JDMA tasks for two suites have been failing with the error:

TypeError: ‘int’ object is not iterable
[FAIL] jdma.py # return-code=1

I recently upgraded both suites to CYLC8, u-dr800 has completed successful JDMA tasks after the upgrade, while u-dr928 is on its first cycle.

I don’t have any idea what might be causing this error, is anyone able to help please?

Thanks, Andrea

2

Hi all,

Just bumping this: does anyone know if JDMA is working as it should, and if other suites are completing JDMA tasks successfully? Is this likely to be a CYLC8 upgrade issue, despite the tasks previously working after the upgrade?

Thanks,
Andrea

3

Hi All,

Just an update, I have found the issue and my jdma tasks are now running.
This issue was that I had changed

[namelist:jasmin_arch]

copy_streams=‘’‘p5’‘’

copy_target=‘’'/gws/nopw/j04/stabilisation/‘’’

to

copy_streams=‘’‘5’‘’

as I still hadn’t found the right setting for this parameter. All streams are copied to JASMIN, instead of just stream 5. However, the integer in copy_streams caused the issue with the task failing.

4

Hi Andrea,

Really sorry; this slipped through my net. The copy_streams functionality doesn’t control what streams are transferred from ARCHER2 to JASMIN. It is a mechanism to leave some streams on JASMIN disk rather than put them all to ET and then have to immediately extract a subset off of tape. It was originally implemented to enable monitoring of key variables as a suite is running.

Cheers,
Ros.

5

Hi Ros,

No problem, thanks for the explanation! Is there a way to define which streams are copied to JASMIN? At the moment it copies them all, which fills up my GWS very fast and I then need to periodically delete the streams I don’t need. It’s not a huge problem, but if there is a mechanism to do this it would be useful!

Cheers,
Andrea

6

Hi Andrea,

You’ll need to configure postproc so that it only process/archives the streams that you want. E.g. see postproc → Atmosphere panel. Switch process_all_streams to false and specify the atmos streams you do want to keep/archive. Or configure the model so it doesn’t even produce the streams you’re aren’t interested in.

If you don’t have space on GWS to temporarily hold your data before it’s put to ET and then auto-deleted from disk, I would suggest you use the JASMIN transfer cache instead.

Cheers,
Ros.

7

Hi Ros,

Thanks for your reply. My config is already using the JASMIN transfer cache, and I do want to keep all streams on tape, but I was hoping to copy only a specific stream to disk for immediate use (rather than bringing it back from tape). It sounds like it’s not possible to do that though (i.e. all streams to tape, but only one stream to disk)?

Thanks for your help!
Andrea

8

Hi Andrea,

Sorry - I feel we’ve just gone full circle. :rofl: That is exactly the what the copy_streams is to be used for. All data is transferred from ARCHER2 to JASMIN xfer cache and then before JDMA runs it copies a subset of those streams to GWS.

Let me just look at the code; I can’t remember how I coded it.

Cheers,
Ros.

9

Oh good to confirm - thanks!

I’ve raised a previous ticket which Grenville answered ( Postproc - copy_streams ), but somehow following these instructions hasn’t worked for me

Thanks!
Andrea

10

Hi Andrea,

It’s not you… it my dodgey code! :zany_face: I obviously didn’t test just a numerical stream. :roll_eyes:

I’ll raise a ticket to remind myself to fix this in the next set of postproc changes I make. For now a quick fix is to specify ‘x5’, ‘5x’ or similar to force python to interpret it as a string rather than an int. Choose a character that you don’t have a stream for.

Cheers,
Ros.

11

Thanks Ros!

Just out of curiosity, why would ‘5x’ or ‘x5’ be interpreted differently than ‘p5’? I’d used p5 which was recognised as a string (i.e. jdma task didn’t crash) but it copied all streams to disk which wasn’t what I was looking for. Will ‘5x’ only copy stream 5?

Thanks for investigating!
Andrea

12

Hi Andrea,

The code takes each character in turn in the copy_streams variable and then does a glob on *.p{}* so specifying a p5 results in the 2 globs *.p5* and *.pp* , the latter matching every single .pp file :rofl: Oooops!!! :exploding_head:

Cheers,
Ros.

13

Thank you Ros for the temporary fix and explanation, it all works as expected now! :smiley:

Cheers,
Andrea