Cfplot 1D land-only data

General
,
21

Hi Andy
I tried out your script for the Annual and Monthly datasets, and it works fine, but for the sif_vars dataset, which has canopy-layered variables, the interactive job dies after a few minutes, after using 150GB of memory, according to top. I think it only processes one of the layered variables (gpp = gross primary productivity). I have 10GB left in my homes quota, but it is only writing a 500MB file for the 1 gpp variable, so 10GB is not the limiting factor right now.

This is what I did:

ssh sci3
cd ~pmcguire/from_andyh
module load jaspy
source /home/users/ajh/cfview_install/bin/cfview_setup.sh
python cf1dland_v1.py /work/scratch-pw2/pmcguire/u-cx688/JULES-GL7.0.vn5.2_sif1.CRUNCEPv7SLURM.S2.sif_vars.1869.nc
ncinfo converted.nc
mv converted.nc JULES-GL7.0.vn5.2_sif1.CRUNCEPv7SLURM.S2.sif_vars.1869_2D.nc

Is there anything I am doing wrong? You said this had worked for you.

Furthermore, I think we should put your code in MOSRS and/or under version control. I think it is quite useful. At some point, I will try to persuade someone on the JULES team to put code for cf-compatible 1D-landonly Input/Output into JULES. But already your code makes it supereasy to look at 1D-landonly data with cfplot or cfview. How long till DavidH can fix the slow loading issue? (i.e., How long till cf-python 3.15.2 is released?)
Patrick

22

Hi Patrick, the sif_vars dataset looked to be working okay to me for the first two variables but then I logged out of jasmin as I shut down my laptop. I’ll set another job running now and see if it completes and how long it takes. I’m using the GWS so the reading / writing process won’t be fast but at least it will have enough disk space to complete the task.

It’s fine to put the code in MOSRS / under version control.
I will check with David and see when we could expect a 3.15.2 release.

23

David and Sadie are hoping for the end of June release. If it doesn’t make that release it’ll be the end of July.

24

Hi Patrick,
the sif_vars file is 44GB and after 9 hours on sci6 converted to 11GB. ncdump on the converted file looks good to me and I’ll check that all looks okay in cfview when I’m in the office tomorrow.

25

Hi AndyH:
That’s great that it worked for you.
I wonder why it didn’t work for me.
Do you think there is a way to make the processing significantly faster than 9 hours?

I am glad that DavidH and Sadie can get the speed up to the cf read() code in by the end the June or by the end of July. Is this specific to the reading of 1D landonly compressed-by-gather data?

I will try to figure out how to get your Python code checked in to MOSRS. Maybe just as a standalone piece of code? (rather than as part of a suite.)
Patrick

26

Hi Patrick,
Using sci6 and /work/scratch-nopw just now took 18 minutes to convert the same file.

I gather there was a bug deep in cfdm that is the primary cause of the slowness. I’ll check with David if it is specific to 1D landonly compressed-by-gather data.

27

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

28

cf-python 3.15.2 is now available and delivers a large speedup on reading 1D land only data.

29

yay!
What did you and DavidH need to do to accomplish this speedup?
Patrick

30

Hi Patrick, David did all the cf-python work. He’s in today so you could ask him what he needed to do. There was nothing needed in cf-plot as cf-python presents the 1D land only data as a 2D lon-lat array which cf-plot understands already.

31

Hi Andy
Is it still ok for me at some point in the future to put this 1D land only code from you in some accessible repository somewhere (like MOSRS)? Did you update the code at all?
Patrick