Jobs not launching, constraint=ivybridge128G

JULES
21

Hi Martin:
Did you get it working yet?
Patrick

22

Hi Patrick,

I didn’t get a chance to try again this week, working on this big grant for August. I will try and have another crack next week once I have more words on the page and feel I can multi-task again. I appreciate you following up, be assured I will be following this up.

Martin

23

Hi Patrick,

OK running now, I will update soon.

M

24

Hi,

It failed at the plotting stage again. I’ve checked and all the sites worked fine so I’ve triggered the plotting script again, it has just been submitted.

M

25

It looks like you’re using a different version of cylc than I am.

Your job script for make_plots:
~mdekauwe/cylc-run/u-co635v2/log/job/1/make_plots/02/job
has:
export CYLC_DIR='/apps/jasmin/metomi/cylc-7.8.7'

whereas a recent version that I think worked for me:
~/cylc-run/u-al752currentworking8/log/job/1/make_plots/01/job
has:
export CYLC_DIR='/apps/contrib/metomi/cylc-7.8.1'

Maybe you can fix this by changing your .bashrc file so that it has:
export PATH=/apps/contrib/metomi/bin:$PATH
instead of:
export PATH=$PATH:/apps/contrib/metomi/bin

then:

  1. log out of cylc1 and log back in
  2. cd ~/roses/u-co635v2
  3. rose suite-run --reload. (or maybe you need to do rose suite-run --restart)
  4. rose sgc
  5. retrigger your make_plots app, by right clicking on it and choosing the retrigger option
  6. grep CYLC_DIR= ~/cylc-run/u-co635v2/log/job/1/make_plots/03/job (or whatever version you are using instead of 03)
  7. the grep should maybe say (if it doesn’t, then maybe something didn’t work as planned):
    export CYLC_DIR='/apps/contrib/metomi/cylc-7.8.1'
  8. maybe it will run properly then. Please let me know.
  9. if it doesn’t work, I might then suggest starting the suite new again, either with rose suite-run --new or by making/running a new copy of the suite with rosie or maybe just with a linux command-line copy.

Patrick

26

Hi Martin:
Is it working better now?
Patrick

27

Hi Patrick,

I just checked and no luck, it failed at the plotting step again. So, that should rule out differences in the cycl, as I’ve matched that step and I started a brand new run.

Thanks,

Martin

28

Hi Martin:
Maybe you can try putting some print statements especially at the beginning of the file:
/home/users/mdekauwe/cylc-run/u-co635v2/bin/make_plots.py
in order to ensure that the Python interpreter is actually properly getting into this file and then crashing?

After you put the print statements in there, you can retrigger the make_plots app in the cylc GUI. I don’t think that you need to do a rose suite-run --reload. The output of the print statements will be in the job.out log file:
/home/users/mdekauwe/cylc-run/u-co635v2/log/job/1/make_plots/NN/job.out
I think the NN version of the job.out file is the latest version.
Patrick

30

Hi Martin
Did you ever get this working?
Patrick

34

Hi Martin
I noticed an issue with make_plots for the u-al752 suite on JASMIN. Since your ticket has been closed, I just reopened the ticket, to respond properly.
The make_plots app fails for some weird reason unless this line is deleted in the [[PLOTTING_CEDA_JASMIN]] section of site/suite.rc.CEDA_JASMIN :
env | grep LD_LIBRARY_PATH
Does that help you?
Patrick

35

Hi Patrick
Will test and let you know.
Martin

36

Hi Patrick,

I can confirm that it the whole suite ran for me when that line was removed.

Thanks,
Martin

37

Hi Martin:
I am very glad to hear that it works now!
The jinja language used in Rose/Cylc must be doing something that I don’t currently understand when the env | grep LD_LIBRARY_PATH is executed for an empty LD_LIBRARY_PATH path. I think that this line of code is not absolutely necessary, so I am deleting it now from the code for the u-al752 suite. The checked-in MOSRS version of this suite now doesn’t have this line of code.
There is a similar line of code elsewhere in the suite’s environment settings, where LD_LIBRARY_PATH contains actual directory names, and that seems to work fine.
Patrick