JULES without MPI and without a Rose suite

JULES
1

Hi Patrick,

Thanks for engaging with ticket http://cms.ncas.ac.uk/ticket/3529 - I do very much appreciate it. Unfortunately, your tips and links don’t yet work for me. I think the problem is that I haven’t yet precisely specified the problem, so I’m going to try that in the attachment here.

In the attached, I demonstrate what I’m trying to do: I am trying to compile JULESvn6.0 without MPI and without using a Rose suite (which would introduce complications because of the various options of the suite.rc). For me, this works with gfortran on JASMIN, but not on ifort and my ticket above the other day was trying to ask why this is. I still think I’m just missing a path and/or compiler option somewhere, but I can’t work out where.

If you have time to skim through the attached, I would greatly appreciate it (it’s not long!). I seem to be very stuck, so any advice would be much appreciated!

Best,
Toby

  1. First, I’ll show you that this does work using gfortran:
  • · Start with a fresh download of JULESvn6.0 and a fresh session on JASMIN Cylc1 with no modules loaded either in the session or in my .bashrc (and I have my MOSRS password correctly cached).
  • · Type: module load jaspy
  • · I change etc/fcm-make/make.cfg line 22 so that instead of just loading in custom.cfg it does the following:
# Load environment variable pre-settings
include = $HERE/envars.cfg
$NETCDF_FORTRAN_ROOT = $CONDA_PREFIX
#CONDA_PREFIX should hold: /apps/jasmin/jaspy/miniconda_envs/jaspy3.7/m3-4.6.14/envs/jaspy3.7-m3-4.6.14-r20200606[[MT1]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msocom_1)
$NETCDF_ROOT =
$JULES_BUILD = normal
$JULES_COMPILER = gfortran
$JULES_MPI = nompi
$JULES_NETCDF = netcdf
$JULES_NETCDF_INC_PATH = $NETCDF_FORTRAN_ROOT/include
$JULES_NETCDF_LIB_PATH = $NETCDF_FORTRAN_ROOT/lib
$JULES_OMP = noomp
$JULES_REMOTE = local
$JULES_FFLAGS_EXTRA =
$JULES_LDFLAGS_EXTRA = [[MT2]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msocom_2)

# Now load the the build config settings based on the supplied environment variables.

include = $HERE/custom.cfg
  • · Now I move into the top directory of JULESvn6.0: cd $JULES_ROOT
  • · and compile: fcm make -j 2 -f etc/fcm-make/make.cfg --new

With gfortran and those settings, the compile works perfectly:

[tmarthews@cylc1 vn6.0_hj]$ fcm make -j 2 -f etc/fcm-make/make.cfg --new
[init] make # 2021-05-09T09:34:50Z
[info] FCM 2017.10.0 (/apps/contrib/metomi/fcm-2017.10.0)
[init] make config-parse # 2021-05-09T09:34:50Z
[info] config-file=/home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/make.cfg
[info] config-file= - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/platform/jasmin_gcc.cfg
[info] config-file= - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/platform/envars.cfg
[info] config-file= - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/platform/custom.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/remote/local.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/compiler/gfortran.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/build/normal.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/omp/noomp.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/ncdf/netcdf.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/mpi/nompi.cfg
[done] make config-parse # 0.9s
[init] make dest-init # 2021-05-09T09:34:51Z
[info] dest=tmarthews@cylc1.jasmin.ac.uk:/home/users/tmarthews/MODELS/vn6.0_hj
[info] mode=new
[done] make dest-init # 3.3s
[init] make extract # 2021-05-09T09:34:54Z
[info] location jules: 0: /home/users/tmarthews/MODELS/vn6.0_hj
[info] dest: 561 [A added]
[info] source: 561 [U from base]
[done] make extract # 4.1s
[init] make preprocess # 2021-05-09T09:34:58Z
[info] sources: total=561, analysed=557, elapsed-time=1.5s, total-time=1.1s
[info] target-tree-analysis: elapsed-time=0.1s
[info] install targets: modified=160, unchanged=0, failed=0, total-time=0.9s
[info] process targets: modified=397, unchanged=0, failed=0, total-time=15.9s
[info] TOTAL targets: modified=557, unchanged=0, failed=0, elapsed-time=9.4s
[done] make preprocess # 11.0s
[init] make build # 2021-05-09T09:35:09Z
[info] sources: total=561, analysed=561, elapsed-time=7.0s, total-time=12.4s
[info] target-tree-analysis: elapsed-time=1.3s
[info] compile targets: modified=393, unchanged=0, failed=0, total-time=125.8s
[info] compile+ targets: modified=353, unchanged=0, failed=0, total-time=1.8s
[info] install targets: modified=1, unchanged=0, failed=0, total-time=0.0s
[info] link targets: modified=1, unchanged=0, failed=0, total-time=3.5s
[info] TOTAL targets: modified=748, unchanged=0, failed=0, elapsed-time=73.8s
[done] make build # 81.0s
[done] make # 100.3s

[tmarthews@cylc1 vn6.0_hj]$

  1. OK: Now I try to do the same thing, except using the ifort compile instead of gfortran. From a fresh login to JASMIN (i.e. I don’t have jaspy modules loaded):

· This time I do the module load commands you suggested on http://cms.ncas.ac.uk/ticket/3529

module load intel/19.0.0
module load contrib/gnu/gcc/7.3.0
module load eb/OpenMPI/intel/3.1.1

· I change etc/fcm-make/make.cfg line 22 so that instead of just loading in custom.cfg it does the following:

# Load environment variable pre-settings
include = $HERE/envars.cfg
$JASMIN_JULES_BASE_DIR = /gws/nopw/j04/jules
$NETCDF_FORTRAN_ROOT = $JASMIN_JULES_BASE_DIR/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0
$NETCDF_ROOT = $NETCDF_FORTRAN_ROOT
$HDF5_LIBDIR = $NETCDF_FORTRAN_ROOT/lib
$I_MPI_PMI_LIBRARY = /usr/lib64/libpmi.so
$LD_LIBRARY_PATH = $LD_LIBRARY_PATH:$HDF5_LIBDIR[[MT3]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msocom_3)
$JULES_BUILD = normal
$JULES_OMP = Nome
$JULES_COMPILER = intel
$JULES_MPI = Nome
$JULES_NETCDF = netcdf
$JULES_NETCDF_INC_PATH = $NETCDF_FORTRAN_ROOT/include
$JULES_NETCDF_LIB_PATH = $NETCDF_ROOT/lib $NETCDF_FORTRAN_ROOT/lib $JASMIN_JULES_BASE_DIR/admin/curl/curl-lotus-parallel-intel/lib/ $HDF5_LIBDIR[[MT4]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msocom_4)
$JULES_REMOTE = local
$JULES_FFLAGS_EXTRA =
$JULES_LDFLAGS_EXTRA = [[MT5]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msocom_5)

# Now load the the build config settings based on the supplied environment variables.
include = $HERE/custom.cfg
  • · Now I move into the top directory of JULESvn6.0: cd $JULES_ROOT
  • · and compile: fcm make -j 2 -f etc/fcm-make/make.cfg --new

With those settings, however, the compile fails:

[tmarthews@cylc1 vn6.0_hj]$ fcm make -j 2 -f etc/fcm-make/make.cfg --new
[init] make # 2021-05-09T09:51:14Z
[info] FCM 2017.10.0 (/apps/contrib/metomi/fcm-2017.10.0)
[init] make config-parse # 2021-05-09T09:51:14Z
[info] config-file=/home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/make.cfg
[info] config-file= - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/platform/jasmin_intel.cfg
[info] config-file= - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/platform/envars.cfg
[info] config-file= - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/platform/custom.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/remote/local.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/compiler/intel.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/build/normal.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/omp/noomp.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/ncdf/netcdf.cfg
[info] config-file= - - - /home/users/tmarthews/MODELS/vn6.0_hj/etc/fcm-make/mpi/nompi.cfg
[done] make config-parse # 1.2s

[init] make dest-init # 2021-05-09T09:51:15Z
[info] dest=tmarthews@cylc1.jasmin.ac.uk:/home/users/tmarthews/MODELS/vn6.0_hj
[info] mode=new
[done] make dest-init # 3.4s
[init] make extract # 2021-05-09T09:51:18Z
[info] location jules: 0: /home/users/tmarthews/MODELS/vn6.0_hj
[info] dest: 561 [A added]

[info] source: 561 [U from base]
[done] make extract # 4.6s
[init] make preprocess # 2021-05-09T09:51:23Z
[info] sources: total=561, analysed=557, elapsed-time=1.7s, total-time=1.4s
[info] target-tree-analysis: elapsed-time=0.2s
[info] install targets: modified=160, unchanged=0, failed=0, total-time=0.9s
[info] process targets: modified=397, unchanged=0, failed=0, total-time=17.4s
[info] TOTAL targets: modified=557, unchanged=0, failed=0, elapsed-time=10.4s
[done] make preprocess # 12.1s

[init] make build # 2021-05-09T09:51:35Z
[info] sources: total=561, analysed=561, elapsed-time=7.8s, total-time=14.1s
[info] target-tree-analysis: elapsed-time=1.0s
[FAIL] ifort -obin/jules.exe o/jules.o -L/tmp/APINSjf10Z -ljules -L/gws/nopw/j04/jules/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0/lib -L/gws/nopw/j04/jules/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0/lib -L/gws/nopw/j04/jules/admin/curl/curl-lotus-parallel-intel/lib/ -L/gws/nopw/j04/jules/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0/lib -lnetcdff -heap-arrays -fp-model precise -traceback -Wl,--rpath=/gws/nopw/j04/jules/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0/lib /gws/nopw/j04/jules/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0/lib /gws/nopw/j04/jules/admin/curl/curl-lotus-parallel-intel/lib/ /gws/nopw/j04/jules/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0/lib # rc=1

[FAIL] ld: /gws/nopw/j04/jules/admin/netcdf/netcdf_par/3.1.1/intel.19.0.0/lib: file not recognized: Is a directory[[MT6]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msocom_6)
[FAIL] link 3.2 ! jules.exe <- jules/src/control/standalone/jules.F90
[info] compile targets: modified=393, unchanged=0, failed=0, total-time=389.5s
[info] compile+ targets: modified=353, unchanged=0, failed=0, total-time=5.3s
[info] install targets: modified=1, unchanged=0, failed=0, total-time=0.0s
[info] link targets: modified=0, unchanged=0, failed=1, total-time=3.2s
[info] TOTAL targets: modified=747, unchanged=0, failed=1, elapsed-time=218.6s
[FAIL] ! jules.exe : update task failed

[FAIL] make build # 226.7s
[FAIL] make # 248.0s

[tmarthews@cylc1 vn6.0_hj]$

So, in summary: that red line is the error I’m encountering.

Any further advice would be hugely appreciated!

Best,

Toby

[[MT1]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msoanchor_1)$CONDA_PREFIX is set by module load jaspy
[[MT2]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msoanchor_2)Note no extra flags set here at all: I just accept the flags set in etc/fcm-make/compiler/gfortran.cfg
[[MT3]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msoanchor_3)This time I don’t have $CONDA_PREFIX so I set the environment variables as you suggest in http://cms.ncas.ac.uk/ticket/3529
[[MT4]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msoanchor_4)These lines I get from the existing jasmin-lotus-intel.cfg file
[[MT5]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msoanchor_5)As before no extra flags set here at all: I just accept the flags set in etc/fcm-make/compiler/intel.cfg
[[MT6]](applewebdata://9C1A8F99-5D09-4AC6-B0E7-B6B1412B6D5D#_msoanchor_6)This is the odd failure and I don’t understand it.

I tried typing module add parallel-netcdf/intel and recompiling, as advised in some sources, but with that JASMIN suddenly can’t recognise “USE netcdf” and the compile stops even earlier.

2

Hi Patrick,

Just to say a big thank you for the help the other week and I have found a solution to the issue I was having. It turns out that the module loads you suggested

             module load intel/19.0.0
             module load contrib/gnu/gcc/7.3.0
             module load eb/OpenMPI/intel/3.1.1

are correct (although actually the gcc one is not necessary for compiling JULES on intel): it was the specification of the library paths in the make that was the issue.

I’ve raised a ticket on the MOSRS system to insert two platform files for JASMIN into JULES and that would solve the situation I had. I’ll try to get it in this code cycle https://code.metoffice.gov.uk/trac/jules/ticket/1145

Do I need to close that NCAS ticket http://cms.ncas.ac.uk/ticket/3529 somehow / indicate that the problem has been resolved, or do I just leave it now?

Best,
Toby

3

Hi Toby
Glad that it is working and that the info I provided was useful.
You can close the CMS ticket if you want, but it might be slightly better if I closed it
myself.
Patrick

4

Hi Patrick,

Sorry: I can’t do that. I don’t have permission to amend or close tickets on CMS. Will have to leave it.

If I get my MOSRS ticket in to JULES, though, it won’t be a problem for future users.

Thanks again for the help!

Toby

5

Hi Patrick,

Much of the information is now on the JULES ticket I’m trying to get in this time round https://code.metoffice.gov.uk/trac/jules/ticket/1145 though so hopefully others will not encounter the same problem I did.

Best regards,
Toby

6

Thanks for posting this, Patrick! Toby