JULES suite not running

,
1

Hi Patrick,

I’ve got a JULES suite to create historical estimates of soil moisture. It was running fine last week when I was doing some tests, but it’s no longer running. The run starts, but within a minute it crashes. Would you be able to have a look please? I suspect it’s something trivial but I’ve had a good look and still none the wiser.

Details provided below.

Thanks very much,
Ross

A copy of the suite is here (which you should be able to access): /gws/nopw/j04/tamsat/rmaidment/JULES/u-cs200

When running, it returns the following error message:

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[host308.jc.rl.ac.uk:30254] 7 more processes have sent help message help-mpi-api.txt / mpi-abort
[host308.jc.rl.ac.uk:30254] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[FAIL] rose-jules-run <<'__STDIN__'
[FAIL] 
[FAIL] '__STDIN__' # return-code=1
2023-02-03T10:45:11Z CRITICAL - failed/EXIT
--------------------------------------------------------------------------

And the last few lines of the run’s output before it crashes is:

{MPI Task 6} [INFO] init_surface_types: Using 4 non-veg surface types
{MPI Task 6} [INFO] init_surface_types: Soil is type #8
{MPI Task 6} [INFO] init_surface_types: Lake (inland water) is type #7
{MPI Task 6} [INFO] init_surface_types: Land ice is type #9
{MPI Task 6} [INFO] init_surface_types: Urban is type #6
{MPI Task 6} [INFO] init_surface_types: No canyon type specified (URBAN-2T or MORUSES)
{MPI Task 6} [INFO] init_surface_types: No roof type specified (URBAN-2T or MORUSES)
{MPI Task 6} [INFO] init_surface: Reading JULES_SURFACE namelist...
{MPI Task 6} [INFO] init_time: Reading JULES_TIME namelist...
{MPI Task 6} [INFO] init_time: Reading JULES_SPINUP namelist...
{MPI Task 6} [INFO] init_time: Timestep is 3600 seconds
2

Hi Ross:
The beginning of your job.err file has:

{MPI Task 0} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime
{MPI Task 4} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime
{MPI Task 7} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime
{MPI Task 3} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime
{MPI Task 1} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime
{MPI Task 5} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime
{MPI Task 2} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime
{MPI Task 6} [FATAL ERROR] datetime_from_string: '^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@' is not a valid datetime

Maybe something is not set up right?
Patrick

3

Hi Patrick,

Thanks for having a look. After some testing, it’s now working. In my suite.rc file, I had several occasions where the I had specified different start and end dates for different test runs (most of which are commented out). For example:

!!spinup_end='2001-01-01 00:00:00'
!!spinup_start='1999-01-01 00:00:00'
spinup_end='2001-01-01 00:00:00'
spinup_start='2000-01-01 00:00:00'
!!spinup_end='1983-01-01 00:00:00'
!!spinup_start='1984-01-01 00:00:00'

It seems that JULES does not like this! In the end, I deleted the bottom two lines and it worked.

Thanks for having a look though!
Ross

4

Hi Ross:
Yes, the !! is not a comment symbol. It is just a way to make a placeholder if you need it.

I’m glad it’s working now.
Patrick

5

Hi Patrick,

Thanks for the clarification and for the help!

Cheers,
Ross