Dear CMS Team,
This is running a HadGEM3-GC3.1-LL coupled suite.
UM job suite: u-eb985
log file path: /work/n02/n02/hjgrosvenor/cylc-run/u-eb985/log/job/18540101T0000Z/coupled/02/job.out
Hoping you are well.
I am working with Charlie Williams (@c.j.r.williams) in running paleoclimate simulations targeted at the Pliocene. We successfully started a late pliocene suite which has run for 20 years where we updated the land-sea mask to late Pliocene continents (u-eb571). We then attempted to start an early Pliocene run (u-eb985), the only difference being to remove land from the Central America Seaway which creates an ocean connection between the Atlantic and the Pacific. Therefore the only difference between this one and the successful is the land mask and the coupling weights/ancillaries shaped by the land-mask, no change in other forcings from our previous run nor from the piControl. This run failed 4 years in, due to an error Charlie hadn’t seen before and I couldn’t find in previous discussion, this is in job.out:
Thermo iteration does not converge,istep1, my_task, i, j: 107225, 125, 9, 15
Ice thickness: 0.5163180522816998
Snow thickness: 2.22659723538334367E-2
dTsf, Tsf_errmax: 0., 5.0000000000000001E-4
Tsf: 0.
fsurf: 0.
fcondtop, fcondbot, fswint 0., -7.99956130641182492E-13, 0.
(enew_save - einit)/dt, enum/dt, (enew_save - einit + enum)/dt = -9.16002256174882267E-4, 0., -9.16002256174882267E-4
enew_save, einit = -37352560.972311735, -37352559.873109028
size(enew_save), size(einit) = 2232
ij, m = 45, 2
fswsfc 0.
Iswabs 40.
Flux conservation error = 9.16002256974838397E-4
Initial snow temperatures:
0.
Initial ice temperatures:
-0.40551575841639259, -0.43392115246281721, -0.49106256190028724, -0.51666623909919474
Matrix ice temperature diff:
40.
dqmathilyr/dt:
40.
Final snow temperatures:
0.
Matrix ice temperature diff:
40.
dqmathilyr/dt:
40.
Final ice temperatures:
-0.40551768935183913, -0.43392119998326251, -0.49106190407190747, -0.51666623909919474
Ice melting temperatures:
-0.10517070073685295, -0.3814421401322427, -0.49103691530736426, -0.51666623909919474
Ice bottom temperature: -3.76499994869733444E-4
dT initial:
0.30034505767953967, 5.24790123305745038E-2, 2.56465929229787726E-5, 0.
dT final:
0.30034698861498621, 5.24790598510198047E-2, 2.49887645432100847E-5, 0.
zSin
1.9476055692009806, 7.0637433357822728, 9.0932762093956345, 9.5678933166517535
istep1, my_task, iblk = 107225, 125, 1
category n = 1
Global block: 126
Global i and j: 307, 308
Lat, Lon: 68.351152821583369, 45.247851024653919
aice: 0.96035132551973401
n: 1 aicen: 2.24213496078072382E-4
ice: Vertical thermo error
I think primarily we are wondering whether chasing this error is a red herring or not, whether there is something technically wrong we have done or whether the error may solve itself after restart?
Best wishes,
Harry