# Atmos\_main failure **URL:** \ **Category:** Unified Model\ **Created:** [5 April 2022 09:49 UTC](https://cms-helpdesk.ncas.ac.uk/t/atmos-main-failure/527 "2022-04-05T09:49:29Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![hannahbryant](https://avatars.discourse-cdn.com/v4/letter/h/b5a626/32.png) [@hannahbryant](https://cms-helpdesk.ncas.ac.uk/u/hannahbryant)\ **Post date:** [5 April 2022 09:49 UTC](https://cms-helpdesk.ncas.ac.uk/t/atmos-main-failure/527/1 "2022-04-05T09:49:29Z")
Hi, I am five years into a run and the most recent atmos\_main task failed with this error: MPICH ERROR [Rank 0] [job id 1382455.0] [Mon Apr 4 22:01:09 2022] [nid001360] - Abort(2685455) (rank 0 in comm 0): Fatal error in PMPI\_Init\_thread: Other MPI error, error stack: MPIDI\_CRAY\_post\_init(2087)…: MPIDI\_CRAY\_ofi\_check\_nic\_symmetry(2705)…: MPIR\_CRAY\_Allreduce(557)…: MPIR\_Allreduce\_impl(298)…: MPIR\_Allreduce\_intra\_auto(210)…: MPIR\_Allreduce\_intra\_recursive\_doubling(232): (unknown)(): Other MPI error aborting job: Fatal error in PMPI\_Init\_thread: Other MPI error, error stack: MPIDI\_CRAY\_post\_init(2087)…: I wondered if this was due to my connection to archer and retriggered the failed task again and this ran successfully (and has now moved onto the next timestep). I have now noticed within the job.err there are new warnings as such: ??? ??? WARNING ??? ? Warning code: -1 ? Warning from routine: UKCA\_TROPOPAUSE::UKCA\_CALC\_TROPOPAUSE ? Warning message: Difficulty diagnosing pv tropopause, Reverting to default tropopause pressure ? Warning from processor: 0 ? Warning number: 78 ??? I am concerned that something is now wrong with the run and what the best thing to do is (restart or if it is possible to go back in the run to before the task that failed? (the ainitial file now appears to have moved past this point)). Thanks, Hannah ---
**Author:** ![dcase](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/dcase/32/468_2.png) [@dcase](https://cms-helpdesk.ncas.ac.uk/u/dcase)\ **Post date:** [5 April 2022 10:07 UTC](https://cms-helpdesk.ncas.ac.uk/t/atmos-main-failure/527/2 "2022-04-05T10:07:15Z")
Someone else may be able to give you a definitive answer on the warning, but I would note that the network on ARCHER2 is not completely robust, and restarting after an MPI failure like that is what I would have done. ---
**Author:** ![hannahbryant](https://avatars.discourse-cdn.com/v4/letter/h/b5a626/32.png) [@hannahbryant](https://cms-helpdesk.ncas.ac.uk/u/hannahbryant)\ **Post date:** [7 April 2022 08:36 UTC](https://cms-helpdesk.ncas.ac.uk/t/atmos-main-failure/527/3 "2022-04-07T08:36:30Z")
Thanks @dcase Just to update for the help desk - the newer atmos\_mains run successfully without the tropopause warning but I am still concerned that restarting has messed up the run. I need to run this for another 10 years, do you have any thoughts on whether it is ok to proceed or if redoing is more sensible? Thanks, Hannah ---
**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/1fd2411499ffcbc299fe756cd5cdf26e44956558.png) [@system](https://cms-helpdesk.ncas.ac.uk/u/system)\ **Post date:** [28 October 2023 19:46 UTC](https://cms-helpdesk.ncas.ac.uk/t/atmos-main-failure/527/4 "2023-10-28T19:46:14Z")
---
**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/cms_support/original/1X/1fd2411499ffcbc299fe756cd5cdf26e44956558.png) [@system](https://cms-helpdesk.ncas.ac.uk/u/system)\ **Post date:** [8 December 2023 18:43 UTC](https://cms-helpdesk.ncas.ac.uk/t/atmos-main-failure/527/5 "2023-12-08T18:43:19Z")
---
**Author:** ![RosalynHatcher](https://dub1.discourse-cdn.com/flex013/user_avatar/cms-helpdesk.ncas.ac.uk/rosalynhatcher/32/11_2.png) [@RosalynHatcher](https://cms-helpdesk.ncas.ac.uk/u/RosalynHatcher)\ **Post date:** [13 December 2023 13:58 UTC](https://cms-helpdesk.ncas.ac.uk/t/atmos-main-failure/527/6 "2023-12-13T13:58:42Z")