Read-only file system error

Rose/Cylc and FCM
1

Hello, I have problem with running my u-ck523 rose suite on JASMIN. I’m getting this error:

[FAIL] [Errno 30] Read-only file system: ‘/work/scratch-pw/jmac87/cylc-run/u-ck523/share/cycle/1’

Any ideas?

When I run “top -u jmac87” I get this (doesn’t seem like I’ve got anything running trying to access the space):

8382 jmac87 20 0 161048 2552 1572 R 0.3 0.0 0:00.05 top
383 jmac87 20 0 122072 900 652 S 0.0 0.0 0:00.01 gpg-agent
32721 jmac87 20 0 156312 2500 1176 S 0.0 0.0 0:00.02 sshd
32733 jmac87 20 0 115640 3544 1760 S 0.0 0.0 0:00.08 bash

Cheers

Jon

2

Hi Jonathan:
The /work/scratch-pw space is being retired TODAY! You should have received an email today. It was made read-only today or yesterday. And is being replaced by /work/scratch-pw2 and /work/scratch-pw3 . You should copy any important data from scratch-pw before mid December.
Does this explain things?
Patrick

3

Hi again Jonathan:
Weren’t your jobs running in batch mode on SLURM/LOTUS instead of interactively on cylc1, where you ran your top?
Patrick

4

Hi Patrick, I have no idea why the rose suite is even trying to access work/scratch-pw. I’ve looked in my suite.rc file and can’t see any reference to it there. Any ideas what is making it access that space?

cheers

Jon

5

Yes, sorry my bad. Do you know how I can check if I have anything running interactively?

6

Hi Jonathan:
I believe the rose/cylc suites are automatically set up so that the work and share subdirectories of the suite are actually on /work/scratch-pw (now /work/scratch-pw2 ?). The cylc-run suite will have symbolic links to the work and share subdirectories. There might be a way to change the location of the scratch space, but that might take some figuring out.
Patrick

7

Hi again, Jonathan:
One way to see if you have anything running interactively on cylc1 is to do the top on cylc1, like you have already done.
Patrick

8

Yes, but presumably everyone should be having this problem if it is the fact that scratch-pw is the issue?

9

Thanks Patrick - based on what I pasted, does it look like anything is running to you? Can I safely just kill all of the tasks running under my name to see if that helps?

10

Hi Jonathan:
Yes, everyone might be having the problem, but only if somehow they had jobs still running on SLURM when the scratch-pw was made read-only. But I would have hoped that SLURM would have been drained completely before scratch-pw was made read-only.
Patrick

11

Hi Jonathan:
Your top on cylc1 has just the sshd process and gpg-agent password process. You don’t have any rose/cylc processes running on cylc1.
Patrick

12

Thanks Patrick - do you think I should take this up with Fatima?

13

Just looked at the rose config files on JASMIN and it looks like they are still pointing to scratch-pw as far as I can see. If so JASMIN will need to change this.

If you want to run before that’s fixed centrally you can set your suite to write to the new scratch space.

In rose-suite.conf file at the top add:

root-dir-share=*=/work/scratch-pw2/$USER
root-dir-work=*=/work/scratch-pw2/$USER

Caveat: If you don’t want to run the suite again from the beginning I think you’ll have to copy over the contents of /work/scratch-pw/jmac87/<suiteid> for the suite to be able to continue.

Cheers,
Ros.

1 Like
14

Thanks Ros, I’ll give this a go!

15

It’s working thank you!

1 Like
16

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.