Has n02 got close to full? I ask as I am getting very slow filesystem performance on /work/n02/shared
Simon
I often see slow fs behaviour on A2. It’s lustre and at the end of it’s life.
Try and avoid doing things that lustre won’t like, such as conda, and so on - although if others are doing it then it may still creek.
Cirrus will have a different fs which I hope will have advantages possibly??
The problem was, partially, resolved by removing and reinstalling my python env[1] on /work. I think I have a lot of files in it and so all the loading from lustre was very very slow. Reinstalling reduced it to very slow…
[1] I am following the archer2 guidance of using the base env and then using pip to add new stuff on top. My venv directory is 525 Mbytes in size with about 21062 files. I think lustre performs poorly if it has to load a lot of files.
Simon
You’re right there. If you follow the advice and make it as minimal as possible then that’s as good as you can get. You can have more than one environment if you have different needs.
The next machines will be slightly different, and you can do other things to put it all in one file if you need to, but Archer is at the end of service so hopefully you can get to the end with things as they are.