I am having trouble running a numpy linalg svd code on a dataset. The dataset is (60480 x 60480).
When run on jasmin servers login1-sci1 I get the error ‘numpy.core._exceptions.MemoryError: Unable to allocate 27.3GiB for an array with shape (60480, 60480) and data type float64’.
This is understandable because at the time the Free Memory on sci1 was 21.4GiB.
If I ssh into login1-sci6 with a Free Memory of 516.0GiB at the time, and run the code again, I get the error ‘Segmentation fault’. Looking this up it seems that this is another Memory Error, but I would not know how to solve this. Solutions online appear to place the blame on the specific numpy build.
Does anyone have any solutions, as I want to use a far larger setof data in the future?
Thank you,
Benedict