Packing options to save space

1

Hello

I’m looking at using lower packing options to save space with some output that’s getting unmanageably large. (8 runs of 1.5km res large domain 40 day simulations). I’ve already switched to netcdf. Previously I’ve used packing option 7 which the help info suggests is suitable for convective scale output, options 2 for example for climate output gives data to 3.s.f. and produces files at like half the size. Can anyone verify if this is sensible? I’ve read the umdps and I think this should be fine although I can’t tell if this is commonly done.

My other option is to compress all the files further, but I think the space saving is less and potentially adds a big bottleneck to the runs.

Cheers
Helen

2

Hi Helen

We too can’t tell if this is commonly done – what do the scientists say about the accuracy of the output?

We don’t see a performance penalty with netcdf compression at “1” – I can only suggest that you run some tests.

How large is unmanageably large ?

Grenville

3

Thanks, we’re trying to keep the output below 100TB for all 8 simulations, the higher prec option is heading towards 200TB! There seems to be quite a slowdown when increasing compression and using higher packing options so we’ll do some experiments with the different packing options to see if the lower option is enough sf for what we want to do.