H5Part development and discussion

[John Biddiscombe <biddisco AT cscs.ch>]

Hello,

I recently ran some IO tests on a blue gene machine using 32-2048
processors for the writes of data. The tests were designed to mimic the
type of usage we expect from SPH codes which have approx 5000 particles
per processor (total in this case fo 2048 nodes = 10million)
Performance using H5Part was very slow compared to using a single block
write using raw HDF5 calls. The main difference was H5Part writes each
scalar array independently (more calls to write, more cache hit/miss
tests)

Has anyone done any performance tests/tuning on large numbers of
processors. Based on the results I have been getting, we will not use
H5Part on the bluegene (unless I can find a way of speeding it up)

Thanks. (NB. I am in the process of collecting statistics which I'll
happily share with you all, but I would like to save myself the trouble
if any of you have done similar studies and published interesting
workarounds etc)

JB