H5Part development and discussion

[Andreas Adelmann <andreas.adelmann AT psi.ch>]

John Biddiscombe wrote:
> Hello,
>
> I recently ran some IO tests on a blue gene machine using 32-2048
> processors for the writes of data. The tests were designed to mimic the
> type of usage we expect from SPH codes which have approx 5000 particles
> per processor (total in this case fo 2048 nodes = 10million)
> Performance using H5Part was very slow compared to using a single block
> write using raw HDF5 calls. The main difference was H5Part writes each
> scalar array independently (more calls to write, more cache hit/miss
> tests)
>
> Has anyone done any performance tests/tuning on large numbers of
> processors. Based on the results I have been getting, we will not use
> H5Part on the bluegene (unless I can find a way of speeding it up)
>
> Thanks. (NB. I am in the process of collecting statistics which I'll
> happily share with you all, but I would like to save myself the trouble
> if any of you have done similar studies and published interesting
> workarounds etc)
>
> JB
>
> _______________________________________________
> H5Part mailing list
> H5Part AT lists.psi.ch
> https://lists.web.psi.ch/mailman/listinfo/h5part
>   

Hello John good to hear from you! Sorry for the delay I am at Los Alamos 
right now!
Attached a paper with old number and an xls sheed John Shalf has compiled
comparing different architectures i.e. machines (in the xls file, HDF5 
== H5part).
The data are not for very large number of nodes.

Achim: can you send the data Roman and Valeri collected, w.r.t. HDF5 
setup/configuration.

However you address a point that has to be looked at: in my opinion we 
have to prevent writing small
junks of data and try to cash as much as possible and then write large 
junks of data.

Can you share the test program with us so that we eventually can fix 
that. I would love to use H5Part on IBM!

Best Andreas



--
Dr. sc. math. Andreas (Andy) Adelmann
Staff Scientist
Paul Scherrer Institut WLGB/132 CH-5232 Villigen PSI
Phone Office: xx41 56 310 42 33 Fax: xx41 56 310 31 91
Phone Home: xx41 62 891 91 44
-------------------------------------------------------
Wednesday: ETH CAB F 10.1  xx41 44 632 82 76
=======================================================
The more exotic, the more abstract the knowledge, 
the more profound will be its consequences."
Leon Lederman 
=======================================================

Attachment: IO.xls
Description: Binary data

Attachment: FPAT083.pdf
Description: Adobe PDF document