Skip to Content.
Sympa Menu

h5part - Re: [H5part] Performance

h5part AT

Subject: H5Part development and discussion

List archive

Re: [H5part] Performance

Chronological Thread 
  • From: Andreas Adelmann <andreas.adelmann AT>
  • To: John Biddiscombe <biddisco AT>
  • Cc: h5part AT
  • Subject: Re: [H5part] Performance
  • Date: Wed, 19 Mar 2008 15:45:11 +0100
  • List-archive: <>
  • List-id: H5Part development and discussion <>

John Biddiscombe wrote:

I recently ran some IO tests on a blue gene machine using 32-2048
processors for the writes of data. The tests were designed to mimic the
type of usage we expect from SPH codes which have approx 5000 particles
per processor (total in this case fo 2048 nodes = 10million)
Performance using H5Part was very slow compared to using a single block
write using raw HDF5 calls. The main difference was H5Part writes each
scalar array independently (more calls to write, more cache hit/miss

Has anyone done any performance tests/tuning on large numbers of
processors. Based on the results I have been getting, we will not use
H5Part on the bluegene (unless I can find a way of speeding it up)

Thanks. (NB. I am in the process of collecting statistics which I'll
happily share with you all, but I would like to save myself the trouble
if any of you have done similar studies and published interesting
workarounds etc)


H5Part mailing list
H5Part AT

Hello John good to hear from you! Sorry for the delay I am at Los Alamos right now!
Attached a paper with old number and an xls sheed John Shalf has compiled
comparing different architectures i.e. machines (in the xls file, HDF5 == H5part).
The data are not for very large number of nodes.

Achim: can you send the data Roman and Valeri collected, w.r.t. HDF5 setup/configuration.

However you address a point that has to be looked at: in my opinion we have to prevent writing small
junks of data and try to cash as much as possible and then write large junks of data.

Can you share the test program with us so that we eventually can fix that. I would love to use H5Part on IBM!

Best Andreas

Dr. sc. math. Andreas (Andy) Adelmann
Staff Scientist
Paul Scherrer Institut WLGB/132 CH-5232 Villigen PSI
Phone Office: xx41 56 310 42 33 Fax: xx41 56 310 31 91
Phone Home: xx41 62 891 91 44
Wednesday: ETH CAB F 10.1 xx41 44 632 82 76
The more exotic, the more abstract the knowledge, the more profound will be its consequences."
Leon Lederman =======================================================

Attachment: IO.xls
Description: Binary data

Attachment: FPAT083.pdf
Description: Adobe PDF document

Archive powered by MHonArc 2.6.19.

Top of Page