Skip to Content.
Sympa Menu

h5part - Re: [H5part] H5Part performance problem

h5part AT

Subject: H5Part development and discussion

List archive

Re: [H5part] H5Part performance problem

Chronological Thread 
  • From: Achim Gsell <achim AT>
  • To: h5part AT
  • Subject: Re: [H5part] H5Part performance problem
  • Date: Tue, 5 Dec 2006 20:51:37 +0100
  • List-archive: <>
  • List-id: H5Part development and discussion <>

On Tuesday 05 December 2006 19:05, Kurt Stockinger wrote:

> Just looked at your code again: In total, you read ~500 *
> (21+12) = 16500 attributes, right?

No, only 500*12 + 21 = 6021. The file attributes are only read

> >> A 85 Giga file with 18263 time steps (same number of
> >> attributes) takes 29:38.52 to read in (half an hour).
> >> That's where it starts to hurt! It should be noted that a
> >> simple h5dump also groans under that file and won't
> >> produce anything before several minutes (I am still waiting
> >> in fact). (These figures are for our merlin00 machine at
> >> PSI.)

So we have to read

18263 * 12 + 21 = 219177

attributes. Calculating the rate attributes per second we get

~ 6021attr / 19s ~= 317 attr/s

for the small file and

~ 219177attr / 1780s ~= 123 attr/s

for the large one. Taking into account, that the small file will
be buffered by the OS (in the code Thomas calls H5GetNumSteps(),
which reads most parts of the file in) and that counting the
number of steps with H5PartGetNumSteps() takes ~600s of the
1780s for the large file, it scales not too bad - but the rate
is far from being sufficient.

> >> Achim suggested to use H5PartGetStepAttribInfo instead of
> >> H5PartReadStepAttrib,
> >> but I don't see how I can replace the functionality of
> >> ReadStepAttrib with
> >> GetStepAttribInfo ,i.e. read a value...

Yes, he can use the H5PartGetStepAttribInfo() function, but this
will not help!

Yesterday I made some wrong assumptions concerning the attribute
interface. As far as I see there is no quick solution for the
performance problem ....


Archive powered by MHonArc 2.6.19.

Top of Page