H5Part development and discussion

[Achim Gsell <achim AT cybercity.ch>]

On Tuesday 05 December 2006 19:05, Kurt Stockinger wrote:

> Just looked at your code again: In total, you read ~500 *
> (21+12) = 16500 attributes, right?

No, only 500*12 + 21 = 6021.  The file attributes are only read 
once.

> >> A 85 Giga file with 18263 time steps (same number of
> >> attributes) takes 29:38.52 to read in (half an hour).
> >> That's where it starts to hurt! It should be noted that a
> >> simple h5dump also groans under that file and won't
> >> produce anything before several minutes (I am still waiting
> >> in fact). (These figures are for our merlin00 machine at
> >> PSI.)

So we have to read 

18263 * 12 + 21 = 219177

attributes. Calculating the rate attributes per second we get 

~ 6021attr / 19s ~= 317 attr/s

for the small file and 

~ 219177attr / 1780s ~= 123 attr/s

for the large one. Taking into account, that the small file will 
be buffered by the OS (in the code Thomas calls H5GetNumSteps(), 
which reads most parts of the file in) and that counting the 
number of steps with H5PartGetNumSteps() takes ~600s of the 
1780s for the large file, it scales not too bad - but the rate 
is far from being sufficient.

> >> Achim suggested to use H5PartGetStepAttribInfo instead of
> >> H5PartReadStepAttrib,
> >> but I don't see how I can replace the functionality of
> >> ReadStepAttrib with
> >> GetStepAttribInfo ,i.e. read a value...

Yes, he can use the H5PartGetStepAttribInfo() function, but this 
will not help! 

Yesterday I made some wrong assumptions concerning the attribute 
interface.  As far as I see there is no quick solution for the 
performance problem ....

Achim