H5Part development and discussion

[John Biddiscombe <biddisco AT cscs.ch>]

Dear H5Part users,

I'd like, if I may, to access the svn repository so that I can
a) update my copy of H5Part which is no doubt quite old
b) potentially contribute some improvements which I'll list below for 
your scrutiny

1) I found that reading particles into {x,y,z} arrays rather than into 
{x}, {y}, {z} separately was desirable from a visualization point of 
view so I have added some logic to my own code which handles the memory 
space issues around this. Also when writing data which is stored as an 
N-component vector field stored as {v0,v1,v2....vn} rather than N 
separate arrays.
I have found that to facilitate the display of data using any scalar 
field as the x,y,z coordinate it's nice to be able to write coordinate 
or vector fields as single component datasets (as is currently done with 
H5Part), but read them back in as a vector array again. I also find a 
lot of data with vector fields as u,v,w in separate arrays and wish to 
recombine these into a single {u,v,w} array.

2) With reference to the above, when reading N components into a single 
vector field, one wishes to supply an array of names. for example, the 
default H5Part "x","y","z" names are used for coordinate variables, but 
If the vector field :"velocity" is written to file as "velocity_0", 
"velocity_1", velocity_2" (for 3D), I may wish to read them back as the 
coordinate array later, so supplying an array of names (e.g. "x", 
"velocity_2", "y") to a read function allows one to arbitrarily map 
datasets onto components.

3) the H5Part naming convention uses "Particles1" "Particles2" etc as 
the group names (renamed I understand in later versions to steps1,2 or 
similar). I would prefer to use an attribute attached at the top level 
which would be a standard string preserved for all time
"ParticleTimeStepName" this attribute would be set to "Particles" in 
earlier files and "steps" in later version. Additionally, I would like 
to add a second attribute which would be the FormatString
"ParticleTimeFormatString" by default one would use %i to represent a 
simple integer, but if %05i were set, then particles would be stored as 
Particles00001 Particles00002, which makes ascii listings of dataset 
contents more intuitive as they appear in the numerical order (as 
opposed to the current particles1 particles10 particles100 particles11 
particles12 ....).
Although most simulations use time steps, it is frequently the case that 
data is dropped due to IO issues and using a Real time value might be 
desireable (eg 0.005, 0.0075, 0.1 - if time steps are not regular, but 
rather adaptive- in this case a special array which holds the actual 
time values should be present
"ParticleTimeValues" where the number of entries should match the number 
of groups or timesteps

4) Support for other data types is lacking - I regularly use int, short, 
float, double, byte and even boolean. Since HDF5 support all, so should 
H5Part. Since the majority of my code is in C++, I have used a 
combination of templates and macros to allow any data type to be 
read/written and it works fine. I would like to explore ways of 
extending this to other languages and adding support in the main h5part 
interface.

5) One of my emails appears in the archive where I mentioned DEM and SPH 
data and saw a reply that discussed meshed data. In this context, I was 
referring to Discrete Element Method particle data and Smoothed Particle 
Hydrodynamics particle data). I am encouraging users in both these 
domains to use H5Part as a storage form. They invariably have vector 
fields - so the use of multi-component scalars is essential. In the case 
of SPH data various kernel functions exist and I would like to store 
these as string attributes - I wonder if standard names (like those 
suggested above) could be chosen for Particle formats and enshrined 
somewhere so that

my own svn repository of H5Part related material is here should anyone 
wish to browse it.
https://svn.cscs.ch/vtkContrib/trunk/vtkCSCS/vtkH5Part/

Thanks for reading. I'm sure most of these issues have already been 
dealt with in your most recent versions of H5Part. I would very much 
like to contribute any that have not.

regards

JB

--
John Biddiscombe,                            email:biddisco @ cscs.ch
http://www.cscs.ch/about/BJohn.php
CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07
Via Cantonale, 6928 Manno, Switzerland      | Fax:  +41 (91) 610.82.82