H5Part development and discussion

[Kurt Stockinger <kstockinger AT lbl.gov>]

Achim Gsell wrote:
> On Friday 26 January 2007 20:49, Kurt Stockinger wrote:
>
>   
>> I have a specific question about the data format. Is currently
>> the assumption that each scalar field is a group (see
>> TestField1 and TestField2) and the name of the dataset in the
>> scalar field is always "0"?
>>     
>
> Yes, if you write a scalar field, a new group in the 
> group "Block" will be created and in this new group a dataset 
> with the name "0". If you write a 3d-vector field, it's exactly 
> the same, but the datasets are named "0", "1" and "2"  
> corresponding to the three dimensions.  This scheme can easily 
> be extended to any dimension.
>
> I wouldn't say that this is an "assumption", this is the 
> *defined* format.
>   
Ok, thanks.
>   
>> Find below a simple example based 
>> on H5BlockParTestScalarField.c where I create 2 scalar fields
>> with
>>
>> ...
>> H5Block3dWriteScalarField ( f, "TestField1", data );
>> ...
>> H5Block3dWriteScalarField ( f, "TestField2", data2 );
>>
>>
>> src/H5PartTest> h5dump -H blockfile2.h5
>> HDF5 "blockfile2.h5" {
>> GROUP "/" {
>>    GROUP "Step#0" {
>>       GROUP "Block" {
>>          GROUP "TestField1" {
>>             DATASET "0" {
>>                DATATYPE  H5T_IEEE_F64LE
>>                DATASPACE  SIMPLE { ( 4, 3, 2 ) / ( 4, 3, 2 ) }
>>             }
>>          }
>>          GROUP "TestField2" {
>>             DATASET "0" {
>>                DATATYPE  H5T_IEEE_F64LE
>>                DATASPACE  SIMPLE { ( 4, 3, 2 ) / ( 4, 3, 2 ) }
>>             }
>>          }
>>       }
>>    }
>> }
>> }
>>     
>
> If I have the time tomorrow, I will write a short specification 
> of the current H5Part file format.  Otherwise you have to wait 
> until I'm back from vacation.
>   
That's ok. I don't want to spoil your vacation. ;-)

I've started a brief writeup for the H5Block that covers the main
methods. See what you think:

In H5Part all datasets are 1-dimensional arrays. H5Blocks stores the
datasets as 3-dimensional arrays that can be scalar fields (one
dataset) or vector fields (3 datasets). The methods for reading and
writing scalar fields are H5Block3dReadScalarField and
H5Block3dWriteScalarField, respectively. These 3-dimensional datasets
can be accessed by multiple processors concurrently where each
processor can read parts of the 3-dimensional dataset. The assignment
of parts of the data to various processors is defined via the structure
H5BlockPartition. For instance, assume a 3-dimensional scalar field
with the rank 4x5x6 is partitioned onto two processors, i.e. the rank
of dimension i is 4, the rank of dimension j is 5 and the rank of
dimension k is 6.The specification for H5BlockPartition for the
processors P1 and P2 could be:

P1: 0,3, 0,5, 0,2
P2: 0,3, 0,5, 3,5

where each of the partitions is defined as follows:
(i-start, i-end, j-start, j-end, k-start, k-end). In this example the
data is partitioned along dimension k. Finally, the partitioning of
blocks to various
processors is used to define the field layout via
H5BlockDefine3DFieldLayout. In other words, for reading and writing the
data in
parallel, the processor assignment needs to be explicitly specified.
 
The following two methods provide information about which data is
accessed by which processor. The method H5Block3dGetProcOf determines
which processor reads a particular part of the dataset. Alternatively,
the method H5Block3dGetPartitionOfProc returns the data partition for
a particular processor. 

Kurt

> Achim
> _______________________________________________
> H5Part mailing list
> H5Part AT lists.psi.ch
> https://lists.web.psi.ch/mailman/listinfo/h5part
>   


-- 
Kurt Stockinger
Computational Research Division
Lawrence Berkeley National Laboratory
Mail Stop 50F-1650, 1 Cyclotron Road
Berkeley, California 94720, USA

Tel: +1 (510) 486 5519, Fax: +1 (510) 486 5812
email: KStockinger AT lbl.gov
http://sdm.lbl.gov/kurts/