Re: [eigen] Eigen binary I/O proposal

[Rohit Garg <rpg.314@xxxxxxxxx>]

PS: I have NO idea whatsoever regarding HDF5 API. Help welcome!

On Tue, Jul 7, 2009 at 10:37 AM, Rohit Garg<rpg.314@xxxxxxxxx> wrote:
> It does not handle endianness. Though there is a _lot_ of space in the
> header to make sure endianness is handled. For Eigen,
> compatibility/backend for hdf5 seems desirable. Does HDF5 support
> mmaped i/o btw? I had a quick look and I don't think it does.
>
> I use a size function instead of sizeof because I wanted the program
> to be able to infer the datatype at runtime. This is the part where
> I'd like feedback on how to make a nice API. Template tricks could
> help, I am sure, but I didn't know how to make best use of them when I
> was writing it (in a hurry).
>
> The point of this to make a binary blob of data on disk look like an
> Eigen Matrix so that we can later do every thing else at a much higher
> level. It is restricted to rank 2 tensors of course.
>
> I don't want to spend time on making it ready for inclusion unless I
> know that it (or something like it) adds value to eigen.
>
>
> On Tue, Jul 7, 2009 at 6:10 AM, Tiago
> Requeijo<tiago.requeijo.dev@xxxxxxxxx> wrote:
>> Why don't we use some external library such as HDF5
>> (http://www.hdfgroup.org/HDF5/)?
>> HDF5 automatically takes care of endianness, on-the-fly compression of data,
>> and a lot more stuff.
>> We use it to store several matrices, vectors, and tables in a few large
>> files (sizes 2-16GB) on a daily basis.
>>
>> Also, an added bonus of using some sort of know format is interoperability
>> with other systems.  For example, you can read and poke into your file with
>> Matlab, Octave, R, etc. There are also libraries for Python that allow to
>> fully manipulate h5 files.
>>
>>
>> TR
>>
>>
>> On Tue, Jul 7, 2009 at 6:45 AM, Gael Guennebaud <gael.guennebaud@xxxxxxxxx>
>> wrote:
>>>
>>>
>>> On Tue, Jul 7, 2009 at 11:59 AM, Benoit Jacob <jacob.benoit.1@xxxxxxxxx>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> had a very quick look at the code, 2 things come to mind:
>>>> 1) make sure endianness issues are taken care of
>>>> 2) why use plain C stuff like printf, maybe use c++ streams instead.
>>>> if you allow the user to use your functions on custom stream types,
>>>> that's more powerful.
>>>
>>> I second that, for instance one could use gzipped stream...
>>>
>>> gael
>>>
>>>>
>>>> Benoit
>>>>
>>>> 2009/7/7 Benoit Jacob <jacob.benoit.1@xxxxxxxxx>:
>>>> > Hi,
>>>> >
>>>> > I'm going to be offline for rougly 4 days starting now, can't look at
>>>> > code now, but if you want to code a new eigen feature the best way you
>>>> > can propose it is by cloning the eigen2 repository at bitbucket and
>>>> > adding the feature in your repo. Makes it easy for people to start
>>>> > using your feature, send you patches, generate diffs, etc.
>>>> >
>>>> > Benoit
>>>> >
>>>> > 2009/7/7 Rohit Garg <rpg.314@xxxxxxxxx>:
>>>> >> Hi,
>>>> >>
>>>> >> First things first.
>>>> >>
>>>> >> http://sites.google.com/site/rpg314/Home/eigen-proposal
>>>> >>
>>>> >> This is a battle-tested, well commented (I think) piece of code along
>>>> >> with unit tests. The largest file I used it with was 1.2 GB.
>>>> >>
>>>> >> There has been some demand for serialization of Eigen objects here
>>>> >>
>>>> >> http://forum.kde.org/viewtopic.php?f=74&t=61960.
>>>> >>
>>>> >> So I think this may fit the bill. There could be a lot of scope for
>>>> >> template tricks to beautify the API, but for my needs, I could manage
>>>> >> to to work with raw C too. I have tested it on Linux 32 bit with gcc
>>>> >> 3.4 and linux 64 bit, (both gcc 4.3 and gcc 4.4) respectively.
>>>> >>
>>>> >> Comments/questions/cribs/suggestions/flames/rants/commendations(!)
>>>> >> welcome.
>>>> >>
>>>> >> Cheers,
>>>> >>
>>>> >> --
>>>> >> Rohit Garg
>>>> >>
>>>> >> http://rpg-314.blogspot.com/
>>>> >>
>>>> >> Senior Undergraduate
>>>> >> Department of Physics
>>>> >> Indian Institute of Technology
>>>> >> Bombay
>>>> >>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>
>>
>>
>
>
>
> --
> Rohit Garg
>
> http://rpg-314.blogspot.com/
>
> Senior Undergraduate
> Department of Physics
> Indian Institute of Technology
> Bombay
>



-- 
Rohit Garg

http://rpg-314.blogspot.com/

Senior Undergraduate
Department of Physics
Indian Institute of Technology
Bombay