Re: [eigen] Eigen binary I/O proposal

[Rohit Garg <rpg.314@xxxxxxxxx>]

It does not handle endianness. Though there is a _lot_ of space in the
header to make sure endianness is handled. For Eigen,
compatibility/backend for hdf5 seems desirable. Does HDF5 support
mmaped i/o btw? I had a quick look and I don't think it does.

I use a size function instead of sizeof because I wanted the program
to be able to infer the datatype at runtime. This is the part where
I'd like feedback on how to make a nice API. Template tricks could
help, I am sure, but I didn't know how to make best use of them when I
was writing it (in a hurry).

The point of this to make a binary blob of data on disk look like an
Eigen Matrix so that we can later do every thing else at a much higher
level. It is restricted to rank 2 tensors of course.

I don't want to spend time on making it ready for inclusion unless I
know that it (or something like it) adds value to eigen.


On Tue, Jul 7, 2009 at 6:10 AM, Tiago
Requeijo<tiago.requeijo.dev@xxxxxxxxx> wrote:
> Why don't we use some external library such as HDF5
> (http://www.hdfgroup.org/HDF5/)?
> HDF5 automatically takes care of endianness, on-the-fly compression of data,
> and a lot more stuff.
> We use it to store several matrices, vectors, and tables in a few large
> files (sizes 2-16GB) on a daily basis.
>
> Also, an added bonus of using some sort of know format is interoperability
> with other systems.  For example, you can read and poke into your file with
> Matlab, Octave, R, etc. There are also libraries for Python that allow to
> fully manipulate h5 files.
>
>
> TR
>
>
> On Tue, Jul 7, 2009 at 6:45 AM, Gael Guennebaud <gael.guennebaud@xxxxxxxxx>
> wrote:
>>
>>
>> On Tue, Jul 7, 2009 at 11:59 AM, Benoit Jacob <jacob.benoit.1@xxxxxxxxx>
>> wrote:
>>>
>>> Hi,
>>>
>>> had a very quick look at the code, 2 things come to mind:
>>> 1) make sure endianness issues are taken care of
>>> 2) why use plain C stuff like printf, maybe use c++ streams instead.
>>> if you allow the user to use your functions on custom stream types,
>>> that's more powerful.
>>
>> I second that, for instance one could use gzipped stream...
>>
>> gael
>>
>>>
>>> Benoit
>>>
>>> 2009/7/7 Benoit Jacob <jacob.benoit.1@xxxxxxxxx>:
>>> > Hi,
>>> >
>>> > I'm going to be offline for rougly 4 days starting now, can't look at
>>> > code now, but if you want to code a new eigen feature the best way you
>>> > can propose it is by cloning the eigen2 repository at bitbucket and
>>> > adding the feature in your repo. Makes it easy for people to start
>>> > using your feature, send you patches, generate diffs, etc.
>>> >
>>> > Benoit
>>> >
>>> > 2009/7/7 Rohit Garg <rpg.314@xxxxxxxxx>:
>>> >> Hi,
>>> >>
>>> >> First things first.
>>> >>
>>> >> http://sites.google.com/site/rpg314/Home/eigen-proposal
>>> >>
>>> >> This is a battle-tested, well commented (I think) piece of code along
>>> >> with unit tests. The largest file I used it with was 1.2 GB.
>>> >>
>>> >> There has been some demand for serialization of Eigen objects here
>>> >>
>>> >> http://forum.kde.org/viewtopic.php?f=74&t=61960.
>>> >>
>>> >> So I think this may fit the bill. There could be a lot of scope for
>>> >> template tricks to beautify the API, but for my needs, I could manage
>>> >> to to work with raw C too. I have tested it on Linux 32 bit with gcc
>>> >> 3.4 and linux 64 bit, (both gcc 4.3 and gcc 4.4) respectively.
>>> >>
>>> >> Comments/questions/cribs/suggestions/flames/rants/commendations(!)
>>> >> welcome.
>>> >>
>>> >> Cheers,
>>> >>
>>> >> --
>>> >> Rohit Garg
>>> >>
>>> >> http://rpg-314.blogspot.com/
>>> >>
>>> >> Senior Undergraduate
>>> >> Department of Physics
>>> >> Indian Institute of Technology
>>> >> Bombay
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>
>
>



-- 
Rohit Garg

http://rpg-314.blogspot.com/

Senior Undergraduate
Department of Physics
Indian Institute of Technology
Bombay