Re: [eigen] Eigen binary I/O proposal

[Tiago Requeijo <tiago.requeijo.dev@xxxxxxxxx>]

Why don't we use some external library such as HDF5 (http://www.hdfgroup.org/HDF5/)?

HDF5 automatically takes care of endianness, on-the-fly compression of data, and a lot more stuff. 

We use it to store several matrices, vectors, and tables in a few large files (sizes 2-16GB) on a daily basis.

 

Also, an added bonus of using some sort of know format is interoperability with other systems.  For example, you can read and poke into your file with Matlab, Octave, R, etc. There are also libraries for Python that allow to fully manipulate h5 files.

 

 

TR

 

On Tue, Jul 7, 2009 at 6:45 AM, Gael Guennebaud <gael.guennebaud@xxxxxxxxx> wrote:

On Tue, Jul 7, 2009 at 11:59 AM, Benoit Jacob <jacob.benoit.1@xxxxxxxxx> wrote:

Hi,

had a very quick look at the code, 2 things come to mind:
1) make sure endianness issues are taken care of
2) why use plain C stuff like printf, maybe use c++ streams instead.
if you allow the user to use your functions on custom stream types,
that's more powerful.

I second that, for instance one could use gzipped stream...

gael
 

Benoit

2009/7/7 Benoit Jacob <jacob.benoit.1@xxxxxxxxx>:

> Hi,
>
> I'm going to be offline for rougly 4 days starting now, can't look at
> code now, but if you want to code a new eigen feature the best way you
> can propose it is by cloning the eigen2 repository at bitbucket and
> adding the feature in your repo. Makes it easy for people to start
> using your feature, send you patches, generate diffs, etc.
>
> Benoit
>
> 2009/7/7 Rohit Garg <rpg.314@xxxxxxxxx>:
>> Hi,
>>
>> First things first.
>>
>> http://sites.google.com/site/rpg314/Home/eigen-proposal
>>
>> This is a battle-tested, well commented (I think) piece of code along
>> with unit tests. The largest file I used it with was 1.2 GB.
>>
>> There has been some demand for serialization of Eigen objects here
>>
>> http://forum.kde.org/viewtopic.php?f=74&t=61960.
>>
>> So I think this may fit the bill. There could be a lot of scope for
>> template tricks to beautify the API, but for my needs, I could manage
>> to to work with raw C too. I have tested it on Linux 32 bit with gcc
>> 3.4 and linux 64 bit, (both gcc 4.3 and gcc 4.4) respectively.
>>
>> Comments/questions/cribs/suggestions/flames/rants/commendations(!) welcome.
>>
>> Cheers,
>>
>> --
>> Rohit Garg
>>
>> http://rpg-314.blogspot.com/
>>
>> Senior Undergraduate
>> Department of Physics
>> Indian Institute of Technology
>> Bombay
>>
>>
>>
>