Re: [eigen] Eigen binary I/O proposal

[Aron Ahmadia <aja2111@xxxxxxxxxxxx>]

Hi Rohit,

That's a great start, and more than I've contributed to Eigen so far!

I think HDF5 may be a good approach for a compatible and high-speed
implementation.

The most C++y way of doing things would be to make Eigen objects
BOOST::Serialize-compatible.  Serialize is great for turning the
matrices into any kind of stream data, not just UNIX files.

If you'd like a look, I've done a very simple (not very efficient)
Serialize hack for my own code so that I can include Eigen Matrix2d
objects in my Serialize calls.  Let me know if you'd like me to send
you the files or check something into a repository.

Cheers,
A

On Tue, Jul 7, 2009 at 9:10 AM, Tiago
Requeijo<tiago.requeijo.dev@xxxxxxxxx> wrote:
> Why don't we use some external library such as HDF5
> (http://www.hdfgroup.org/HDF5/)?
> HDF5 automatically takes care of endianness, on-the-fly compression of data,
> and a lot more stuff.
> We use it to store several matrices, vectors, and tables in a few large
> files (sizes 2-16GB) on a daily basis.
>
> Also, an added bonus of using some sort of know format is interoperability
> with other systems.  For example, you can read and poke into your file with
> Matlab, Octave, R, etc. There are also libraries for Python that allow to
> fully manipulate h5 files.
>
>
> TR
>
>
> On Tue, Jul 7, 2009 at 6:45 AM, Gael Guennebaud <gael.guennebaud@xxxxxxxxx>
> wrote:
>>
>>
>> On Tue, Jul 7, 2009 at 11:59 AM, Benoit Jacob <jacob.benoit.1@xxxxxxxxx>
>> wrote:
>>>
>>> Hi,
>>>
>>> had a very quick look at the code, 2 things come to mind:
>>> 1) make sure endianness issues are taken care of
>>> 2) why use plain C stuff like printf, maybe use c++ streams instead.
>>> if you allow the user to use your functions on custom stream types,
>>> that's more powerful.
>>
>> I second that, for instance one could use gzipped stream...
>>
>> gael
>>
>>>
>>> Benoit
>>>
>>> 2009/7/7 Benoit Jacob <jacob.benoit.1@xxxxxxxxx>:
>>> > Hi,
>>> >
>>> > I'm going to be offline for rougly 4 days starting now, can't look at
>>> > code now, but if you want to code a new eigen feature the best way you
>>> > can propose it is by cloning the eigen2 repository at bitbucket and
>>> > adding the feature in your repo. Makes it easy for people to start
>>> > using your feature, send you patches, generate diffs, etc.
>>> >
>>> > Benoit
>>> >
>>> > 2009/7/7 Rohit Garg <rpg.314@xxxxxxxxx>:
>>> >> Hi,
>>> >>
>>> >> First things first.
>>> >>
>>> >> http://sites.google.com/site/rpg314/Home/eigen-proposal
>>> >>
>>> >> This is a battle-tested, well commented (I think) piece of code along
>>> >> with unit tests. The largest file I used it with was 1.2 GB.
>>> >>
>>> >> There has been some demand for serialization of Eigen objects here
>>> >>
>>> >> http://forum.kde.org/viewtopic.php?f=74&t=61960.
>>> >>
>>> >> So I think this may fit the bill. There could be a lot of scope for
>>> >> template tricks to beautify the API, but for my needs, I could manage
>>> >> to to work with raw C too. I have tested it on Linux 32 bit with gcc
>>> >> 3.4 and linux 64 bit, (both gcc 4.3 and gcc 4.4) respectively.
>>> >>
>>> >> Comments/questions/cribs/suggestions/flames/rants/commendations(!)
>>> >> welcome.
>>> >>
>>> >> Cheers,
>>> >>
>>> >> --
>>> >> Rohit Garg
>>> >>
>>> >> http://rpg-314.blogspot.com/
>>> >>
>>> >> Senior Undergraduate
>>> >> Department of Physics
>>> >> Indian Institute of Technology
>>> >> Bombay
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>
>
>