Re: [eigen] Eigen binary I/O proposal

[ Thread Index | Date Index | More lists.tuxfamily.org/eigen Archives ]

It seems like there is a desire for as many "flavors" of persistence as there are applications 

   * iostreams
   * boost streams
   * hdf5
   * netcdf
   * C stdio
   * plain old binary
   * sockets/pipes/fifos (generic file descriptors)
We could either try to come up with a common interface that allowed all of them or treat each as a special case, completely separable from the others.
I may not understand all the issues, but in general I would recommend the latter -- treat the different methods individually. It will allow features to get implemented only as there is sufficient support ("you ain't gonna need it"). That is not to say there won't be some commonality between things (e.g. endian-swapping). This approach should lower the barrier to entry for making a new persistence type, since it won't need to adhere to an artificial notion of persistence that we dream up. This will limit external dependencies which reduces tarball size, compiled size, and headache size. The different persistence frameworks could even be distributed as "add-ons" to Eigen. 

-- Mark  Borgerding
P.S. I haven't forgotten about Eigen::FFT -- I will implement Benoit's suggestions sometime soon. 


On 07/07/2009 09:10 AM, Tiago Requeijo wrote:
Why don't we use some external library such as HDF5 (http://www.hdfgroup.org/HDF5/)? HDF5 automatically takes care of endianness, on-the-fly compression of data, and a lot more stuff. We use it to store several matrices, vectors, and tables in a few large files (sizes 2-16GB) on a daily basis. Also, an added bonus of using some sort of know format is interoperability with other systems. For example, you can read and poke into your file with Matlab, Octave, R, etc. There are also libraries for Python that allow to fully manipulate h5 files. 
TR
On Tue, Jul 7, 2009 at 6:45 AM, Gael Guennebaud <gael.guennebaud@xxxxxxxxx <mailto:gael.guennebaud@xxxxxxxxx>> wrote: 



    On Tue, Jul 7, 2009 at 11:59 AM, Benoit Jacob
    <jacob.benoit.1@xxxxxxxxx <mailto:jacob.benoit.1@xxxxxxxxx>> wrote:

        Hi,

        had a very quick look at the code, 2 things come to mind:
        1) make sure endianness issues are taken care of
        2) why use plain C stuff like printf, maybe use c++ streams
        instead.
        if you allow the user to use your functions on custom stream
        types,
        that's more powerful.


    I second that, for instance one could use gzipped stream...

    gael

        Benoit

        2009/7/7 Benoit Jacob <jacob.benoit.1@xxxxxxxxx
        <mailto:jacob.benoit.1@xxxxxxxxx>>:
        > Hi,
        >
        > I'm going to be offline for rougly 4 days starting now,
        can't look at
        > code now, but if you want to code a new eigen feature the
        best way you
        > can propose it is by cloning the eigen2 repository at
        bitbucket and
        > adding the feature in your repo. Makes it easy for people to
        start
        > using your feature, send you patches, generate diffs, etc.
        >
        > Benoit
        >
        > 2009/7/7 Rohit Garg <rpg.314@xxxxxxxxx
        <mailto:rpg.314@xxxxxxxxx>>:
        >> Hi,
        >>
        >> First things first.
        >>
        >> http://sites.google.com/site/rpg314/Home/eigen-proposal
        >>
        >> This is a battle-tested, well commented (I think) piece of
        code along
        >> with unit tests. The largest file I used it with was 1.2 GB.
        >>
        >> There has been some demand for serialization of Eigen
        objects here
        >>
        >> http://forum.kde.org/viewtopic.php?f=74&t=61960
        <http://forum.kde.org/viewtopic.php?f=74&t=61960>.
        >>
        >> So I think this may fit the bill. There could be a lot of
        scope for
        >> template tricks to beautify the API, but for my needs, I
        could manage
        >> to to work with raw C too. I have tested it on Linux 32 bit
        with gcc
        >> 3.4 and linux 64 bit, (both gcc 4.3 and gcc 4.4) respectively.
        >>
        >>
        Comments/questions/cribs/suggestions/flames/rants/commendations(!)
        welcome.
        >>
        >> Cheers,
        >>
        >> --
        >> Rohit Garg
        >>
        >> http://rpg-314.blogspot.com/
        >>
        >> Senior Undergraduate
        >> Department of Physics
        >> Indian Institute of Technology
        >> Bombay
        >>
        >>
        >>
        >

Mail converted by MHonArc 2.6.19+ http://listengine.tuxfamily.org/