| Re: [eigen] Dropping C++14 compatibility discussion |
[ Thread Index | Date Index | More lists.tuxfamily.org/eigen Archives ]
Hello,
I dispute the Eigen "seems effectively abandoned" claim. If you visit the repo, there are commits listed as recently as 6 days ago. In my opinion the core devs do sterling work.
However, I'm already on record on this list as thinking Eigen should be updated to a newer C++ standard. I agree with the central point below that old versions do not disappear. ITK (https://itk.org) is a codebase of similar age to Eigen, and has moved up to C++11 over the last couple of years. They have seen big improvements in performance and usability. This change happened fairly organically - they decided that version 5 would be the first to require C++11, and then gradually over a year or two and a couple of minor versions different parts of the codebase were slowly updated to use lambdas etc.
At the risk of being the annoying user who requests too much - I can't find it right now but I have memories of a mailing list discussion from several years ago where Gael proposed that the next major version change in Eigen would merge the Tensor and main Eigen codebases, such that Array<> and Matrix<> became specialisations of a rank 2 Tensor. I am sure that is also a huge amount of work but that is what I would truly like to see in Eigen 4, and would require the switch to C++11 as a minimum. The ability to store a rank 3 Tensor and then natively chip slices out to an Array or Matrix would be very helpful.
Best wishes,
Toby
From: Michael Hofmann <kmhofmann@xxxxxxxxx>
Reply to: "eigen@xxxxxxxxxxxxxxxxxxx" <eigen@xxxxxxxxxxxxxxxxxxx>
Date: Wednesday, 6 May 2020 at 08:19
To: "eigen@xxxxxxxxxxxxxxxxxxx" <eigen@xxxxxxxxxxxxxxxxxxx>
Subject: Re: [eigen] Dropping C++14 compatibility discussion
There are simple and established solutions to the problem of having an old system compiler or standard libraries on a cluster-type computing system. The Environment Modules system (http://modules.sourceforge.net/) is one of them. Local compiler installs or even Docker are others. Any sysadmin worth their salt can solve these issues for users quite easily, if they just wanted to. (They could additionally upgrade the system, if they just wanted to.)
> However, conservatism certainly has value. Particularly when you advertise yourself as a production library (not a research or proof of concept toy).
> Eigen development is slow. The code has reached a very stable state. The last major release was in 2018. These facts are probably good for most users of Eigen.
I heavily disagree. In my opinion, there is absolutely no value in this kind of conservatism. Instead, it *only* harms.
Conservatism in library development has little effect on "production quality". In fact, I'd argue that libraries that choose to move faster are often better, including for production use cases, since they have judiciously cleaned up their cruft, thus attract a larger developer and user base and have a healthier ecosystem. It is proven that in software development, small lead and cycle times and frequent "deploys", be it upstream or downstream, in libraries or services, are strongly beneficial.
At least part of the reason why Eigen development has slowed to a halt is said refusal to move with the times, at the right time (i.e. not six years too late).
We have a classic case of "code rot" here, due to changing requirements from the user base over time. Both the internals of Eigen have become harder to maintain (things are not expressed as simply and elegantly as they could be) and the externals harder to use (crufty interfaces, slow compilation speeds, myriads of spurious warnings on newer compilers, etc.).
Because nothing happened for years due to the conservative governance model, modernizing the code has now accumulated to a huge task, and it's on a developer base that has already shrunken quite a bit, judging from the average level of activity on this mailing list. This greatly exacerbates the problem. Continuous and incremental modernization in a timely(!) manner is the best way to prevent this scenario from happening. This includes (but is not limited to) moving to new C++ standards around the time they are introduced.
It doesn't really matter whether one calls a newer version 3.x or 4.x -- these are just semantics in the end. No matter how a project is governed (SemVer or not, level of promised API or ABI stability, etc.), the fundamental insight has to be that *old versions do not go away* and can still be used by anyone who wishes to do so. Conservative users may choose to upgrade more rarely, at the cost of much larger pain when doing so. (A model that I do not believe in.) Most users should be encouraged to migrate to the latest released version quickly, and to be part of the continuous, incremental maintenance cycle.
I have been an Eigen user and defender for years, but I cannot recommend this library anymore to others without giving huge caveats ("outdated", "seems effectively abandoned"). A strong commitment to modernity and moving with the times by the maintainers is the best and maybe only way to bring back users and thus developers. As part of this, I do hope that Eigen governance commits to moving to C++17 in the very near future.
Cheers,
Michael
On Wed, May 6, 2020 at 5:53 AM Rob McDonald <rob.a.mcdonald@xxxxxxxxx> wrote:
I don't have access to the machine, so I can't answer the detailed questions.
My application isn't super intensive -- but it can be used to generate input files for much more expensive computations and optimization studies. Consequently, it sometimes gets run on big iron.
My user who recently had problems because my app developed a dependency on std::regexp -- which isn't really supported by gcc 4.8. The problem is less the compiler and more the standard libraries installed on the machine. He is not building on the big machine -- he builds on a local vm on a laptop and then transfers the binary up. They also prefer to do their local setup on images that are as similar as possible to the computation environment. Perhaps they could set up local compilers and libraries, but that is a much bigger hassle than using system installed standard libraries.
The machine is not at TACC. It has nearly 250k cores and 1PB memory. It was installed when RHEL 7 was brand new. I have no idea how many cores the users are using -- a small fraction no doubt.
Rob
On Tue, May 5, 2020 at 8:01 PM Jeff Hammond <jeff.science@xxxxxxxxx> wrote:
Is the primary compiler on said supercomputer really /usr/bin/g++? How many nodes are users running on when they build apps with this compiler?
My guess is most of the users of that system use mpicxx or something similar from modules that provides a trivial solution to this problem. If you’re referring to TACC, I’m sure they’ll recommend a bunch of compilers besides GCC-4.8.
RHEL has devtoolset for a reason. All major HPC systems have modules for a dozen or more compilers that don’t live in /usr. Eigen shouldn’t hold itself back for people who refuse to help themselves.
Jeff
On Tue, May 5, 2020 at 3:21 PM Rob McDonald <rob.a.mcdonald@xxxxxxxxx> wrote:
I certainly understand the need to move forward. However, conservatism certainly has value. Particularly when you advertise yourself as a production library (not a research or proof of concept toy).
Just as an example, the main national supercomputer for a major US government research agency is running RHEL 7. That means gcc 4.8.x and the standard libraries that go with it. That means C++11 and most of the stuff that goes with it -- but not everything.
RHEL 8 was only released 364 days ago (depending on your time zone). It is certainly not unreasonable for installed high value production computing assets like the above machine to still be running RHEL 7. While it would be great if RHEL moved compilers forward more quickly -- that isn't how it works.
The fact is -- Eigen development is slow. The code has reached a very stable state. The last major release was in 2018. These facts are probably good for most users of Eigen.
If permission to jump to some advanced version means someone is going to go through and wholesale clean up the code and move it forward -- who is that someone? Where is this development energy coming from? Users making a request doesn't mean there is anyone to get the job done.
Eigen 3.x should remain pretty much the way it is. I hope/assume that any big jump like this would start by creating an Eigen 4.x branch/fork on GitHub and moving forward from there -- hopefully the energetic people calling for this change are the ones who will step forward to do the work.
Rob
On Tue, May 5, 2020 at 3:01 PM Janek Kozicki (yade) <jkozicki-yade@xxxxxxxxx> wrote:
I suggest to move forward, and use C++17. There is no reason to
stick with old libraries.
If you prefer to keep for a year or two the ability to compile with
C++14, then place such obsolete code inside some #ifdef #else #endif,
so that more efficient code is generated with C++17 and less efficient
code when someone insists on compiling with C++14.
In yade[1] we have moved to C++17 when building packages for ubuntu
focal 20.04 or debian buster or bullseye. The C++14 is used only when
building packages for older systems such as ubuntu xenial 16.04.
Yade can compile with eigen using C++17 without any problems. Which
is good. For example the coinor library had some problems with C++17:
and older version worked only with C++17.
best regards
Janek
[1] https://yade-dem.org/doc/ and https://gitlab.com/yade-dev/trunk
Rasmus Munk Larsen said: (by the date of Tue, 5 May 2020 14:14:19 -0700)
> I agree with you, but it seems the maintainers are very busy with other
> projects, and I am guessing they do not have time for major initiatives to
> clean up Eigen. We have dropped support for c++03 in the Tensor library,
> but a lot of things could be cleaned up if we more fully embraced c++11 or
> c++14. At this point, I only see this happening slowly, or if somebody from
> the open source community is willing to help.
>
> On Tue, May 5, 2020 at 2:05 PM Patrik Huber <patrikhuber@xxxxxxxxx> wrote:
>
> > Hi all,
> >
> > I'm a quite sad to see this post got no replies. I can only add that I
> > fully agree and think along the very same lines. I would love to see Eigen
> > move forward, move to C++14 in the 3.5 release, and drop the old cruft.
> > It's 2020 now, 6 years after C++14.
> >
> > Best wishes,
> > Patrik
> >
> > On Wed, 8 Apr 2020 at 17:20, Martin Beeger <martin.beeger@xxxxxxxxx>
> > wrote:
> >
> >> Hallo!
> >>
> >> This is continuation of the discussion from last year about
> >> compatibility, but I wanted to bring it up again, because I have some
> >> new data from my experience.
> >>
> >> Our company uses Eigen extensively and has a larger ecosystem built on
> >> it, as do many of the voices on this list. We are in the embedded world,
> >> so we are often hindered to adapt quickly to new tools, very much like
> >> the HPC community has been.
> >> But we managed to move to C++14 with our codebase in 2019. When doing
> >> that, we somewhat monitored performance and compile time during the
> >> adaption.
> >>
> >> When we starting compiling our C++98 codebase as is with C+14,
> >> performance already went up slightly. C++14 does silently move or elide
> >> copies, and modern compiler have better optimizers, and the quality of
> >> implementation of STL types got better. So if you are at all bound by
> >> performance of STL types, I would really recommend using a new compiler
> >> in C++14 mode even with your old code.
> >>
> >> This was the obvious part. Another part was less obvious. We profiled
> >> compile time, which didn't change much after the switch and started to
> >> look into stuff which was expensive to compile. And then selectively in
> >> places, where the compile time was bad we simplified the code using
> >> C++14 features to improve compile time. This allowed us with very
> >> localized changes to cut our compile time by almost in half, while, at
> >> the same time, making the code in question often a lot simpler and new
> >> features (e.g. for performance improvements) easier to implement.
> >>
> >> This is also the experience which was observed with other template heavy
> >> codebases like boost::mpl in comparison to hana and other stuff. The
> >> gains in simplicity and compile time, especially from constexpr and
> >> lambda features are not minor. They can often cut your code in half and
> >> more than double the compile time.
> >>
> >> The problem is, I am at a point where its hard to do much more to
> >> improve the compile time of my codebase significantly, because most of
> >> the compile time is brought in by 2 libraries: boost.test and Eigen. We
> >> will most likely at some point abandon boost.test due to this, like many
> >> others have already (which slowed the development and improvement of
> >> boost.test further, while the alternatives got better and getting into
> >> this downwards spiral). I would be vary happy if I am not forced to
> >> abandon Eigen after the great 10 years we had with this library. In
> >> order to avoid this I pulled a lot of tricks like explicit template
> >> instatiations, tricks to reduce includes, even pimpl-like encapsulation
> >> at performance cost to isolate from the problem, but that gets you only
> >> so far.
> >>
> >> I may well be that my use case is special, but I strongly assume that
> >> new users which today want to adopt Eigen and have to look into its
> >> internals (as you inevitable need to do at some points), will see how
> >> its written and quickly run for alternatives. This amount of macros,
> >> boilerplate and similar stuff will be an argument against this great
> >> library some day and this day may already have come.
> >>
> >> The important part is here: Eigen compile time and internal _expression_
> >> template engine code readability it was great by 2010s standards, it was
> >> ok by 2015 standards, it is borderline by 2020 standards, and unless
> >> something changes, it will be unacceptable by 2025 standards, unless the
> >> Eigen library moves along.
> >>
> >> As C++ users, we do care about backwards compatibility greatly and that
> >> is even good for me, but we should not go the C way and care about too
> >> ancient compilers. The C++ comittee doesn't (that why int is required to
> >> be 2s complement in C++20), so Eigen library maintainers IMHO should
> >> follow.
> >>
> >> What are the chances to get Eigen 3.4 out of the door with C++98 support
> >> and drop it on the devel branch afterwards and jump to C++14?
> >> What is the Eigen promises about how old yours compilers may be?
> >> Can we explicitly agree on a statement like: We vow to support up to 3
> >> year old compilers (or 5 years)?
> >>
> >> If we could agree on clear and conservative rules like we will go 3
> >> years back or 5 years back and state these on the Eigen front page,
> >> Eigen user may look at our codebase and be much more willing to accept
> >> older standards code, knowing that it will improve over time and that
> >> the user gets some useful guarantees about the future in return.
> >> There is agrument to be made that if you use a 5+ years old compiler,
> >> you really do not care about performance, something Eigen uses generally
> >> care about. So you are not in a targeted user group of Eigen. Little HPC
> >> clusters do not at all offer any way to install a more recent compilers
> >> that 5 years (this has changed a lot from a decade ago) and even in the
> >> embedded world, vendors tend to drop support or upgrade for platforms
> >> with more than 5 years old toolchains too (this has also very much
> >> changed from 10 years ago).
> >>
> >> Eigen users should be able to get a clear answer on the question when we
> >> drop C++98 (of if). That belongs on the front page IMHO.
> >>
> >> Kind regards,
> >> Martin
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
--
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)
--
Jeff Hammond
jeff.science@xxxxxxxxx
http://jeffhammond.github.io/
| Mail converted by MHonArc 2.6.19+ | http://listengine.tuxfamily.org/ |