| Hi all,
as several discussions before, this got again a good amount of responses which shows even more the importance of the topic. However, I think we have to distinguish several points mentioned here and in earlier mail threads. But first let me sum up the current status as I see it:
- Eigen is C++98 compatible but provides some (mostly API-wise) modern features not limited to one particular standard. We have move ctors (C++11) or initialisation using variadic templates (C++14) - The unsupported Tensor module has officially moved to C++11 - We agreed during earlier discussions that we should seriously consider moving to newer standards and that a simple poll would be great to get insights on what users and developers want. I've set up such a poll and sent an early version to the list but have initially postponed to release it intentionally because of the Gitlab migration and later postponed it again unintentionally (basically, I forgot it). However, I'll try to send the poll this week - The last thing I heard is that there are currently no ideas on if there will be Eigen 3.9 and how it could look like.
Some thoughts on some of the topics discussed:
1. Transparency of Eigen's current status and its future development
What are the chances to get Eigen 3.4 out of the door with C++98 support
and drop it on the devel branch afterwards and jump to C++14?
Eigen users should be able to get a clear answer on the question when we
drop C++98 (of if). That belongs on the front page IMHO. A big +1 from my side. We need an agreement on what we want and when we think this could happen. If a project has to make the decision to use Eigen or not it should have all the facts at hand. Stating that we do not know something can sometimes be more helpful than saying nothing at all. We could e.g. add a paragraph to the main page stating that we are discussing about adapting newer standards but there is no roadmap yet. This would at least show existing and potential users that we are aware of the topic.
2. Should Eigen switch to a newer standard at all? How and when should this happen?
In my opinion, there is absolutely no value in this kind of conservatism. Instead, it *only* harms.
Eigen 3.x should remain pretty much the way it is.
Just as an example, the main national supercomputer for a major US government research agency is running RHEL 7.
I guess if we all could just develop new software for brand new machines we all would instantly move on to C++20 but this is not the case. The soft- and hardware landscape is diverse and stories as the above are not uncommon, especially for applications in scientific computing which often run on big machines.
Overall this is not a simple yes/no question and there is a lot to consider here: Is it enough to provide a C++11/14/17 API? Should a release supporting newer standards also include refactored internals? E.g. Eigen heavily relies on polymorphism to implement _expression_ templates and tries to avoid virtual dispatching for performance reasons. However, modern compilers can usually de-virtualise quite well and other projects (I know e.g. of Facebook SPARTA) implement _expression_ templates using virtual functions and don't expect performance regression. In another project we just recently implemented _expression_ templates using C++17 features (e.g. extensive use of return type deduction) which works amazingly well.
Is an incremental approach preferable or should all be done in one big junk?
In my personal opinion some convenient API features could easily be implemented (even now) and we had some discussions regarding such things (e.g. vector initialisation using variadic templates or structured bindings for visitors).
3. Eigen's overall development policy
but I have noticed a certain lack of communication outside the PRs
While this is certainly true I think we all should be aware that the core devs mostly develop Eigen on a voluntary basis and what they've created is really amazing. My personal experience with open source in the end is that if you want to have something you'd be better prepared to provide it yourself. And here I don't only talk about implementing features but also about reviewing code, commenting on issues and being active on this list. We all should be willing to answer questions we can and shouldn't just wait for the core team to do it.
hopefully the energetic people calling for this change are the ones who will step forward to do the work.
To stop preaching and start doing something right away I'll refactor the survey, send it to the list and hope for a lot of participation. Hopefully we can then share some insights in a few weeks and have a better feeling on what the community wants.
Cheers, David
There are simple and established solutions to the problem of having an old system compiler or standard libraries on a cluster-type computing system. The Environment Modules system ( http://modules.sourceforge.net/) is one of them. Local compiler installs or even Docker are others. Any sysadmin worth their salt can solve these issues for users quite easily, if they just wanted to. (They could additionally upgrade the system, if they just wanted to.) > However, conservatism certainly has value. Particularly when you advertise yourself as a production library (not a research or proof of concept toy). > Eigen development is slow. The code has reached a very stable state. The last major release was in 2018. These facts are probably good for most users of Eigen. I heavily disagree. In my opinion, there is absolutely no value in this kind of conservatism. Instead, it *only* harms. Conservatism in library development has little effect on "production quality". In fact, I'd argue that libraries that choose to move faster are often better, including for production use cases, since they have judiciously cleaned up their cruft, thus attract a larger developer and user base and have a healthier ecosystem. It is proven that in software development, small lead and cycle times and frequent "deploys", be it upstream or downstream, in libraries or services, are strongly beneficial. At least part of the reason why Eigen development has slowed to a halt is said refusal to move with the times, at the right time (i.e. not six years too late). We have a classic case of "code rot" here, due to changing requirements from the user base over time. Both the internals of Eigen have become harder to maintain (things are not expressed as simply and elegantly as they could be) and the externals harder to use (crufty interfaces, slow compilation speeds, myriads of spurious warnings on newer compilers, etc.). Because nothing happened for years due to the conservative governance model, modernizing the code has now accumulated to a huge task, and it's on a developer base that has already shrunken quite a bit, judging from the average level of activity on this mailing list. This greatly exacerbates the problem. Continuous and incremental modernization in a timely(!) manner is the best way to prevent this scenario from happening. This includes (but is not limited to) moving to new C++ standards around the time they are introduced. It doesn't really matter whether one calls a newer version 3.x or 4.x -- these are just semantics in the end. No matter how a project is governed (SemVer or not, level of promised API or ABI stability, etc.), the fundamental insight has to be that *old versions do not go away* and can still be used by anyone who wishes to do so. Conservative users may choose to upgrade more rarely, at the cost of much larger pain when doing so. (A model that I do not believe in.) Most users should be encouraged to migrate to the latest released version quickly, and to be part of the continuous, incremental maintenance cycle. I have been an Eigen user and defender for years, but I cannot recommend this library anymore to others without giving huge caveats ("outdated", "seems effectively abandoned"). A strong commitment to modernity and moving with the times by the maintainers is the best and maybe only way to bring back users and thus developers. As part of this, I do hope that Eigen governance commits to moving to C++17 in the very near future. Cheers, Michael I don't have access to the machine, so I can't answer the detailed questions.
My application isn't super intensive -- but it can be used to generate input files for much more expensive computations and optimization studies. Consequently, it sometimes gets run on big iron.
My user who recently had problems because my app developed a dependency on std::regexp -- which isn't really supported by gcc 4.8. The problem is less the compiler and more the standard libraries installed on the machine. He is not building on the big machine -- he builds on a local vm on a laptop and then transfers the binary up. They also prefer to do their local setup on images that are as similar as possible to the computation environment. Perhaps they could set up local compilers and libraries, but that is a much bigger hassle than using system installed standard libraries.
The machine is not at TACC. It has nearly 250k cores and 1PB memory. It was installed when RHEL 7 was brand new. I have no idea how many cores the users are using -- a small fraction no doubt.
Rob
Is the primary compiler on said supercomputer really /usr/bin/g++? How many nodes are users running on when they build apps with this compiler?
My guess is most of the users of that system use mpicxx or something similar from modules that provides a trivial solution to this problem. If you’re referring to TACC, I’m sure they’ll recommend a bunch of compilers besides GCC-4.8.
RHEL has devtoolset for a reason. All major HPC systems have modules for a dozen or more compilers that don’t live in /usr. Eigen shouldn’t hold itself back for people who refuse to help themselves.
Jeff
I certainly understand the need to move forward. However, conservatism certainly has value. Particularly when you advertise yourself as a production library (not a research or proof of concept toy).
Just as an example, the main national supercomputer for a major US government research agency is running RHEL 7. That means gcc 4.8.x and the standard libraries that go with it. That means C++11 and most of the stuff that goes with it -- but not everything.
RHEL 8 was only released 364 days ago (depending on your time zone). It is certainly not unreasonable for installed high value production computing assets like the above machine to still be running RHEL 7. While it would be great if RHEL moved compilers forward more quickly -- that isn't how it works.
The fact is -- Eigen development is slow. The code has reached a very stable state. The last major release was in 2018. These facts are probably good for most users of Eigen.
If permission to jump to some advanced version means someone is going to go through and wholesale clean up the code and move it forward -- who is that someone? Where is this development energy coming from? Users making a request doesn't mean there is anyone to get the job done.
Eigen 3.x should remain pretty much the way it is. I hope/assume that any big jump like this would start by creating an Eigen 4.x branch/fork on GitHub and moving forward from there -- hopefully the energetic people calling for this change are the ones who will step forward to do the work.
I suggest to move forward, and use C++17. There is no reason to
stick with old libraries.
If you prefer to keep for a year or two the ability to compile with
C++14, then place such obsolete code inside some #ifdef #else #endif,
so that more efficient code is generated with C++17 and less efficient
code when someone insists on compiling with C++14.
In yade[1] we have moved to C++17 when building packages for ubuntu
focal 20.04 or debian buster or bullseye. The C++14 is used only when
building packages for older systems such as ubuntu xenial 16.04.
Yade can compile with eigen using C++17 without any problems. Which
is good. For example the coinor library had some problems with C++17:
and older version worked only with C++17.
best regards
Janek
[1] https://yade-dem.org/doc/ and https://gitlab.com/yade-dev/trunk
Rasmus Munk Larsen said: (by the date of Tue, 5 May 2020 14:14:19 -0700)
> I agree with you, but it seems the maintainers are very busy with other
> projects, and I am guessing they do not have time for major initiatives to
> clean up Eigen. We have dropped support for c++03 in the Tensor library,
> but a lot of things could be cleaned up if we more fully embraced c++11 or
> c++14. At this point, I only see this happening slowly, or if somebody from
> the open source community is willing to help.
>
> On Tue, May 5, 2020 at 2:05 PM Patrik Huber <patrikhuber@xxxxxxxxx> wrote:
>
> > Hi all,
> >
> > I'm a quite sad to see this post got no replies. I can only add that I
> > fully agree and think along the very same lines. I would love to see Eigen
> > move forward, move to C++14 in the 3.5 release, and drop the old cruft.
> > It's 2020 now, 6 years after C++14.
> >
> > Best wishes,
> > Patrik
> >
> > On Wed, 8 Apr 2020 at 17:20, Martin Beeger <martin.beeger@xxxxxxxxx>
> > wrote:
> >
> >> Hallo!
> >>
> >> This is continuation of the discussion from last year about
> >> compatibility, but I wanted to bring it up again, because I have some
> >> new data from my experience.
> >>
> >> Our company uses Eigen extensively and has a larger ecosystem built on
> >> it, as do many of the voices on this list. We are in the embedded world,
> >> so we are often hindered to adapt quickly to new tools, very much like
> >> the HPC community has been.
> >> But we managed to move to C++14 with our codebase in 2019. When doing
> >> that, we somewhat monitored performance and compile time during the
> >> adaption.
> >>
> >> When we starting compiling our C++98 codebase as is with C+14,
> >> performance already went up slightly. C++14 does silently move or elide
> >> copies, and modern compiler have better optimizers, and the quality of
> >> implementation of STL types got better. So if you are at all bound by
> >> performance of STL types, I would really recommend using a new compiler
> >> in C++14 mode even with your old code.
> >>
> >> This was the obvious part. Another part was less obvious. We profiled
> >> compile time, which didn't change much after the switch and started to
> >> look into stuff which was expensive to compile. And then selectively in
> >> places, where the compile time was bad we simplified the code using
> >> C++14 features to improve compile time. This allowed us with very
> >> localized changes to cut our compile time by almost in half, while, at
> >> the same time, making the code in question often a lot simpler and new
> >> features (e.g. for performance improvements) easier to implement.
> >>
> >> This is also the experience which was observed with other template heavy
> >> codebases like boost::mpl in comparison to hana and other stuff. The
> >> gains in simplicity and compile time, especially from constexpr and
> >> lambda features are not minor. They can often cut your code in half and
> >> more than double the compile time.
> >>
> >> The problem is, I am at a point where its hard to do much more to
> >> improve the compile time of my codebase significantly, because most of
> >> the compile time is brought in by 2 libraries: boost.test and Eigen. We
> >> will most likely at some point abandon boost.test due to this, like many
> >> others have already (which slowed the development and improvement of
> >> boost.test further, while the alternatives got better and getting into
> >> this downwards spiral). I would be vary happy if I am not forced to
> >> abandon Eigen after the great 10 years we had with this library. In
> >> order to avoid this I pulled a lot of tricks like explicit template
> >> instatiations, tricks to reduce includes, even pimpl-like encapsulation
> >> at performance cost to isolate from the problem, but that gets you only
> >> so far.
> >>
> >> I may well be that my use case is special, but I strongly assume that
> >> new users which today want to adopt Eigen and have to look into its
> >> internals (as you inevitable need to do at some points), will see how
> >> its written and quickly run for alternatives. This amount of macros,
> >> boilerplate and similar stuff will be an argument against this great
> >> library some day and this day may already have come.
> >>
> >> The important part is here: Eigen compile time and internal _expression_
> >> template engine code readability it was great by 2010s standards, it was
> >> ok by 2015 standards, it is borderline by 2020 standards, and unless
> >> something changes, it will be unacceptable by 2025 standards, unless the
> >> Eigen library moves along.
> >>
> >> As C++ users, we do care about backwards compatibility greatly and that
> >> is even good for me, but we should not go the C way and care about too
> >> ancient compilers. The C++ comittee doesn't (that why int is required to
> >> be 2s complement in C++20), so Eigen library maintainers IMHO should
> >> follow.
> >>
> >> What are the chances to get Eigen 3.4 out of the door with C++98 support
> >> and drop it on the devel branch afterwards and jump to C++14?
> >> What is the Eigen promises about how old yours compilers may be?
> >> Can we explicitly agree on a statement like: We vow to support up to 3
> >> year old compilers (or 5 years)?
> >>
> >> If we could agree on clear and conservative rules like we will go 3
> >> years back or 5 years back and state these on the Eigen front page,
> >> Eigen user may look at our codebase and be much more willing to accept
> >> older standards code, knowing that it will improve over time and that
> >> the user gets some useful guarantees about the future in return.
> >> There is agrument to be made that if you use a 5+ years old compiler,
> >> you really do not care about performance, something Eigen uses generally
> >> care about. So you are not in a targeted user group of Eigen. Little HPC
> >> clusters do not at all offer any way to install a more recent compilers
> >> that 5 years (this has changed a lot from a decade ago) and even in the
> >> embedded world, vendors tend to drop support or upgrade for platforms
> >> with more than 5 years old toolchains too (this has also very much
> >> changed from 10 years ago).
> >>
> >> Eigen users should be able to get a clear answer on the question when we
> >> drop C++98 (of if). That belongs on the front page IMHO.
> >>
> >> Kind regards,
> >> Martin
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
--
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)
|