Re: [eigen] Dropping C++17 compatibility discussion

["Janek Kozicki (yade)" <jkozicki-yade@xxxxxxxxx>]

This is very well put, Michael.

I still suggest that moving to C++17 is a better option.

best regards
Janek

Michael Hofmann said:     (by the date of Wed, 6 May 2020 09:17:40 +0200)

> There are simple and established solutions to the problem of having an old
> system compiler or standard libraries on a cluster-type computing system.
> The Environment Modules system (http://modules.sourceforge.net/) is one of
> them. Local compiler installs or even Docker are others. Any sysadmin worth
> their salt can solve these issues for users quite easily, if they just
> wanted to. (They could additionally upgrade the system, if they just wanted
> to.)
> 
> > However, conservatism certainly has value.  Particularly when you
> advertise yourself as a production library (not a research or proof of
> concept toy).
> > Eigen development is slow.  The code has reached a very stable state.
> The last major release was in 2018.  These facts are probably good for most
> users of Eigen.
> 
> I heavily disagree. In my opinion, there is absolutely no value in this
> kind of conservatism. Instead, it *only* harms.
> 
> Conservatism in library development has little effect on "production
> quality". In fact, I'd argue that libraries that choose to move faster are
> often better, including for production use cases, since they have
> judiciously cleaned up their cruft, thus attract a larger developer and
> user base and have a healthier ecosystem. It is proven that in software
> development, small lead and cycle times and frequent "deploys", be it
> upstream or downstream, in libraries or services, are strongly beneficial.
> At least part of the reason why Eigen development has slowed to a halt is
> said refusal to move with the times, at the right time (i.e. not six years
> too late).
> 
> We have a classic case of "code rot" here, due to changing requirements
> from the user base over time. Both the internals of Eigen have become
> harder to maintain (things are not expressed as simply and elegantly as
> they could be) and the externals harder to use (crufty interfaces, slow
> compilation speeds, myriads of spurious warnings on newer compilers, etc.).
> 
> Because nothing happened for years due to the conservative governance
> model, modernizing the code has now accumulated to a huge task, and it's on
> a developer base that has already shrunken quite a bit, judging from the
> average level of activity on this mailing list. This greatly exacerbates
> the problem. Continuous and incremental modernization in a timely(!) manner
> is the best way to prevent this scenario from happening. This includes (but
> is not limited to) moving to new C++ standards around the time they are
> introduced.
> 
> It doesn't really matter whether one calls a newer version 3.x or 4.x --
> these are just semantics in the end. No matter how a project is governed
> (SemVer or not, level of promised API or ABI stability, etc.), the
> fundamental insight has to be that *old versions do not go away* and can
> still be used by anyone who wishes to do so. Conservative users may choose
> to upgrade more rarely, at the cost of much larger pain when doing so. (A
> model that I do not believe in.) Most users should be encouraged to migrate
> to the latest released version quickly, and to be part of the continuous,
> incremental maintenance cycle.
> 
> I have been an Eigen user and defender for years, but I cannot recommend
> this library anymore to others without giving huge caveats ("outdated",
> "seems effectively abandoned"). A strong commitment to modernity and moving
> with the times by the maintainers is the best and maybe only way to bring
> back users and thus developers. As part of this, I do hope that Eigen
> governance commits to moving to C++17 in the very near future.
> 
> Cheers,
> Michael
> 
> On Wed, May 6, 2020 at 5:53 AM Rob McDonald <rob.a.mcdonald@xxxxxxxxx>
> wrote:
> 
> > I don't have access to the machine, so I can't answer the detailed
> > questions.
> >
> > My application isn't super intensive -- but it can be used to generate
> > input files for much more expensive computations and optimization studies.
> > Consequently, it sometimes gets run on big iron.
> >
> > My user who recently had problems because my app developed a dependency on
> > std::regexp -- which isn't really supported by gcc 4.8.  The problem is
> > less the compiler and more the standard libraries installed on the
> > machine.  He is not building on the big machine -- he builds on a local vm
> > on a laptop and then transfers the binary up.  They also prefer to do their
> > local setup on images that are as similar as possible to the computation
> > environment.  Perhaps they could set up local compilers and libraries, but
> > that is a much bigger hassle than using system installed standard libraries.
> >
> > The machine is not at TACC.  It has nearly 250k cores and 1PB memory.  It
> > was installed when RHEL 7 was brand new.  I have no idea how many cores the
> > users are using -- a small fraction no doubt.
> >
> > Rob
> >
> >
> > On Tue, May 5, 2020 at 8:01 PM Jeff Hammond <jeff.science@xxxxxxxxx>
> > wrote:
> >
> >> Is the primary compiler on said supercomputer really /usr/bin/g++? How
> >> many nodes are users running on when they build apps with this compiler?
> >>
> >> My guess is most of the users of that system use mpicxx or something
> >> similar from modules that provides a trivial solution to this problem. If
> >> you’re referring to TACC, I’m sure they’ll recommend a bunch of compilers
> >> besides GCC-4.8.
> >>
> >> RHEL has devtoolset for a reason. All major HPC systems have modules for
> >> a dozen or more compilers that don’t live in /usr. Eigen shouldn’t hold
> >> itself back for people who refuse to help themselves.
> >>
> >> Jeff
> >>
> >> On Tue, May 5, 2020 at 3:21 PM Rob McDonald <rob.a.mcdonald@xxxxxxxxx>
> >> wrote:
> >>
> >>> I certainly understand the need to move forward.  However, conservatism
> >>> certainly has value.  Particularly when you advertise yourself as a
> >>> production library (not a research or proof of concept toy).
> >>>
> >>> Just as an example, the main national supercomputer for a major US
> >>> government research agency is running RHEL 7.  That means gcc 4.8.x and the
> >>> standard libraries that go with it.  That means C++11 and most of the stuff
> >>> that goes with it -- but not everything.
> >>>
> >>> RHEL 8 was only released 364 days ago (depending on your time zone).  It
> >>> is certainly not unreasonable for installed high value production computing
> >>> assets like the above machine to still be running RHEL 7.  While it would
> >>> be great if RHEL moved compilers forward more quickly -- that isn't how it
> >>> works.
> >>>
> >>> The fact is -- Eigen development is slow.  The code has reached a very
> >>> stable state.  The last major release was in 2018.  These facts are
> >>> probably good for most users of Eigen.
> >>>
> >>> If permission to jump to some advanced version means someone is going to
> >>> go through and wholesale clean up the code and move it forward -- who is
> >>> that someone?  Where is this development energy coming from?  Users making
> >>> a request doesn't mean there is anyone to get the job done.
> >>>
> >>> Eigen 3.x should remain pretty much the way it is.  I hope/assume that
> >>> any big jump like this would start by creating an Eigen 4.x branch/fork on
> >>> GitHub and moving forward from there -- hopefully the energetic people
> >>> calling for this change are the ones who will step forward to do the work.
> >>>
> >>> Rob
> >>>
> >>>
> >>>
> >>> On Tue, May 5, 2020 at 3:01 PM Janek Kozicki (yade) <
> >>> jkozicki-yade@xxxxxxxxx> wrote:
> >>>
> >>>> I suggest to move forward, and use C++17. There is no reason to
> >>>> stick with old libraries.
> >>>>
> >>>> If you prefer to keep for a year or two the ability to compile with
> >>>> C++14, then place such obsolete code inside some #ifdef #else #endif,
> >>>> so that more efficient code is generated with C++17 and less efficient
> >>>> code when someone insists on compiling with C++14.
> >>>>
> >>>> In yade[1] we have moved to C++17 when building packages for ubuntu
> >>>> focal 20.04 or debian buster or bullseye. The C++14 is used only when
> >>>> building packages for older systems such as ubuntu xenial 16.04.
> >>>>
> >>>> Yade can compile with eigen using C++17 without any problems. Which
> >>>> is good. For example the coinor library had some problems with C++17:
> >>>> and older version worked only with C++17.
> >>>>
> >>>> best regards
> >>>> Janek
> >>>>
> >>>> [1] https://yade-dem.org/doc/ and https://gitlab.com/yade-dev/trunk
> >>>>
> >>>>
> >>>>
> >>>> Rasmus Munk Larsen said:     (by the date of Tue, 5 May 2020 14:14:19
> >>>> -0700)
> >>>>
> >>>> > I agree with you, but it seems the maintainers are very busy with
> >>>> other
> >>>> > projects, and I am guessing they do not have time for major
> >>>> initiatives to
> >>>> > clean up Eigen. We have dropped support for c++03 in the Tensor
> >>>> library,
> >>>> > but a lot of things could be cleaned up if we more fully embraced
> >>>> c++11 or
> >>>> > c++14. At this point, I only see this happening slowly, or if
> >>>> somebody from
> >>>> > the open source community is willing to help.
> >>>> >
> >>>> > On Tue, May 5, 2020 at 2:05 PM Patrik Huber <patrikhuber@xxxxxxxxx>
> >>>> wrote:
> >>>> >
> >>>> > > Hi all,
> >>>> > >
> >>>> > > I'm a quite sad to see this post got no replies. I can only add
> >>>> that I
> >>>> > > fully agree and think along the very same lines. I would love to
> >>>> see Eigen
> >>>> > > move forward, move to C++14 in the 3.5 release, and drop the old
> >>>> cruft.
> >>>> > > It's 2020 now, 6 years after C++14.
> >>>> > >
> >>>> > > Best wishes,
> >>>> > > Patrik
> >>>> > >
> >>>> > > On Wed, 8 Apr 2020 at 17:20, Martin Beeger <martin.beeger@xxxxxxxxxx
> >>>> >
> >>>> > > wrote:
> >>>> > >
> >>>> > >> Hallo!
> >>>> > >>
> >>>> > >> This is continuation of the discussion from last year about
> >>>> > >> compatibility, but I wanted to bring it up again, because I have
> >>>> some
> >>>> > >> new data from my experience.
> >>>> > >>
> >>>> > >> Our company uses Eigen extensively and has a larger ecosystem
> >>>> built on
> >>>> > >> it, as do many of the voices on this list. We are in the embedded
> >>>> world,
> >>>> > >> so we are often hindered to adapt quickly to new tools, very much
> >>>> like
> >>>> > >> the HPC community has been.
> >>>> > >> But we managed to move to C++14 with our codebase in 2019. When
> >>>> doing
> >>>> > >> that, we somewhat monitored performance and compile time during the
> >>>> > >> adaption.
> >>>> > >>
> >>>> > >> When we starting compiling our C++98 codebase as is with C+14,
> >>>> > >> performance already went up slightly. C++14 does silently move or
> >>>> elide
> >>>> > >> copies, and modern compiler have better optimizers, and the
> >>>> quality of
> >>>> > >> implementation of  STL types got better. So if you are at all
> >>>> bound by
> >>>> > >> performance of STL types, I would really recommend using a new
> >>>> compiler
> >>>> > >> in C++14 mode even with your old code.
> >>>> > >>
> >>>> > >> This was the obvious part. Another part was less obvious. We
> >>>> profiled
> >>>> > >> compile time, which didn't change much after the switch and
> >>>> started to
> >>>> > >> look into stuff which was expensive to compile. And then
> >>>> selectively in
> >>>> > >> places, where the compile time was bad we simplified the code using
> >>>> > >> C++14 features to improve compile time. This allowed us with very
> >>>> > >> localized changes to cut our compile time by almost in half,
> >>>> while, at
> >>>> > >> the same time, making the code in question often a lot simpler and
> >>>> new
> >>>> > >> features (e.g. for performance improvements) easier to implement.
> >>>> > >>
> >>>> > >> This is also the experience which was observed with other template
> >>>> heavy
> >>>> > >> codebases like boost::mpl in comparison to hana and other stuff.
> >>>> The
> >>>> > >> gains in simplicity and compile time, especially from constexpr and
> >>>> > >> lambda features are not minor. They can often cut your code in
> >>>> half and
> >>>> > >> more than double the compile time.
> >>>> > >>
> >>>> > >> The problem is, I am at a point where its hard to do much more to
> >>>> > >> improve the compile time of my codebase significantly, because
> >>>> most of
> >>>> > >> the compile time is brought in by 2 libraries: boost.test and
> >>>> Eigen. We
> >>>> > >> will most likely at some point abandon boost.test due to this,
> >>>> like many
> >>>> > >> others have already (which slowed the development and improvement
> >>>> of
> >>>> > >> boost.test further, while the alternatives got better and getting
> >>>> into
> >>>> > >> this downwards spiral). I would be vary happy if I am not forced to
> >>>> > >> abandon Eigen after the great 10 years we had with this library.. In
> >>>> > >> order to avoid this I pulled a lot of tricks like explicit template
> >>>> > >> instatiations, tricks to reduce includes, even pimpl-like
> >>>> encapsulation
> >>>> > >> at performance cost to isolate from the problem, but that gets you
> >>>> only
> >>>> > >> so far.
> >>>> > >>
> >>>> > >> I may well be that my use case is special, but I strongly assume
> >>>> that
> >>>> > >> new users which today want to adopt Eigen and have to look into its
> >>>> > >> internals (as you inevitable need to do at some points), will see
> >>>> how
> >>>> > >> its written and quickly run for alternatives. This amount of
> >>>> macros,
> >>>> > >> boilerplate and similar stuff will be an argument against this
> >>>> great
> >>>> > >> library some day and this day may already have come.
> >>>> > >>
> >>>> > >> The important part is here: Eigen compile time and internal
> >>>> expression
> >>>> > >> template engine code readability it was great by 2010s standards,
> >>>> it was
> >>>> > >> ok by 2015 standards, it is borderline by 2020 standards, and
> >>>> unless
> >>>> > >> something changes, it will be unacceptable by 2025 standards,
> >>>> unless the
> >>>> > >> Eigen library moves along.
> >>>> > >>
> >>>> > >> As C++ users, we do care about backwards compatibility greatly and
> >>>> that
> >>>> > >> is even good for me, but we should not go the C way and care about
> >>>> too
> >>>> > >> ancient compilers. The C++ comittee doesn't (that why int is
> >>>> required to
> >>>> > >> be 2s complement in C++20), so Eigen library maintainers IMHO
> >>>> should
> >>>> > >> follow.
> >>>> > >>
> >>>> > >> What are the chances to get Eigen 3.4 out of the door with C++98
> >>>> support
> >>>> > >> and drop it on the devel branch afterwards and jump to C++14?
> >>>> > >> What is the Eigen promises about how old yours compilers may be?
> >>>> > >> Can we explicitly agree on a statement like: We vow to support up
> >>>> to 3
> >>>> > >> year old compilers (or 5 years)?
> >>>> > >>
> >>>> > >> If we could agree on clear and conservative rules like we will go 3
> >>>> > >> years back or 5 years back and state these on the Eigen front page,
> >>>> > >> Eigen user may look at our codebase and be much more willing to
> >>>> accept
> >>>> > >> older standards code, knowing that it will improve over time and
> >>>> that
> >>>> > >> the user gets some useful guarantees about the future in return.
> >>>> > >> There is agrument to be made that if you use a 5+ years old
> >>>> compiler,
> >>>> > >> you really do not care about performance, something Eigen uses
> >>>> generally
> >>>> > >> care about. So you are not in a targeted user group of Eigen.
> >>>> Little HPC
> >>>> > >> clusters do not at all offer any way to install a more recent
> >>>> compilers
> >>>> > >> that 5 years (this has changed a lot from a decade ago) and even
> >>>> in the
> >>>> > >> embedded world, vendors tend to drop support or upgrade for
> >>>> platforms
> >>>> > >> with more than 5 years old toolchains too (this has also very much
> >>>> > >> changed from 10 years ago).
> >>>> > >>
> >>>> > >> Eigen users should be able to get a clear answer on the question
> >>>> when we
> >>>> > >> drop C++98 (of if). That belongs on the front page IMHO.
> >>>> > >>
> >>>> > >> Kind regards,
> >>>> > >> Martin
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>> > >>
> >>>>
> >>>>
> >>>> --
> >>>> --
> >>>> Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
> >>>> Gdańsk University of Technology
> >>>> Faculty of Applied Physics and Mathematics
> >>>> Department of Theoretical Physics and Quantum Information
> >>>> --
> >>>> http://yade-dem.org/
> >>>> http://pg.edu.pl/jkozicki (click English flag on top right)
> >>>>
> >>>>
> >>>> --
> >> Jeff Hammond
> >> jeff.science@xxxxxxxxx
> >> http://jeffhammond.github.io/
> >>
> >


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)