Re: [eigen] Windows Testing Issues

[ Thread Index | Date Index | More lists.tuxfamily.org/eigen Archives ]


Hi!
I was able to reproduce Robert´s results with the latest MSVC 2019 and tip. You can find my timings here. You can see the corresponding timings for GCC 7 here
Firstly, it seems the tests sparse_product, sparse_extra and sparse_basic are anomalies. Those tests took ~1h to compile for Robert and for me they failed to compile with “Fatal Error C1002: compiler is out of heap space in pass 2” (On a machine with 32 GB RAM).

1. I would suggest splitting up the long functions in sparse_product.cpp and sparse_basic.cpp. They do seem unnecessarily long.

Apart from that, it seems that MSVC has a problem with Eigen´s use of FORCE_INLINE. 

I did a bisect with some tests and found that the following two commits were mainly responsible for increasing compile times:

commit 558698b0ff5d3a26a328d67c8fdece999edd0f96
Author: Gael Guennebaud <g.gael@xxxxxxx>
Date:   Sun Jan 31 16:34:10 2016 +0100

    Bug 667: declare several critical functions as FORECE_INLINE to make ICC happier.

commit 51e535dc66869c66325861c91a6c26bf3bbe9891
Author: Gael Guennebaud <g.gael@xxxxxxx>
Date:   Tue May 31 17:23:42 2016 +0200

    Bug 1181: help MSVC inlining.

In particular 558698b0ff5 pushed one test from 30s to 20m compile time. It is going to take some effort to measure the gain in execution time vs the cost in compile time.  

I issued a tracking bug here.

Den tors 13 juni 2019 kl 22:14 skrev Rob Conde <rob.conde@xxxxxxxxxxxxxxxx>:
Ok - so I found out the crashes were due to this bug:

https://developercommunity.visualstudio.com/content/problem/516494/code-generation-bug-sse-unaligned-load-stripped-aw.html

I updated to the latest VS2019, and retested again with the C++11 tests enabled. The build still took ~7 hours and all tests passed. So, some sanity has been restored. I realized I wasn't on the latest VS2017 either, so I will re-test that as well. Even if there's no improvement there, I assume there is probably little hope of anything getting fixed there.

After that, I will test the head again.

Rob

From: Christoph Hertzberg <chtz@xxxxxxxxxxxxxxxxxxxxxxxx>
Sent: Wednesday, June 12, 2019 4:56 AM
To: eigen@xxxxxxxxxxxxxxxxxxx
Subject: Re: [eigen] Windows Testing Issues
 
Hi!

On 12/06/2019 02.59, Rob Conde wrote:
> Total Build Time: ~7 hours
>
> [...]
>
> So a great improvement there...although 7 hours is still very long. The eigen docs quote '3 hours', but I'm unsure if that is using multiple cores and/or when that number was captured? This, and the former run, were all done serially.

I think the 3 hours were for a serial build, but that section has not
been updated for years, so the number is quite outdated.

>
> However, there are more failures:
>
>    *   [...]  (Exit code 0xc0000409)
>    *   [...]  (SEGFAULT)

Some searching tells me that "Exit code 0xc0000409" means
STATUS_STACK_BUFFER_OVERRUN, i.e., it looks like Eigen uses too much
stack space for some of the tests (this could also explain the
segmentation faults).
You can either decrease EIGEN_STACK_ALLOCATION_LIMIT (this defaults to
(128*1024), but that is per temporary/per matrix, i.e., with some mild
recursion this could easily exceed a few MiB).
Or you can tell your compiler to increase the stack-size. According to
https://docs.microsoft.com/en-us/cpp/build/reference/f-set-stack-size
the corresponding option is "/F". I guess on 64bit systems there is
nothing wrong with simply setting it to 16MiB or higher (this should
just be virtual memory until it is actually used).

You can also run each test manually from the command-line to see more
details when it crashes.


> I should also note that in this test, and the former VS2017 test, I was using the default CMake settings...so C++11 tests were not enabled/tested.. Although, many cxx11_tensor_xxxxx tests DID run...which I find confusing. I need to go back and time on gcc as well for reference.

That is expected behavior. Admittedly, the naming is not 100% accurate
here: With C++11 disabled, some of the cxx11_tensor-tests run in
"emulated c++11 mode" (mostly these are overloads for functions/classes
with 1-5 parameters instead of variadic templates -- and
re-implementations of simple classes like std::array).


Cheers,
Christoph


>
> Rob
> ________________________________
> From: Christoph Hertzberg <chtz@xxxxxxxxxxxxxxxxxxxxxxxx>
> Sent: Tuesday, June 11, 2019 8:08 AM
> To: eigen@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [eigen] Windows Testing Issues
>
> Hi,
>
> thanks for investigating this!
>
> These build-times are really far from great, to say it mildly ...
>
> If you have time to further investigate this, you could start removing
> parts of each (or some) test until build-times get reasonable (i.e.,
> cancel the build if it takes more than a few minutes -- depending on
> your patience). If that works reasonably well, we can further split the
> current tests into smaller ones.
>
> On the other hand, if you find very small tests which take a lot to
> build, this could help the MSVC maintainers to look for issues in their
> compiler, or it might help us to find workarounds for this.
>
>
> Also, if you have time, could you check the current development branch
> ("default" in hg) and the latest 3.2 branch if there were any large
> regressions or progressions in build-time (some tests were newly
> introduced or split/renumbered between releases, though)
>
> Cheers,
> Christoph
>
> On 11/06/2019 01.58, Rob Conde wrote:
>> Ok - I ran 3.3.7 through the testing.
>>
>>     *   cmake ..\eigen_3.3.7 -G "Visual Studio 15 Win64" -Thost=x64
>>        *   I later realized that VS also uses the 32-bit toolchain by default, but I doubt it has a big effect here
>>     *   ctest -C Release -D Experimental
>>
>> The complete build took about 22 hours. I flagged anything that took longer than ten minutes (I calculated build time as the modified time of the test minus the modified time of the previous test).
>>
>>     *   array_reverse_8 (16 mins)
>>     *   geo_quaternion_2 (25 mins)
>>     *   geo_quaternion_4 (30 mins)
>>     *   mixingtypes_5 (12 mins)
>>     *   product_small_3 (11 mins)
>>     *   product_small_4 (23 mins)
>>     *   sparse_extra_1 (98 mins)
>>     *   sparse_extra_2 (215 mins)
>>     *   sparse_extra_3 (105 mins)
>>     *   sparse_product_1 (108 mins)
>>     *   sparse_product_2 (256 mins)
>>     *   sparse_product_3 (104 mins)
>>
>> Not sure if that list forms a pattern. I will continue investigating when i get a chance.
>>
>> The actual tests took...12 mins. Only 1 failed.
>>
>> Rob
>> ________________________________
>> From: Rob Conde <rob.conde@xxxxxxxxxxxxxxxx>
>> Sent: Monday, June 10, 2019 9:53 AM
>> To: eigen@xxxxxxxxxxxxxxxxxxx
>> Subject: Re: [eigen] Windows Testing Issues
>>
>> Hey Edward,
>>      I did try that quickly at one point for a single test but I don't remember seeing a big difference, but I will try to do something more comprehensive. When I get a chance (sigh) I will also see if I can distill a simple example which i can report to the visual studio team. At this point I haven't spent much time trying to find a pattern in the things that take a long time to compile, if there is one, but that would be useful too I think.
>>
>> Rob
>> ________________________________
>> From: Edward Lam <edward@xxxxxxxxxx>
>> Sent: Monday, June 10, 2019 9:03 AM
>> To: eigen@xxxxxxxxxxxxxxxxxxx
>> Subject: Re: [eigen] Windows Testing Issues
>>
>> Hi Rob,
>>
>> On 6/10/2019 7:57 AM, Rob Conde wrote:
>>> This has been running for about 11 hours now (still on the build part).
>>> Obviously this is a visual studio issue, but I'm not sure when it was
>>> introduced. We're not seeing anything like this in the our usage of
>>> Eigen...although to a smaller degree it might be hard to tease out of the larger
>>> picture.
>>
>> It might be worthwhile trying out the latest Visual Studio 2019 just to see if
>> there's any behaviour change.
>>
>> -Edward
>>
>>
>>
>
> --
>    Dr.-Ing. Christoph Hertzberg
>
>    Besuchsadresse der Nebengeschäftsstelle:
>    DFKI GmbH
>    Robotics Innovation Center
>    Robert-Hooke-Straße 5
>    28359 Bremen, Germany
>
>    Postadresse der Hauptgeschäftsstelle Standort Bremen:
>    DFKI GmbH
>    Robotics Innovation Center
>    Robert-Hooke-Straße 1
>    28359 Bremen, Germany
>
>    Tel.:     +49 421 178 45-4021
>    Zentrale: +49 421 178 45-0
>    E-Mail:   christoph.hertzberg@xxxxxxx
>
>    Weitere Informationen: http://www.dfki.de/robotik
>     -------------------------------------------------------------
>     Deutsches Forschungszentrum für Künstliche Intelligenz GmbH
>     Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany
>
>     Geschäftsführung:
>     Prof. Dr. Jana Koehler (Vorsitzende)
>     Dr. Walter Olthoff
>
>     Vorsitzender des Aufsichtsrats:
>     Prof. Dr. h.c. Hans A. Aukes
>     Amtsgericht Kaiserslautern, HRB 2313
>     -------------------------------------------------------------
>
>
>
>

--
  Dr.-Ing. Christoph Hertzberg

  Besuchsadresse der Nebengeschäftsstelle:
  DFKI GmbH
  Robotics Innovation Center
  Robert-Hooke-Straße 5
  28359 Bremen, Germany

  Postadresse der Hauptgeschäftsstelle Standort Bremen:
  DFKI GmbH
  Robotics Innovation Center
  Robert-Hooke-Straße 1
  28359 Bremen, Germany

  Tel.:     +49 421 178 45-4021
  Zentrale: +49 421 178 45-0
  E-Mail:   christoph.hertzberg@xxxxxxx

  Weitere Informationen: http://www.dfki.de/robotik
   -------------------------------------------------------------
   Deutsches Forschungszentrum für Künstliche Intelligenz GmbH
   Trippstadter Strasse 122, D-67663 Kaiserslautern, Germany

   Geschäftsführung:
   Prof. Dr. Jana Koehler (Vorsitzende)
   Dr. Walter Olthoff

   Vorsitzender des Aufsichtsrats:
   Prof. Dr. h.c. Hans A. Aukes
   Amtsgericht Kaiserslautern, HRB 2313
   -------------------------------------------------------------





Mail converted by MHonArc 2.6.19+ http://listengine.tuxfamily.org/