Re: [eigen] SSE square root

[Cyril Flaig <cflaig@xxxxxxxxxxx>]

Gael Guennebaud wrote:
> hm, sorry that's not entirely true. I increased the number of samples,
> and for the fastest version:
> 
> 1 iter = 42% of failures with  max error = 2.78308e-07
> 2 iter = 36% of failures with  max error = 2.15188e-07
> 3 iter = 34% of failures with  max error = 1.87297e-07

In the correct version it gets rounded to a float or just truncated?

-cyril

Attachment: signature.asc
Description: OpenPGP digital signature