I agree with Hauke that this deserves checking, and it should probably be checked by default in debug builds, otherwise people will just not think of enabling the right preprocessor switch.
But it's probably never OK to assert on anything that can depend on floating point values: that gives too finicky crashes (it doesn't help much that it's only #ifndef NDEBUG).
What you could in debug builds, is mimic the behavior of floating point numbers in 'bad' cases: you could simply return a rotation filled with NaN coefficients (based on a check you only perform in debug builds).
You would still have to use a quite large tolerance to avoid this kicking in when it's not wanted... do we still have the 'dummy' precision values? That would be a decent choice.
NaN helps debugging in 2 ways:
- it's particularly obvious that something went wrong;
- you can turn on FP signals, so you could catch the first such error in your debugger.
Benoit