Re: [eigen] Alignment of derived matrix class

[Piotr Trojanek <piotr.trojanek@xxxxxxxxx>]

Thanks Benoit!

Unfortunately the target architecture is plain old x86, but... I think that finally I have traced this issue!

It looks like alignment is not done for a threaded code (which is actually exactly what happen in my
robotic code - the assertion fails in a thread created with boost::thread). Please take a look at a simple
test code attached. It works perfect on Linux (x86, g++/icpc) and Solaris (sparc, g++), but the assertion
for a BAD variable fails on QNX (x86, QCC/g++).

Quick googling points to this old post for FreeBSD:
http://lists.freebsd.org/pipermail/freebsd-threads/2004-November/002730.html
and similar for OpenSolaris:
http://defect.opensolaris.org/bz/show_bug.cgi?id=10932

The problem seems, that gcc alignment works according to initial thread's stack alignment.
On QNX this alignment is done for a main thread (and main() routine), but not for a threads created
with pthread_create(). I believe this is a fault of QNX's C library, not the compiler.

I will try to dig into this issue and hope to find a solution (QNX code is opensourced) and hopefully
send at least a problem report (or hopefully a patch) to QNX.

Piotr

On Sat, Mar 6, 2010 at 06:35, Benoit Jacob <jacob.benoit.1@xxxxxxxxx> wrote:

Just a BIG HUGE question...... what CPU architecture are you on...?

We just discussed that GCC alignment attributes don't work on ARM....
so if you are on ARM it is probably the same problem, not QNX's fault!

Benoit

2010/3/5 Piotr Trojanek <piotr.trojanek@xxxxxxxxx>:
> Yes, this is Eigen 2.0 branch, exactly the lastest release 2.0.12 with all
> my std:: patches
> backported from main 2.0 branch.
>
> I have just completed one more test:
>
> 10) New alpha version of QNX, recently (just few days ago) pre released,
> http://community.qnx.com/sf/wiki/do/viewPage/projects.community/wiki/SDP_6.5.0
> which ships with gcc-4.4.2 based compiler does not solve the issue.
>
> Answering your suggestion - no I do not care about vectorization and I can
> surely live
> with EIGEN_DONT_ALIGN. Even without vectorization Eigen is next century
> product
> comparing to the hand-crafted student's matrix libraries we have been using
> for a years :-)
> The main advantage of Eigen is template-basing, which is what we need for
> real-time
> robot control. Vectorization and performance are much less important.
>
> My proposal is as follows - it seems, that I am the only user of Eigen on
> QNX.
> I will just add EIGEN_DONT_ALIGN into my framework's Makefile. If anyone
> double-report
> my issues - I suggest to add #if (defined __QNXNTO__) test in Eigen headers
> to disable vectorization.
>
> At the beginning I was thinking, that the issue is caused by broken
> inheritance,
> but apparently it is caused by compiler or the OS.
>
> Thanks all for support!
>
> Piotr
>
> On Fri, Mar 5, 2010 at 15:34, Benoit Jacob <jacob.benoit.1@xxxxxxxxx> wrote:
>>
>> This is Eigen 2.0 right?
>>
>> Piotr, do you care about vectorization in your case? If not, the
>> simple fix is to disable alignment altogether (EIGEN_DONT_ALIGN) when
>> QCC is detected. Acceptable to you?
>>
>> A more subte, yet possible, fix is to disable stack alignment
>> (attributes) but keep heap alignment which will still allow to
>> vectorize MatrixXf (but not Matrix4f). I'd only do that in the 2.0
>> branch if that's really important to you...
>>
>> In any case, if QCC doesn't honor align attributes, we can't vectorize
>> fixed size types like Matrix4f.
>>
>> Benoit
>>
>> 2010/3/5 Piotr Trojanek <piotr.trojanek@xxxxxxxxx>:
>> > I have found this page:
>> > http://eigen.tuxfamily.org/dox/WrongStackAlignment..html,
>> > which seems to be based on this post:
>> > http://forum.kde.org/viewtopic.php?f=74&t=61935
>> > but it did not solves the issue:
>> >
>> > 4) my code compiled with -mstackrealign segfaults just at the beginning
>> > (backtrace points to some std::string internals)
>> >
>> > 5) option -mincoming-stack-boundary=2 is unrecognized by QCC (I guess it
>> > comes from gcc-4.4, while current version
>> > of QCC compiler is based on gcc-4.3.1)
>> >
>> > The following options does not align data and fails with the alignment
>> > assert, line from gdb backtrace included:
>> >
>> > 6) -mpreferred-stack-boundary=2: ei_matrix_array (this=0x7f42e28)
>> >
>> > 7) -mpreferred-stack-boundary=3: ei_matrix_array (this=0x7f42e0c)
>> >
>> > 8) -malign-double: ei_matrix_array (this=0x7f42dc4)
>> >
>> > 9) both variants of forced alignment fails:
>> > EIGEN_ALIGN_128 Eigen::Matrix<double, 6, 1> servo_real_kartez_pos;
>> > __attribute__((aligned(16))) Eigen::Matrix<double, 6, 1>
>> > servo_real_kartez_pos;
>> >
>> > It looks like QCC does only does alignment when it likes... (?).
>> >
>> > On Fri, Mar 5, 2010 at 14:10, Gael Guennebaud
>> > <gael.guennebaud@xxxxxxxxx>
>> > wrote:
>> >>
>> >>
>> >> On Fri, Mar 5, 2010 at 1:58 PM, Piotr Trojanek
>> >> <piotr.trojanek@xxxxxxxxx>
>> >> wrote:
>> >>>
>> >>> Gael, thanks for your suggestions!
>> >>>
>> >>> I have verified the following:
>> >>>
>> >>> 1)  the code that breaks assertion is even the most simple variant (so
>> >>> this has nothing to do with inheritance):
>> >>>
>> >>> * Eigen::Matrix<double, 6, 1> servo_real_kartez_pos;
>> >>
>> >> not good!
>> >>
>> >> so what about:
>> >>
>> >> EIGEN_ALIGN_128 Eigen::Matrix<double, 6, 1> servo_real_kartez_pos;
>> >>
>> >> ??
>> >>
>> >> and if that does not work:
>> >>
>> >> __attribute__((aligned(16))) Eigen::Matrix<double, 6, 1>
>> >> servo_real_kartez_pos;
>> >>
>> >>
>> >> (paranoia)
>> >>
>> >>
>> >>
>> >>>
>> >>> 2) for 100% right variant of macro is used:
>> >>> #define EIGEN_ALIGN_128 __attribute__((aligned(16)))
>> >>> (added #error to all the other variants).
>> >>>
>> >>
>> >> ok
>> >>
>> >>>
>> >>> 3) my code works just fine with -DEIGEN_DONT_ALIGN switch.
>> >>>
>> >>
>> >> yes, in that case the assertions are removed.
>> >>
>> >>>
>> >>> 4) alignment attribute is honored by QCC compiler (which uses
>> >>> gcc-4.3.1
>> >>> as a backend),
>> >>> which I have verified with the following result of a test program
>> >>> attached:
>> >>>
>> >>> # ../a.out
>> >>> aligned: 8047b50
>> >>> notaligned: 8047b98
>> >>> In function nonaligned -- align.cc:24
>> >>> (reinterpret_cast<std::size_t>(array) & 0xf) ==0 -- assertion failed
>> >>> Abort (core dumped)
>> >>>
>> >>> I will appreciate any idea... It looks like QCC compiler does align
>> >>> the
>> >>> data only when it wants (?!).
>> >>
>> >> that's weird. And what if in this simple example you include Eigen, and
>> >> using EIGEN_ALIGN_128 instead of the attribute ?
>> >>
>> >> and then, what if you replace the member array by a
>> >> Eigen::Matrix<double,
>> >> 6, 1> ?
>> >>
>> >> gael
>> >>
>> >>
>> >>>
>> >>> Piotr
>> >>>
>> >>> On Fri, Mar 5, 2010 at 09:35, Gael Guennebaud
>> >>> <gael.guennebaud@xxxxxxxxx>
>> >>> wrote:
>> >>>>
>> >>>> Hi Piotr,
>> >>>>
>> >>>> this sounds really strange to me. To help to understand the cause of
>> >>>> the
>> >>>> issue have you tried, or could you try the following:
>> >>>>
>> >>>> replace B servo_real_kartez_pos; by each of the following:
>> >>>>
>> >>>> * Eigen::Matrix<double, 6, 1> servo_real_kartez_pos;
>> >>>> * A servo_real_kartez_pos;
>> >>>> * EIGEN_ALIGN_128 B servo_real_kartez_pos;
>> >>>>
>> >>>> Check that sizeof(B)==sizeof(Eigen::Matrix<double, 6, 1>)
>> >>>>
>> >>>> gael
>> >>>>
>> >>>> On Fri, Mar 5, 2010 at 12:45 AM, Piotr Trojanek
>> >>>> <piotr.trojanek@xxxxxxxxx> wrote:
>> >>>>>
>> >>>>> Dear Eigen experts,
>> >>>>>
>> >>>>> while porting some robotics matrix code to Eigen2 I have encountered
>> >>>>> the following problem
>> >>>>> on QNX platform, but I do not think it is QNX specific.
>> >>>>>
>> >>>>> I have the following class hierarchy, which I have created using
>> >>>>> guidelines from:
>> >>>>> http://www.ros.org/wiki/eigen#Creating_typedefs_for_Eigen_types
>> >>>>>
>> >>>>> // Base class (with not copy-pasted robot-specific methods)
>> >>>>> class A : public Eigen::Matrix<double, 6, 1> {
>> >>>>> public:
>> >>>>>     // Copy constructor from any Eigen matrix type
>> >>>>>     template<typename OtherDerived>
>> >>>>>     A(const Eigen::MatrixBase<OtherDerived>& other)
>> >>>>>         : BaseClass(other)
>> >>>>>     {}
>> >>>>>
>> >>>>>     // Reuse assignment operators from base class
>> >>>>>     using BaseClass::operator=;
>> >>>>>
>> >>>>>     EIGEN_MAKE_ALIGNED_OPERATOR_NEW
>> >>>>> };
>> >>>>>
>> >>>>> // Derived class type, which will hold additional methods
>> >>>>> class B : public A {
>> >>>>> public:
>> >>>>>     template<typename OtherDerived>
>> >>>>>     B(const Eigen::MatrixBase<OtherDerived>& other)
>> >>>>>         : Ft_v_vector(other)
>> >>>>>     {}
>> >>>>>
>> >>>>>     EIGEN_MAKE_ALIGNED_OPERATOR_NEW
>> >>>>> };
>> >>>>>
>> >>>>> When I run this code on Linux - it works just fine.
>> >>>>> When I run this on QNX - it fails with Unalligned Array Assert. The
>> >>>>> backtrace is here:
>> >>>>>
>> >>>>> #0  0xb033dad1 in SignalKill () from
>> >>>>> /usr/qnx641/target/qnx6/x86/lib/libc.so.3
>> >>>>> #1  0xb032c82e in raise () from
>> >>>>> /usr/qnx641/target/qnx6/x86/lib/libc.so.3
>> >>>>> #2  0xb032ab08 in abort () from
>> >>>>> /usr/qnx641/target/qnx6/x86/lib/libc.so.3
>> >>>>> #3  0xb032ad29 in __assert () from
>> >>>>> /usr/qnx641/target/qnx6/x86/lib/libc.so.3
>> >>>>> #4  0x080534ec in ei_matrix_array (this=0x7f42db4) at
>> >>>>>
>> >>>>> /opt/qnx641/target/qnx6/mrlib/include/eigen2/Eigen/src/Core/MatrixStorage.h:43
>> >>>>> #5  0x0805350f in ei_matrix_storage (this=0x7f42db4) at
>> >>>>>
>> >>>>> /opt/qnx641/target/qnx6/mrlib/include/eigen2/Eigen/src/Core/MatrixStorage.h:79
>> >>>>> #6  0x0809936f in
>> >>>>> Matrix<Eigen::CwiseNullaryOp<Eigen::ei_scalar_constant_op<double>,
>> >>>>> Eigen::Matrix<double, 6, 1, 2, 6, 1> > > (
>> >>>>>     this=0x7f42db4, other=@0x7f42ccc) at
>> >>>>>
>> >>>>> /opt/qnx641/target/qnx6/mrlib/include/eigen2/Eigen/src/Core/Matrix.h:405
>> >>>>> #7  0x080981a4 in A (this=0x7f42db4) at mrmath/ft_v_vector.cc:28
>> >>>>> #8  0x0809820f in B (this=0x7f42db4) at mrmath/ft_v_vector.cc:185
>> >>>>> #9  0x08078293 in
>> >>>>> mrrocpp::edp::common::manip_effector::compute_servo_joints_and_frame
>> >>>>> (this=0x80d7fc0) at edp_e_manip.cc:55
>> >>>>>
>> >>>>> The code, that causes the abort is just creating local, temporary
>> >>>>> variable of class B:
>> >>>>> B servo_real_kartez_pos;
>> >>>>>
>> >>>>> I have checked, that QNX compiler honours EIGEN_ALIGN_128 statement
>> >>>>> (I
>> >>>>> am 100% sure,
>> >>>>> checked with gdb for very simple structure alignment with and
>> >>>>> without
>> >>>>> this statement).
>> >>>>>
>> >>>>> The question is - where is my mistake? Is it correct to make inherit
>> >>>>> in
>> >>>>> the above way?
>> >>>>> The problem seems to be, that in frames #8 to #4 "this" is not
>> >>>>> aligned
>> >>>>> to 128bit boundary.
>> >>>>> I have tried adding EIGEN_ALIGN_128 and virtual inheritance, but
>> >>>>> this
>> >>>>> did not solve the problem.
>> >>>>>
>> >>>>> Thanks in advance!
>> >>>>>
>> >>>>> --
>> >>>>> Piotr Trojanek
>> >>>>> Robot Control and Recognition Systems Team,
>> >>>>> Institute of Control & Computation Engineering,
>> >>>>> Warsaw University of Technology
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Piotr Trojanek
>> >>
>> >
>> >
>> >
>> > --
>> > Piotr Trojanek
>> >
>>
>>
>
>
>
> --
> Piotr Trojanek
>

-- 
Piotr Trojanek
#include <iostream>
#include <cassert>
#include <pthread.h>
#include <stdio.h>

template <typename T, std::size_t size>
struct align2
{
		T array[size] __attribute__((aligned(16)));

		align2 () {
				printf("test size: %u, array address: %p\n", size, array);
				assert((reinterpret_cast<std::size_t>(array) & 0xf) == 0);
		};
};

void *
test_thread(void *)
{
	align2<double, 1> BAD;
	return NULL;
}

int
main(int argc, char *argv[])
{
		align2<double, 1> OK;
		pthread_t tid;
		pthread_create(&tid, NULL, test_thread, NULL);
		pthread_join(tid, NULL);
		return 0;
}