You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Aleš Trček <Al...@halcom.si> on 2013/12/11 19:11:41 UTC
RE: Building qpid C++ broker on AIX
Hi all,
After some while I got back to building qpidc (just the client part actually) on AIX. In the meanwhile I got an upgraded box with AIX 7.1 (on POWER CPU) and latest compiler xlc 12.1. I then built boost 1.47.0 (with some IBM patch required for xlc) and after a few more problems managed to build qpid (make all) v0.24 to the point of linking qpidd, which fails because of some undefined boost program_options symbols. I'll deal with this later, since I need the client and required libraries libqpidtypes.so, libqpidcommon.so, libqpidclient.so and libqpidmessaging.so got built OK AFAICT.
So, I try to build a simple client using example from the web site, and the executable is created OK. When I run it though, it freezes at one point and stays like this indefinetly. It has to do something with connection and threads but I haven't been able to figure out the cause. I did some analysis and here are the facts:
gdb output (we can see where it waits in main thread):
++++++++++++++++++
#4 0xd051d660 in pthread_cond_wait () from /usr/lib/libpthreads.a(shr_xpg5.o)
#5 0xd6c83cc8 in qpid::sys::Condition::wait( (qpid::sys::Mutex &)) (this=0x201b97b0, mutex=@0x201b977c: ???)
at /temporar/qpid_build/qpidc-0.24/src/../include/qpid/sys/posix/Condition.h:63
#6 0xd6c83d9c in qpid::sys::Monitor::wait( (void)) (this=0x201b977c) at /temporar/qpid_build/qpidc-0.24/src/../include/qpid/sys/Monitor.h:41
#7 0xd6c83e60 in waitFor__Q3_4qpid6client12StateManagerFQ2_3std3setXTiTQ2_3std4lessXTi_TQ2_3std9allocatorXTi__ (this=0x201b9778, desired=@0x2ff21910: ???)
at /temporar/qpid_build/qpidc-0.24/src/qpid/client/StateManager.cpp:51
...
++++++++++++++++++
Specifically this is here:
++++++++++++++++++
qpid/sys/posix/Condition.h:
62 void Condition::wait(Mutex& mutex) {
63 QPID_POSIX_ASSERT_THROW_IF(pthread_cond_wait(&condition, &mutex.mutex)); // <====
64 }
++++++++++++++++++
Qpid logs:
Client log:
++++++++++++++++++
2013-12-10 17:35:47 [Client] debug Created connection 192.168.0.142:5672 with {}
2013-12-10 17:35:47 [Client] debug Starting connection, urls=[192.168.0.142:5672]
2013-12-10 17:35:47 [Client] info Trying to connect to 192.168.0.142:5672...
2013-12-10 17:35:47 [Client] debug Created IO thread: 0
2013-12-10 17:35:47 [Network] debug TCPConnector created for 0-10
2013-12-10 17:35:47 [Client] info Set TCP_NODELAY
2013-12-10 17:35:47 [System] info Connecting: 192.168.0.142:5672
++++++++++++++++++
Server log (this is trace level!):
++++++++++++++++++
2013-12-11 16:41:32 [System] error Connection qpid.192.168.0.142:5672-192.168.0.135:47415 No protocol received closing
++++++++++++++++++
Netstat first shows established connection and then close_wait:
++++++++++++++++++
tcp4 0 0 192.168.0.135.47401 192.168.0.142.5672 ESTABLISHED
...
tcp4 0 0 192.168.0.135.47401 192.168.0.142.5672 CLOSE_WAIT
...
++++++++++++++++++
Output of truss (similar to strace on Linuxes):
++++++++++++++++++
...
8454278: 29950155: socket(2, 1, 0) = 5
8454278: 29950155: kioctl(5, -2147195266, 0x2FF20F10, 0x00000000) = 0
8454278: 29950155: kioctl(5, -2147195267, 0x2FF20F10, 0x00000000) = 0
8454278: 29950155: kfcntl(5, F_SETFL, 0x00000004) = 0
8454278: 29950155: getsockopt(5, 65535, 4104, 0x2FF21374, 0x2FF21370) = 0
8454278: 29950155: connext(5, 0x201FB688, 16) Err#55 EINPROGRESS
8454278: 29950155: ngetsockname(5, 0x2FF20BD0, 0x2FF210DC) = 0
8454278: 29950155: kwrite(4, " ", 1) = 1
[and then nothing]
++++++++++++++++++
Based on all this it seems that the connection get established, but then something weird is happening with the Poller thread. When I try to debug the run() function it gets stuck on if clause in line 724:
++++++++++++++++++
if (lastReturnedHandle) { // <== here
impl->resetMode(*lastReturnedHandle);
lastReturnedHandle = 0;
}
++++++++++++++++++
gdb session with trace log on stdout:
++++++++++++++++++
(gdb) break PosixPoller.cpp:724
No source file named PosixPoller.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (PosixPoller.cpp:724) pending.
(gdb) run
Starting program: /temporar/atk/qtmin
[New Thread 1]
entry
args
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Client] debug Created connection 192.168.0.142:5672 with {}
conn
2013-12-11 18:52:23 [Client] debug Starting connection, urls=[192.168.0.142:5672]
2013-12-11 18:52:23 [Client] info Trying to connect to 192.168.0.142:5672...
2013-12-11 18:52:23 [Client] debug Created IO thread: 0
[New Thread 258]
[Switching to Thread 258]
Breakpoint 1, qpid::sys::Poller::wait( (qpid::sys::Duration)) (this=0x201b7e98, event=@0x201d8a88: ???, timeout=<incomplete type>)
at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:724
724 if (lastReturnedHandle) {
(gdb) bt
#0 qpid::sys::Poller::wait( (qpid::sys::Duration)) (this=0x201b7e98, event=@0x201d8a88: ???, timeout=<incomplete type>)
at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:724
#1 0xd6070dd4 in qpid::sys::Poller::run( (void)) (this=0x201b7e98) at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:690
#2 0xd622fb44 in qpid::sys::Thread.cpp-0::runRunnable( (void *)) (p=0x201b7e98) at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/Thread.cpp:35
#3 0xd04fad88 in _pthread_body () from /usr/lib/libpthreads.a(shr_xpg5.o)
#4 0x00000000 in ?? ()
(gdb) step
2013-12-11 18:52:33 [Network] debug TCPConnector created for 0-10
2013-12-11 18:52:33 [Client] info Set TCP_NODELAY
2013-12-11 18:52:33 [System] info Connecting: 192.168.0.142:5672
[and then the endless wait]
++++++++++++++++++
If you managed to stay with me to this point, any ideas? :) Please note that I had to hack quite some things to make it build on AIX, though semantically there should be no change. Beside this I had to take ifaddrs.h and ifaddrs.cpp from somebody on samba.org list and link with it, since this is not included in AIX - I don't know if this should have any impact.
Btw, this is how I built the exe:
xlC_r -g -o qtmin ifaddrs.c qtest.cpp -I/temporar/qpid_build/qpidc-0.24/include -L/temporar/qpid_build/qpidc-0.24/build/src/ -lqpidcommon -lqpidmessaging -lpthreads -Wl,-brtl
I really appreciate any help with this.
Regards,
Ales
RE: Building qpid C++ broker on AIX
Posted by Aleš Trček <Al...@halcom.si>.
Heh, I got it. The only problem here was that the qpid libraries were built with 'xlC' instead of 'xlC_r', which is the exe name you should use when using threads (it uses thread safe versions of libraries or sth). After rebuild, it just started working. Interesting that there are no warnings or errors just freeze. Well, I'm glad this works now, I just have to test with a real application instead of test sample now.
When I manage to build the broker too (or if I get stuck, heh) I'll let you know. I'll post patches for AIX too when everything is tested.
Regards,
Ales
-----Original Message-----
From: Aleš Trček [mailto:Ales.Trcek@halcom.si]
Sent: Wednesday, December 11, 2013 7:12 PM
To: users@qpid.apache.org
Subject: RE: Building qpid C++ broker on AIX
Hi all,
After some while I got back to building qpidc (just the client part actually) on AIX. In the meanwhile I got an upgraded box with AIX 7.1 (on POWER CPU) and latest compiler xlc 12.1. I then built boost 1.47.0 (with some IBM patch required for xlc) and after a few more problems managed to build qpid (make all) v0.24 to the point of linking qpidd, which fails because of some undefined boost program_options symbols. I'll deal with this later, since I need the client and required libraries libqpidtypes.so, libqpidcommon.so, libqpidclient.so and libqpidmessaging.so got built OK AFAICT.
So, I try to build a simple client using example from the web site, and the executable is created OK. When I run it though, it freezes at one point and stays like this indefinetly. It has to do something with connection and threads but I haven't been able to figure out the cause. I did some analysis and here are the facts:
gdb output (we can see where it waits in main thread):
++++++++++++++++++
#4 0xd051d660 in pthread_cond_wait () from /usr/lib/libpthreads.a(shr_xpg5.o)
#5 0xd6c83cc8 in qpid::sys::Condition::wait( (qpid::sys::Mutex &)) (this=0x201b97b0, mutex=@0x201b977c: ???)
at /temporar/qpid_build/qpidc-0.24/src/../include/qpid/sys/posix/Condition.h:63
#6 0xd6c83d9c in qpid::sys::Monitor::wait( (void)) (this=0x201b977c) at /temporar/qpid_build/qpidc-0.24/src/../include/qpid/sys/Monitor.h:41
#7 0xd6c83e60 in waitFor__Q3_4qpid6client12StateManagerFQ2_3std3setXTiTQ2_3std4lessXTi_TQ2_3std9allocatorXTi__ (this=0x201b9778, desired=@0x2ff21910: ???)
at /temporar/qpid_build/qpidc-0.24/src/qpid/client/StateManager.cpp:51
...
++++++++++++++++++
Specifically this is here:
++++++++++++++++++
qpid/sys/posix/Condition.h:
62 void Condition::wait(Mutex& mutex) {
63 QPID_POSIX_ASSERT_THROW_IF(pthread_cond_wait(&condition, &mutex.mutex)); // <====
64 }
++++++++++++++++++
Qpid logs:
Client log:
++++++++++++++++++
2013-12-10 17:35:47 [Client] debug Created connection 192.168.0.142:5672 with {}
2013-12-10 17:35:47 [Client] debug Starting connection, urls=[192.168.0.142:5672]
2013-12-10 17:35:47 [Client] info Trying to connect to 192.168.0.142:5672...
2013-12-10 17:35:47 [Client] debug Created IO thread: 0
2013-12-10 17:35:47 [Network] debug TCPConnector created for 0-10
2013-12-10 17:35:47 [Client] info Set TCP_NODELAY
2013-12-10 17:35:47 [System] info Connecting: 192.168.0.142:5672
++++++++++++++++++
Server log (this is trace level!):
++++++++++++++++++
2013-12-11 16:41:32 [System] error Connection qpid.192.168.0.142:5672-192.168.0.135:47415 No protocol received closing
++++++++++++++++++
Netstat first shows established connection and then close_wait:
++++++++++++++++++
tcp4 0 0 192.168.0.135.47401 192.168.0.142.5672 ESTABLISHED
...
tcp4 0 0 192.168.0.135.47401 192.168.0.142.5672 CLOSE_WAIT
...
++++++++++++++++++
Output of truss (similar to strace on Linuxes):
++++++++++++++++++
...
8454278: 29950155: socket(2, 1, 0) = 5
8454278: 29950155: kioctl(5, -2147195266, 0x2FF20F10, 0x00000000) = 0
8454278: 29950155: kioctl(5, -2147195267, 0x2FF20F10, 0x00000000) = 0
8454278: 29950155: kfcntl(5, F_SETFL, 0x00000004) = 0
8454278: 29950155: getsockopt(5, 65535, 4104, 0x2FF21374, 0x2FF21370) = 0
8454278: 29950155: connext(5, 0x201FB688, 16) Err#55 EINPROGRESS
8454278: 29950155: ngetsockname(5, 0x2FF20BD0, 0x2FF210DC) = 0
8454278: 29950155: kwrite(4, " ", 1) = 1
[and then nothing]
++++++++++++++++++
Based on all this it seems that the connection get established, but then something weird is happening with the Poller thread. When I try to debug the run() function it gets stuck on if clause in line 724:
++++++++++++++++++
if (lastReturnedHandle) { // <== here
impl->resetMode(*lastReturnedHandle);
lastReturnedHandle = 0;
}
++++++++++++++++++
gdb session with trace log on stdout:
++++++++++++++++++
(gdb) break PosixPoller.cpp:724
No source file named PosixPoller.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (PosixPoller.cpp:724) pending.
(gdb) run
Starting program: /temporar/atk/qtmin
[New Thread 1]
entry
args
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Unspecified] debug Config file not read: /usr/local/etc/qpid/qpidc.conf
2013-12-11 18:52:23 [Client] debug Created connection 192.168.0.142:5672 with {} conn
2013-12-11 18:52:23 [Client] debug Starting connection, urls=[192.168.0.142:5672]
2013-12-11 18:52:23 [Client] info Trying to connect to 192.168.0.142:5672...
2013-12-11 18:52:23 [Client] debug Created IO thread: 0 [New Thread 258] [Switching to Thread 258]
Breakpoint 1, qpid::sys::Poller::wait( (qpid::sys::Duration)) (this=0x201b7e98, event=@0x201d8a88: ???, timeout=<incomplete type>)
at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:724
724 if (lastReturnedHandle) {
(gdb) bt
#0 qpid::sys::Poller::wait( (qpid::sys::Duration)) (this=0x201b7e98, event=@0x201d8a88: ???, timeout=<incomplete type>)
at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:724
#1 0xd6070dd4 in qpid::sys::Poller::run( (void)) (this=0x201b7e98) at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/PosixPoller.cpp:690
#2 0xd622fb44 in qpid::sys::Thread.cpp-0::runRunnable( (void *)) (p=0x201b7e98) at /temporar/qpid_build/qpidc-0.24/src/qpid/sys/posix/Thread.cpp:35
#3 0xd04fad88 in _pthread_body () from /usr/lib/libpthreads.a(shr_xpg5.o)
#4 0x00000000 in ?? ()
(gdb) step
2013-12-11 18:52:33 [Network] debug TCPConnector created for 0-10
2013-12-11 18:52:33 [Client] info Set TCP_NODELAY
2013-12-11 18:52:33 [System] info Connecting: 192.168.0.142:5672
[and then the endless wait]
++++++++++++++++++
If you managed to stay with me to this point, any ideas? :) Please note that I had to hack quite some things to make it build on AIX, though semantically there should be no change. Beside this I had to take ifaddrs.h and ifaddrs.cpp from somebody on samba.org list and link with it, since this is not included in AIX - I don't know if this should have any impact.
Btw, this is how I built the exe:
xlC_r -g -o qtmin ifaddrs.c qtest.cpp -I/temporar/qpid_build/qpidc-0.24/include -L/temporar/qpid_build/qpidc-0.24/build/src/ -lqpidcommon -lqpidmessaging -lpthreads -Wl,-brtl
I really appreciate any help with this.
Regards,
Ales