You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Tiaan Wessels <ti...@gmail.com> on 2023/03/07 17:36:45 UTC
on_connection_open never called
Hi,
I am trying a direct send between two programs, one listening and upon
accepting, starts a receiver and the other connecting and should start a
sender upon connecting but the on_connection_open never gets called in the
sending program. I can see with netstat the connection is established and
my listening program gets on_accept, on_transport_open and then
on_connection_open upon which a do open_receiver whereafter i get
on_session_open and then on_receiver_open upon which i add 100 credits but
that's where the action stops. The sender only ever gets on_container_start
upon which i connect and the source code would have opened a sender in the
on_connection_open but this never gets called even though the other end and
netstat shows the connection is in fact established. A pstack on the sender
shows the run thread in epoll and select most of the time.
I removed the rpm's and downloaded 0.38.0 sources and started adding debug
prints to understand what's going on. My first venture in qpid proton
sources and all i could see that is suspect is
void on_connection_bound(messaging_handler& handler, pn_event_t* event) {
connection c(make_wrapper(pn_event_connection(event)));
}
which makes a connection object but does nothing with it. Call's now
handler. This is the only case in proton::messaging_adapter::dispatch which
does not fall through to default in my sending app.
It's difficult to provide a working example as the code is integrated into
a bigger app which comes with a lot of libraries and other sources. My
on_* is modeled quite closely to the examples so i suspect something in my
integration code is breaking it but i have no clue what.
My question here is, is there any suggestions how to debug this with the
qpid proton source code at hand ? Why would my on_connection_open not be
called or is it not called ? Should it have been called ?
Thanks
Re: on_connection_open never called
Posted by Robbie Gemmell <ro...@gmail.com>.
The trace log suggests the listener side did not actually open the
connection, i.e it did not send the Open frame, and so nothing else
then actually happens from it because the connection is not open yet.
The below (and re-reading the original description) also suggests your
listening / 'server' is pre-emptively trying to create the receiving
side of a link, even though you said you already have a sending
'client' program explicitly trying [but not yet succeeding] to connect
and create a sender to send messages to the listening side. Whilst the
spec allowed to have both sides 'initiate' in an overlapping fashion,
it's also quite unusual vs having the server just handle/respond to
the attach of a sender link from the client.
As Andrew said, a reproducer would probably be helpful..and perhaps a
clearer idea of what it is you are trying to do. It actually reads a
lot like you want something closer to the 'simple_send + direct_recv'
example combination covered in the tutorial
https://qpid.apache.org/releases/qpid-proton-0.38.0/proton/cpp/api/tutorial_page.html.
On Wed, 8 Mar 2023 at 18:21, Tiaan Wessels <ti...@gmail.com> wrote:
>
> Hi Andrew,
> On my receiving end this is what I get (PN_LOG had no effect but
> PN_TRACE_FRM spew something)
>
> 253998 140157087659776 static void* CAAMQPAgent::run(void*)
> 253998 140157087659776 virtual void
> CAAMQPAgent::CAQPID::on_container_start(proton::container&): 0x7f78d0000b60
> 253998 140157087659776 listen on amqp://127.0.0.1:5672 ...
> [0x7f78d0004110]: <- AMQP
> [0x7f78d0004110]:0 <- @open(16)
> [container-id="bb53b030-f349-4476-8ecf-be0757438931", hostname="127.0.0.1",
> channel-max=32767]
> [0x7f78d0004110]: -> AMQP
> 253998 140157087659776 virtual void
> CAAMQPAgent::CAQPID::on_transport_open(proton::transport&)
> 253998 140157087659776 virtual void
> CAAMQPAgent::CAQPID::on_connection_open(proton::connection&)
> 253998 140157087659776 make receiver [example]
>
> (i've mixed the frame output from the lib with my own debug output which
> might be helpful)
>
> on the send side
>
> 254008 139716829640448 static void* CAAMQPAgent::run(void*)
> 254008 139716829640448 virtual void
> CAAMQPAgent::CAQPID::on_container_start(proton::container&): 0x7f1254000b60
> 254008 139716829640448 connect to amqp://127.0.0.1:5672 ...
> [0x7f1254003260]: -> AMQP
> [0x7f1254003260]:0 -> @open(16)
> [container-id="bb53b030-f349-4476-8ecf-be0757438931", hostname="127.0.0.1",
> channel-max=32767]
> [0x7f1254003260]: <- AMQP
>
> And that's where things stall
>
> Both processes have a thread waiting on epoll
>
> #0 0x00007f78e144746e in epoll_wait (epfd=8, events=0x7f78dd6c299c,
> maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
> #1 0x00007f78e0cdf888 in ?? () from
> /lib/x86_64-linux-gnu/libqpid-proton-proactor.so.1
> #2 0x00007f78e16b6a95 in proton::container::impl::thread() () from
> /lib/x86_64-linux-gnu/libqpid-proton-cpp.so.12
> #3 0x00007f78e16b6dae in proton::container::impl::run(int) () from
> /lib/x86_64-linux-gnu/libqpid-proton-cpp.so.12
> #4 0x000055e3c4ac5f48 in CAAMQPAgent::run (p=0x55e3c66b37f0) at
> amqpagent.cpp:907
> #5 0x00007f78e18d4609 in start_thread (arg=<optimised out>) at
> pthread_create.c:477
> #6 0x00007f78e1447133 in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
> I've verified that override gives no errors on compile
>
> Thanks
>
>
>
> On Wed, 8 Mar 2023 at 13:54, Andrew Stitcher <as...@redhat.com> wrote:
>
> > I'm afraid that it's going to be very difficult to diagnose the issue here
> > without a solid reproducer.
> >
> > One approach might be to turn frame level logging on at one end of the
> > connection or other using the PN_LOG environment variable. Set if to
> > "frame" (PN_LOG=frame) to get an output on stderr of the AMQP frames being
> > sent/received. If you post that here (or maybe link to pastebin/git gist)
> > then we can see a bit more clearly what's going on at a protocol level.
> >
> > Are you sure that you are correctly overloading the messaging_adapter
> > class? If you are using a different signature for on_connection_open then
> > the overload won't work. To get the compiler to alert you use the
> > 'override' keyword.
> >
> > Andrew
> >
> > On Tue, Mar 7, 2023 at 5:46 PM Tiaan Wessels <ti...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I am trying a direct send between two programs, one listening and upon
> > > accepting, starts a receiver and the other connecting and should start a
> > > sender upon connecting but the on_connection_open never gets called in
> > the
> > > sending program. I can see with netstat the connection is established and
> > > my listening program gets on_accept, on_transport_open and then
> > > on_connection_open upon which a do open_receiver whereafter i get
> > > on_session_open and then on_receiver_open upon which i add 100 credits
> > but
> > > that's where the action stops. The sender only ever gets
> > on_container_start
> > > upon which i connect and the source code would have opened a sender in
> > the
> > > on_connection_open but this never gets called even though the other end
> > and
> > > netstat shows the connection is in fact established. A pstack on the
> > sender
> > > shows the run thread in epoll and select most of the time.
> > > I removed the rpm's and downloaded 0.38.0 sources and started adding
> > debug
> > > prints to understand what's going on. My first venture in qpid proton
> > > sources and all i could see that is suspect is
> > >
> > > void on_connection_bound(messaging_handler& handler, pn_event_t* event) {
> > > connection c(make_wrapper(pn_event_connection(event)));
> > > }
> > >
> > > which makes a connection object but does nothing with it. Call's now
> > > handler. This is the only case in proton::messaging_adapter::dispatch
> > which
> > > does not fall through to default in my sending app.
> > > It's difficult to provide a working example as the code is integrated
> > into
> > > a bigger app which comes with a lot of libraries and other sources. My
> > > on_* is modeled quite closely to the examples so i suspect something in
> > my
> > > integration code is breaking it but i have no clue what.
> > > My question here is, is there any suggestions how to debug this with the
> > > qpid proton source code at hand ? Why would my on_connection_open not be
> > > called or is it not called ? Should it have been called ?
> > > Thanks
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: on_connection_open never called
Posted by Tiaan Wessels <ti...@gmail.com>.
Hi Andrew,
On my receiving end this is what I get (PN_LOG had no effect but
PN_TRACE_FRM spew something)
253998 140157087659776 static void* CAAMQPAgent::run(void*)
253998 140157087659776 virtual void
CAAMQPAgent::CAQPID::on_container_start(proton::container&): 0x7f78d0000b60
253998 140157087659776 listen on amqp://127.0.0.1:5672 ...
[0x7f78d0004110]: <- AMQP
[0x7f78d0004110]:0 <- @open(16)
[container-id="bb53b030-f349-4476-8ecf-be0757438931", hostname="127.0.0.1",
channel-max=32767]
[0x7f78d0004110]: -> AMQP
253998 140157087659776 virtual void
CAAMQPAgent::CAQPID::on_transport_open(proton::transport&)
253998 140157087659776 virtual void
CAAMQPAgent::CAQPID::on_connection_open(proton::connection&)
253998 140157087659776 make receiver [example]
(i've mixed the frame output from the lib with my own debug output which
might be helpful)
on the send side
254008 139716829640448 static void* CAAMQPAgent::run(void*)
254008 139716829640448 virtual void
CAAMQPAgent::CAQPID::on_container_start(proton::container&): 0x7f1254000b60
254008 139716829640448 connect to amqp://127.0.0.1:5672 ...
[0x7f1254003260]: -> AMQP
[0x7f1254003260]:0 -> @open(16)
[container-id="bb53b030-f349-4476-8ecf-be0757438931", hostname="127.0.0.1",
channel-max=32767]
[0x7f1254003260]: <- AMQP
And that's where things stall
Both processes have a thread waiting on epoll
#0 0x00007f78e144746e in epoll_wait (epfd=8, events=0x7f78dd6c299c,
maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007f78e0cdf888 in ?? () from
/lib/x86_64-linux-gnu/libqpid-proton-proactor.so.1
#2 0x00007f78e16b6a95 in proton::container::impl::thread() () from
/lib/x86_64-linux-gnu/libqpid-proton-cpp.so.12
#3 0x00007f78e16b6dae in proton::container::impl::run(int) () from
/lib/x86_64-linux-gnu/libqpid-proton-cpp.so.12
#4 0x000055e3c4ac5f48 in CAAMQPAgent::run (p=0x55e3c66b37f0) at
amqpagent.cpp:907
#5 0x00007f78e18d4609 in start_thread (arg=<optimised out>) at
pthread_create.c:477
#6 0x00007f78e1447133 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
I've verified that override gives no errors on compile
Thanks
On Wed, 8 Mar 2023 at 13:54, Andrew Stitcher <as...@redhat.com> wrote:
> I'm afraid that it's going to be very difficult to diagnose the issue here
> without a solid reproducer.
>
> One approach might be to turn frame level logging on at one end of the
> connection or other using the PN_LOG environment variable. Set if to
> "frame" (PN_LOG=frame) to get an output on stderr of the AMQP frames being
> sent/received. If you post that here (or maybe link to pastebin/git gist)
> then we can see a bit more clearly what's going on at a protocol level.
>
> Are you sure that you are correctly overloading the messaging_adapter
> class? If you are using a different signature for on_connection_open then
> the overload won't work. To get the compiler to alert you use the
> 'override' keyword.
>
> Andrew
>
> On Tue, Mar 7, 2023 at 5:46 PM Tiaan Wessels <ti...@gmail.com>
> wrote:
>
> > Hi,
> > I am trying a direct send between two programs, one listening and upon
> > accepting, starts a receiver and the other connecting and should start a
> > sender upon connecting but the on_connection_open never gets called in
> the
> > sending program. I can see with netstat the connection is established and
> > my listening program gets on_accept, on_transport_open and then
> > on_connection_open upon which a do open_receiver whereafter i get
> > on_session_open and then on_receiver_open upon which i add 100 credits
> but
> > that's where the action stops. The sender only ever gets
> on_container_start
> > upon which i connect and the source code would have opened a sender in
> the
> > on_connection_open but this never gets called even though the other end
> and
> > netstat shows the connection is in fact established. A pstack on the
> sender
> > shows the run thread in epoll and select most of the time.
> > I removed the rpm's and downloaded 0.38.0 sources and started adding
> debug
> > prints to understand what's going on. My first venture in qpid proton
> > sources and all i could see that is suspect is
> >
> > void on_connection_bound(messaging_handler& handler, pn_event_t* event) {
> > connection c(make_wrapper(pn_event_connection(event)));
> > }
> >
> > which makes a connection object but does nothing with it. Call's now
> > handler. This is the only case in proton::messaging_adapter::dispatch
> which
> > does not fall through to default in my sending app.
> > It's difficult to provide a working example as the code is integrated
> into
> > a bigger app which comes with a lot of libraries and other sources. My
> > on_* is modeled quite closely to the examples so i suspect something in
> my
> > integration code is breaking it but i have no clue what.
> > My question here is, is there any suggestions how to debug this with the
> > qpid proton source code at hand ? Why would my on_connection_open not be
> > called or is it not called ? Should it have been called ?
> > Thanks
> >
>
Re: on_connection_open never called
Posted by Andrew Stitcher <as...@redhat.com>.
I'm afraid that it's going to be very difficult to diagnose the issue here
without a solid reproducer.
One approach might be to turn frame level logging on at one end of the
connection or other using the PN_LOG environment variable. Set if to
"frame" (PN_LOG=frame) to get an output on stderr of the AMQP frames being
sent/received. If you post that here (or maybe link to pastebin/git gist)
then we can see a bit more clearly what's going on at a protocol level.
Are you sure that you are correctly overloading the messaging_adapter
class? If you are using a different signature for on_connection_open then
the overload won't work. To get the compiler to alert you use the
'override' keyword.
Andrew
On Tue, Mar 7, 2023 at 5:46 PM Tiaan Wessels <ti...@gmail.com> wrote:
> Hi,
> I am trying a direct send between two programs, one listening and upon
> accepting, starts a receiver and the other connecting and should start a
> sender upon connecting but the on_connection_open never gets called in the
> sending program. I can see with netstat the connection is established and
> my listening program gets on_accept, on_transport_open and then
> on_connection_open upon which a do open_receiver whereafter i get
> on_session_open and then on_receiver_open upon which i add 100 credits but
> that's where the action stops. The sender only ever gets on_container_start
> upon which i connect and the source code would have opened a sender in the
> on_connection_open but this never gets called even though the other end and
> netstat shows the connection is in fact established. A pstack on the sender
> shows the run thread in epoll and select most of the time.
> I removed the rpm's and downloaded 0.38.0 sources and started adding debug
> prints to understand what's going on. My first venture in qpid proton
> sources and all i could see that is suspect is
>
> void on_connection_bound(messaging_handler& handler, pn_event_t* event) {
> connection c(make_wrapper(pn_event_connection(event)));
> }
>
> which makes a connection object but does nothing with it. Call's now
> handler. This is the only case in proton::messaging_adapter::dispatch which
> does not fall through to default in my sending app.
> It's difficult to provide a working example as the code is integrated into
> a bigger app which comes with a lot of libraries and other sources. My
> on_* is modeled quite closely to the examples so i suspect something in my
> integration code is breaking it but i have no clue what.
> My question here is, is there any suggestions how to debug this with the
> qpid proton source code at hand ? Why would my on_connection_open not be
> called or is it not called ? Should it have been called ?
> Thanks
>