You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Bostjan Polanc (Jira)" <ji...@apache.org> on 2026/04/04 18:39:00 UTC
[jira] [Updated] (PROTON-2928) Certain network conditions cause proton container to hang during connect phase
[ https://issues.apache.org/jira/browse/PROTON-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bostjan Polanc updated PROTON-2928:
-----------------------------------
Description:
Running a proton container where route to host is not known, will randomly hang the container during connection attempt. Container does not stop, no exception is thrown and no callback called. During connection attempt CPU usage spikes to 100%.
Steps to reproduce:
1) Add a firewall rule so that access to server is blocked with ICMP error
iptables -A INPUT -p tcp --dport 5672 -j REJECT --reject-with icmp-host-unreachable
2) Modify the C++ helloworld sample so that it makes several attempts to run the container (while true...).
Sample cpp: [https://github.com/bospol624/qpid-proton/blob/main/cpp/examples/helloworld.cpp]
3) Run helloworld. Exception thrown while iterating should be "proton:io: No route to host - disconnected"
After random number of iterations, program stops. Note that while it is still iterating, CPU load while proton container is running is at 100%.
I attached a Dockerfile that sets up this test (clean build + helloworld sample). Firewall rule on host machine still needs to be applied.
My comment on the issue from users@qpid.apache.org:
Using a debugger I traced this to c/src/proactor/epool.c.
Inside the pconnection_process function there seems to be a race condition
on which is called first: recv or send.
If send is called first via write_flush (line 1298), it fails and
psocket_error function is called. psocket_error function will among other
things add PN_TRANSPORT_CLOSED event to collector.
This event batch will however never be returned to calling functions
(next_event_batch/process), because after write_flush call, event batch is
no longer returned.
Because there is no new trigger from FD and the event batch was not
returned, next_event_batch will just loop inside poller_do_epoll.
For my project I made a temporary patch (epool.c/pconnection_process:1298):
write_flush(pc);
*if (pconnection_has_event(pc)) {pc->output_drained = false;return
&pc->batch;}*
So basically just add an additional check, in case if write_flush generated
any events.
was:
Running a proton container where route to host is not known, will randomly hang the container during connection attempt. Container does not stop, no exception is thrown and no callback called. During connection attempt CPU usage spikes to 100%.
Steps to reproduce:
1) Add a firewall rule so that access to server is blocked with ICMP error
iptables -A INPUT -p tcp --dport 5672 -j REJECT --reject-with icmp-host-unreachable
2) Modify the C++ helloworld sample so that it makes several attempts to run the container (while true...).
Sample cpp: https://github.com/bospol624/qpid-proton/blob/main/cpp/examples/helloworld.cpp
3) Run helloworld. Exception thrown while iterating, should be "proton:io: No route to host - disconnected"
After random number of iterations, program stops. Note that while it is still iterating, CPU load while proton container is running is at 100%.
I attached a Dockerfile that sets up this test (clean build + helloworld sample). Firewall rule on host machine still needs to be applied.
My comment on the issue from users@qpid.apache.org:
Using a debugger I traced this to c/src/proactor/epool.c.
Inside the pconnection_process function there seems to be a race condition
on which is called first: recv or send.
If send is called first via write_flush (line 1298), it fails and
psocket_error function is called. psocket_error function will among other
things add PN_TRANSPORT_CLOSED event to collector.
This event batch will however never be returned to calling functions
(next_event_batch/process), because after write_flush call, event batch is
no longer returned.
Because there is no new trigger from FD and the event batch was not
returned, next_event_batch will just loop inside poller_do_epoll.
For my project I made a temporary patch (epool.c/pconnection_process:1298):
write_flush(pc);
*if (pconnection_has_event(pc)) \{pc->output_drained = false;return
&pc->batch;}*
So basically just add an additional check, in case if write_flush generated
any events.
> Certain network conditions cause proton container to hang during connect phase
> ------------------------------------------------------------------------------
>
> Key: PROTON-2928
> URL: https://issues.apache.org/jira/browse/PROTON-2928
> Project: Qpid Proton
> Issue Type: Bug
> Components: cpp-binding, proton-c
> Affects Versions: proton-c-0.40.0
> Environment: Ubuntu 22.04
> Reporter: Bostjan Polanc
> Priority: Critical
> Attachments: Dockerfile
>
>
> Running a proton container where route to host is not known, will randomly hang the container during connection attempt. Container does not stop, no exception is thrown and no callback called. During connection attempt CPU usage spikes to 100%.
>
> Steps to reproduce:
> 1) Add a firewall rule so that access to server is blocked with ICMP error
> iptables -A INPUT -p tcp --dport 5672 -j REJECT --reject-with icmp-host-unreachable
> 2) Modify the C++ helloworld sample so that it makes several attempts to run the container (while true...).
> Sample cpp: [https://github.com/bospol624/qpid-proton/blob/main/cpp/examples/helloworld.cpp]
> 3) Run helloworld. Exception thrown while iterating should be "proton:io: No route to host - disconnected"
> After random number of iterations, program stops. Note that while it is still iterating, CPU load while proton container is running is at 100%.
>
> I attached a Dockerfile that sets up this test (clean build + helloworld sample). Firewall rule on host machine still needs to be applied.
>
> My comment on the issue from users@qpid.apache.org:
>
> Using a debugger I traced this to c/src/proactor/epool.c.
> Inside the pconnection_process function there seems to be a race condition
> on which is called first: recv or send.
> If send is called first via write_flush (line 1298), it fails and
> psocket_error function is called. psocket_error function will among other
> things add PN_TRANSPORT_CLOSED event to collector.
> This event batch will however never be returned to calling functions
> (next_event_batch/process), because after write_flush call, event batch is
> no longer returned.
> Because there is no new trigger from FD and the event batch was not
> returned, next_event_batch will just loop inside poller_do_epoll.
> For my project I made a temporary patch (epool.c/pconnection_process:1298):
> write_flush(pc);
> *if (pconnection_has_event(pc)) {pc->output_drained = false;return
> &pc->batch;}*
> So basically just add an additional check, in case if write_flush generated
> any events.
>
>
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org