You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Nicolae Mihalache <xp...@gmail.com> on 2017/11/09 15:28:05 UTC
[Artemis] message flow with core bridges
Hello,
I have setup with a core bridge configuration between two artemis servers.
The latency between the servers is 12-13 ms (round trip).
I have a producer that produces about 150 messages per second and my
problem is that the bridge does not manage to keep up with the producer.
The sender address is configured for paging and the paging directory keeps
increasing in size.
When I run a tcpdump I see that after each tcp send there is a response
received so it looks to me that artemis sender waits for a confirmation
from the receiver before sending the next message.
I also see in the netstat that there is no tcp buffer on the sender, load
average is also very small so I'm very confident that it's not a network
bandwith or cpu load problem.
I tried all kind of configs:
in the connector configuration preAcknowledge=true
in the bridge configuration setting a large value for
confirmation-window-size
none of them seems to have any effect.
I'm using artemis core 1.5.4
Any idea?
thanks,
nicolae
Re: [Artemis] message flow with core bridges
Posted by Nicolae Mihalache <xp...@gmail.com>.
No, it was just because I disabled duplicate detection and that made it
switch to an synchronous delivery.
I didn't see that documented anywhere but I found it by debugging the
source code (long life open-source):
https://github.com/apache/activemq-artemis/blob/8886ec29247fdaf1500d5d216015c0cc9ea05f75/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/cluster/ClusterManager.java#L457
On Fri, Nov 10, 2017 at 1:15 PM, Martyn Taylor <mt...@redhat.com> wrote:
> Hi Nicolae,
>
> Is broker 2 blocking production to the target address? This would result
> in back pressure on the bridge, meaning that messages will build up in
> broker 1.
>
> Logs of the brokers and bridges (with trace enabled) would help diagnose.
>
> Cheers
> Martyn
>
>
>
>
> On Thu, Nov 9, 2017 at 4:49 PM, Nicolae Mihalache <xp...@gmail.com>
> wrote:
>
> > Some more info:
> >
> > I enabled logging and I see these messages in the logs indicating that
> the
> > sender is blockign after each message sent:
> > Nov 09 16:07:46.730 [30]
> > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
> > sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
> > channelID=10, packetObject=SessionSendMessage,
> > message=ServerMessage[messageID=30065833866,durable=
> > false,userID=null,priority=4,
> > bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
> > durable=false,
> > address=aces-sim.tm_dump,properties=TypedProperties[_
> AMQ_ORIG_MESSAGE_ID=
> > 30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=
> > aces-sim.tm_dump]]@2011757486,
> > requiresResponse=true]
> >
> > I tried to follow the sourcecode to see why it's blocking, without
> success
> > unfortunately. In
> > the org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:
> >
> > https://github.com/apache/activemq-artemis/blob/1.5.4/
> > artemis-server/src/main/java/org/apache/activemq/artemis/
> > core/server/cluster/impl/BridgeImpl.java#L864
> >
> >
> > the producer session (from the bridge to the remote server) seems to be
> > created with pre-acknowledgment so I don't understand anything :(
> >
> >
> > thanks for any hint
> > nicolae
> >
> >
> > On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <xp...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I have setup with a core bridge configuration between two artemis
> > servers.
> > > The latency between the servers is 12-13 ms (round trip).
> > > I have a producer that produces about 150 messages per second and my
> > > problem is that the bridge does not manage to keep up with the
> producer.
> > > The sender address is configured for paging and the paging directory
> > keeps
> > > increasing in size.
> > >
> > > When I run a tcpdump I see that after each tcp send there is a response
> > > received so it looks to me that artemis sender waits for a confirmation
> > > from the receiver before sending the next message.
> > >
> > > I also see in the netstat that there is no tcp buffer on the sender,
> load
> > > average is also very small so I'm very confident that it's not a
> network
> > > bandwith or cpu load problem.
> > >
> > > I tried all kind of configs:
> > > in the connector configuration preAcknowledge=true
> > > in the bridge configuration setting a large value for
> > > confirmation-window-size
> > >
> > > none of them seems to have any effect.
> > >
> > > I'm using artemis core 1.5.4
> > >
> > > Any idea?
> > >
> > >
> > > thanks,
> > > nicolae
> > >
> > >
> > >
> >
>
Re: [Artemis] message flow with core bridges
Posted by Nicolae Mihalache <xp...@gmail.com>.
No, it was just because I disabled duplicate detection and that made it
switch to an synchronous delivery.
I didn't see that documented anywhere but I found it by debugging the
source code (long life open-source):
https://github.com/apache/activemq-artemis/blob/
8886ec29247fdaf1500d5d216015c0cc9ea05f75/artemis-server/src/
main/java/org/apache/activemq/artemis/core/server/cluster/
ClusterManager.java#L457
On Fri, Nov 10, 2017 at 1:15 PM, Martyn Taylor <mt...@redhat.com> wrote:
> Hi Nicolae,
>
> Is broker 2 blocking production to the target address? This would result
> in back pressure on the bridge, meaning that messages will build up in
> broker 1.
>
> Logs of the brokers and bridges (with trace enabled) would help diagnose.
>
> Cheers
> Martyn
>
>
>
>
> On Thu, Nov 9, 2017 at 4:49 PM, Nicolae Mihalache <xp...@gmail.com>
> wrote:
>
> > Some more info:
> >
> > I enabled logging and I see these messages in the logs indicating that
> the
> > sender is blockign after each message sent:
> > Nov 09 16:07:46.730 [30]
> > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
> > sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
> > channelID=10, packetObject=SessionSendMessage,
> > message=ServerMessage[messageID=30065833866,durable=
> > false,userID=null,priority=4,
> > bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
> > durable=false,
> > address=aces-sim.tm_dump,properties=TypedProperties[_AMQ_
> ORIG_MESSAGE_ID=
> > 30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=
> > aces-sim.tm_dump]]@2011757486,
> > requiresResponse=true]
> >
> > I tried to follow the sourcecode to see why it's blocking, without
> success
> > unfortunately. In
> > the org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:
> >
> > https://github.com/apache/activemq-artemis/blob/1.5.4/
> > artemis-server/src/main/java/org/apache/activemq/artemis/
> > core/server/cluster/impl/BridgeImpl.java#L864
> >
> >
> > the producer session (from the bridge to the remote server) seems to be
> > created with pre-acknowledgment so I don't understand anything :(
> >
> >
> > thanks for any hint
> > nicolae
> >
> >
> > On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <xp...@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I have setup with a core bridge configuration between two artemis
> > servers.
> > > The latency between the servers is 12-13 ms (round trip).
> > > I have a producer that produces about 150 messages per second and my
> > > problem is that the bridge does not manage to keep up with the
> producer.
> > > The sender address is configured for paging and the paging directory
> > keeps
> > > increasing in size.
> > >
> > > When I run a tcpdump I see that after each tcp send there is a response
> > > received so it looks to me that artemis sender waits for a confirmation
> > > from the receiver before sending the next message.
> > >
> > > I also see in the netstat that there is no tcp buffer on the sender,
> load
> > > average is also very small so I'm very confident that it's not a
> network
> > > bandwith or cpu load problem.
> > >
> > > I tried all kind of configs:
> > > in the connector configuration preAcknowledge=true
> > > in the bridge configuration setting a large value for
> > > confirmation-window-size
> > >
> > > none of them seems to have any effect.
> > >
> > > I'm using artemis core 1.5.4
> > >
> > > Any idea?
> > >
> > >
> > > thanks,
> > > nicolae
> > >
> > >
> > >
> >
>
Re: [Artemis] message flow with core bridges
Posted by Martyn Taylor <mt...@redhat.com>.
Hi Nicolae,
Is broker 2 blocking production to the target address? This would result
in back pressure on the bridge, meaning that messages will build up in
broker 1.
Logs of the brokers and bridges (with trace enabled) would help diagnose.
Cheers
Martyn
On Thu, Nov 9, 2017 at 4:49 PM, Nicolae Mihalache <xp...@gmail.com>
wrote:
> Some more info:
>
> I enabled logging and I see these messages in the logs indicating that the
> sender is blockign after each message sent:
> Nov 09 16:07:46.730 [30]
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
> sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
> channelID=10, packetObject=SessionSendMessage,
> message=ServerMessage[messageID=30065833866,durable=
> false,userID=null,priority=4,
> bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
> durable=false,
> address=aces-sim.tm_dump,properties=TypedProperties[_AMQ_ORIG_MESSAGE_ID=
> 30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=
> aces-sim.tm_dump]]@2011757486,
> requiresResponse=true]
>
> I tried to follow the sourcecode to see why it's blocking, without success
> unfortunately. In
> the org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:
>
> https://github.com/apache/activemq-artemis/blob/1.5.4/
> artemis-server/src/main/java/org/apache/activemq/artemis/
> core/server/cluster/impl/BridgeImpl.java#L864
>
>
> the producer session (from the bridge to the remote server) seems to be
> created with pre-acknowledgment so I don't understand anything :(
>
>
> thanks for any hint
> nicolae
>
>
> On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <xp...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I have setup with a core bridge configuration between two artemis
> servers.
> > The latency between the servers is 12-13 ms (round trip).
> > I have a producer that produces about 150 messages per second and my
> > problem is that the bridge does not manage to keep up with the producer.
> > The sender address is configured for paging and the paging directory
> keeps
> > increasing in size.
> >
> > When I run a tcpdump I see that after each tcp send there is a response
> > received so it looks to me that artemis sender waits for a confirmation
> > from the receiver before sending the next message.
> >
> > I also see in the netstat that there is no tcp buffer on the sender, load
> > average is also very small so I'm very confident that it's not a network
> > bandwith or cpu load problem.
> >
> > I tried all kind of configs:
> > in the connector configuration preAcknowledge=true
> > in the bridge configuration setting a large value for
> > confirmation-window-size
> >
> > none of them seems to have any effect.
> >
> > I'm using artemis core 1.5.4
> >
> > Any idea?
> >
> >
> > thanks,
> > nicolae
> >
> >
> >
>
Re: [Artemis] message flow with core bridges
Posted by Nicolae Mihalache <xp...@gmail.com>.
Some more info:
I enabled logging and I see these messages in the logs indicating that the
sender is blockign after each message sent:
Nov 09 16:07:46.730 [30]
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
channelID=10, packetObject=SessionSendMessage,
message=ServerMessage[messageID=30065833866,durable=false,userID=null,priority=4,
bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
durable=false,
address=aces-sim.tm_dump,properties=TypedProperties[_AMQ_ORIG_MESSAGE_ID=30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=aces-sim.tm_dump]]@2011757486,
requiresResponse=true]
I tried to follow the sourcecode to see why it's blocking, without success
unfortunately. In
the org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:
https://github.com/apache/activemq-artemis/blob/1.5.4/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/cluster/impl/BridgeImpl.java#L864
the producer session (from the bridge to the remote server) seems to be
created with pre-acknowledgment so I don't understand anything :(
thanks for any hint
nicolae
On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <xp...@gmail.com>
wrote:
> Hello,
>
> I have setup with a core bridge configuration between two artemis servers.
> The latency between the servers is 12-13 ms (round trip).
> I have a producer that produces about 150 messages per second and my
> problem is that the bridge does not manage to keep up with the producer.
> The sender address is configured for paging and the paging directory keeps
> increasing in size.
>
> When I run a tcpdump I see that after each tcp send there is a response
> received so it looks to me that artemis sender waits for a confirmation
> from the receiver before sending the next message.
>
> I also see in the netstat that there is no tcp buffer on the sender, load
> average is also very small so I'm very confident that it's not a network
> bandwith or cpu load problem.
>
> I tried all kind of configs:
> in the connector configuration preAcknowledge=true
> in the bridge configuration setting a large value for
> confirmation-window-size
>
> none of them seems to have any effect.
>
> I'm using artemis core 1.5.4
>
> Any idea?
>
>
> thanks,
> nicolae
>
>
>