You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Ekta (Jira)" <ji...@apache.org> on 2021/09/23 19:15:00 UTC
[jira] [Updated] (DISPATCH-2254) Routes messages to broker with
large header
[ https://issues.apache.org/jira/browse/DISPATCH-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ekta updated DISPATCH-2254:
---------------------------
Description:
Hello,
Below is what our application and messaging architecture looks like.
We have a Microservice --> nlb --> qpid ---> amq brokers (Master/slave)
We recently saw a scenario where a java microservice application pushed a few messages to our amq brokers where the message was containing a larger header than the normal size header, which led to a very big disaster and caused our whole amq env to shutdown. To avoid such cases like these, is there a way to discard a message or handle this situation at the qpid layer so that it does not endup on our brokers as qpid for us is basically acting like a proxy and simply routes the traffic to brokers. Though the broker rejects this large header message but still stores it in its memory and goes in a infinite loop and keeps retrying the process the message which causes a over head on the brokers and immediately all the brokers go in shutdown state.
After filtering out the headers to a smaller size, we could see the issue is resolved. If there a way to handle this a little better on on amq incase it is not possible on qpid layer. Would appreciate any feedback if anyone have faced this issue before as it could again happen in future.
Caused by: javax.jms.JMSException: AMQ149005: Message of 564,590 bytes is bigger than the max record size of 501,760 bytes. You should try to move large application properties to the message body. [condition = failed]
2021-09-22 07:37:01,092 WARN [org.apache.activemq.artemis.core.server] AMQ222086: error handling packet ReplicationSyncFileMessage(LARGE_MESSAGE, id=*********) for replication: java.io.FileNotFoundException: /app/test/large-messages/*********.msg (Too many open files)
Thanks
was:
Hello,
Below is what our application and messaging architecture looks like.
We have a Microservice --> nlb --> qpid ---> amq brokers (Master/slave)
We recently saw a scenario where a java microservice application pushed a few messages to our amq brokers where the message was containing a larger header than the normal size header, which led to a very big disaster and caused our whole amq env to shutdown. To avoid such cases like these, is there a way to discard a message or handle this situation at the qpid layer so that it does not endup on our brokers as qpid for us is basically acting like a proxy and simply routes the traffic to brokers. Though the broker rejects this large header message but still stores it in its memory and goes in a infinite loop and keeps retrying the process the message which causes a over head on the brokers and immediately all the brokers go in shutdown state.
If there a way to handle this a little better on on amq incase it is not possible on qpid layer. Would appreciate any feedback if anyone have faced this issue before.
Thanks
> Routes messages to broker with large header
> -------------------------------------------
>
> Key: DISPATCH-2254
> URL: https://issues.apache.org/jira/browse/DISPATCH-2254
> Project: Qpid Dispatch
> Issue Type: Improvement
> Components: Routing Engine
> Affects Versions: 1.16.0
> Environment: PROD
> Reporter: Ekta
> Priority: Major
>
> Hello,
> Below is what our application and messaging architecture looks like.
> We have a Microservice --> nlb --> qpid ---> amq brokers (Master/slave)
> We recently saw a scenario where a java microservice application pushed a few messages to our amq brokers where the message was containing a larger header than the normal size header, which led to a very big disaster and caused our whole amq env to shutdown. To avoid such cases like these, is there a way to discard a message or handle this situation at the qpid layer so that it does not endup on our brokers as qpid for us is basically acting like a proxy and simply routes the traffic to brokers. Though the broker rejects this large header message but still stores it in its memory and goes in a infinite loop and keeps retrying the process the message which causes a over head on the brokers and immediately all the brokers go in shutdown state.
> After filtering out the headers to a smaller size, we could see the issue is resolved. If there a way to handle this a little better on on amq incase it is not possible on qpid layer. Would appreciate any feedback if anyone have faced this issue before as it could again happen in future.
> Caused by: javax.jms.JMSException: AMQ149005: Message of 564,590 bytes is bigger than the max record size of 501,760 bytes. You should try to move large application properties to the message body. [condition = failed]
> 2021-09-22 07:37:01,092 WARN [org.apache.activemq.artemis.core.server] AMQ222086: error handling packet ReplicationSyncFileMessage(LARGE_MESSAGE, id=*********) for replication: java.io.FileNotFoundException: /app/test/large-messages/*********.msg (Too many open files)
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org