You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by "Sergio M." <si...@hotmail.com> on 2021/03/05 13:50:02 UTC

ConsumeJMS and PublishJMS performance

In our Nifi, we currently have some pipelines that connect to MQ queues. In these flows, we use the ConsumeJMS and PublishJMS processors.

When we execute several pipelines at the same time, we see that the level of resource consumption increases.

In the PublishJMS bibliography (https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms. processors.PublishJMS /), there is a note regarding its memory consumption:

"An instance of this component can cause high usage of this system resource. Multiple instances or high concurrency settings may result a degradation of performance."

In the future, we will have many more new pipelines and the consumption impact on the Nifi servers will be even greater.

Do you know any solution or configuration to help us deal with this?

Thanks!

RE: ConsumeJMS and PublishJMS performance

Posted by "Sergio M." <si...@hotmail.com>.

Hi Mark,

Thank you so much! It helped us understand how JMS works internally.

We found a solved improvement in Jira related to the optimization in the use of JMS sessions. Do you know if this change can improve Nifi's use of server resources?

https://issues.apache.org/jira/browse/NIFI-7563

Do you know how to scale Nifi with a lots of JMS connections?

Thanks!

Sergio
________________________________
De: Mark Payne <ma...@hotmail.com>
Enviado: viernes, 5 de marzo de 2021 16:47
Para: users@nifi.apache.org <us...@nifi.apache.org>
Asunto: Re: ConsumeJMS and PublishJMS performance

Sergio,

The reason these processors have this warning is because JMS requires that the contents of the message be fully loaded into memory. So for PublishJMS, each FlowFile is loaded into memory in order to push to the JMS broker. For ConsumeJMS, the message is received as an in-memory byte array before being written to the NiFi repository.

Each Thread/Concurrent Task, then, will hold the contents of one FlowFile in memory. So if you’re sending 1 KB FlowFiles with 10 Concurrent Tasks, that’s going to be 10 KB of content held in memory, roughly. That’s fine. But if you’ve got 500 PublishJMS processors, each with 10 Concurrent Tasks, and each trying to send/receive 1 MB messages, then you’re talking about 5,000 messages being operated on concurrently and each taking 1 MB of heap - for a total of 5 GB of your heap. Then you may be in trouble.

Does that help?

Thanks
-Mark

On Mar 5, 2021, at 8:50 AM, Sergio M. <si...@hotmail.com>> wrote:

In our Nifi, we currently have some pipelines that connect to MQ queues. In these flows, we use the ConsumeJMS and PublishJMS processors.

When we execute several pipelines at the same time, we see that the level of resource consumption increases.

In the PublishJMS bibliography (https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-jms-processors-nar/1.11.4/org.apache.nifi.jms. processors.PublishJMS /), there is a note regarding its memory consumption:

"An instance of this component can cause high usage of this system resource. Multiple instances or high concurrency settings may result a degradation of performance."

In the future, we will have many more new pipelines and the consumption impact on the Nifi servers will be even greater.

Do you know any solution or configuration to help us deal with this?

Thanks!

Re: ConsumeJMS and PublishJMS performance

Posted by Mark Payne <ma...@hotmail.com>.

Sergio,

Does that help?

Thanks
-Mark

On Mar 5, 2021, at 8:50 AM, Sergio M. <si...@hotmail.com>> wrote:

In our Nifi, we currently have some pipelines that connect to MQ queues. In these flows, we use the ConsumeJMS and PublishJMS processors.

When we execute several pipelines at the same time, we see that the level of resource consumption increases.

"An instance of this component can cause high usage of this system resource. Multiple instances or high concurrency settings may result a degradation of performance."

In the future, we will have many more new pipelines and the consumption impact on the Nifi servers will be even greater.

Do you know any solution or configuration to help us deal with this?

Thanks!