[jira] [Created] (NIFI-3531) modify session.recover behaviour in nifi-jms-processors to cope with high-traffic JMS

["Dominik Benz (JIRA)" <ji...@apache.org>]

Dominik Benz created NIFI-3531:
----------------------------------

             Summary: modify session.recover behaviour in nifi-jms-processors to cope with high-traffic JMS
                 Key: NIFI-3531
                 URL: https://issues.apache.org/jira/browse/NIFI-3531
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
            Reporter: Dominik Benz


As described in this mailing list post

http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-td14940.html

the current implementation of nifi-jms-processor, especially of JMSConsumer

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSConsumer.java

causes frequent re-delivery of JMS messages due to the following behaviour:

1) several threads perform the JMS session callback in parallel
2) each callback performs a session.recover
3) during high traffic, the situation arises that the ACKs from another thread may not (yet) have arrived at the JMS server
4) this implies that the pointer of session.recover will reconsume the not-yet-acked message from another thread 

I understood Nifi prefers message duplication over message loss - however, in our case the number of re-delivered messages is "piling up" over time, leading to a growing number of un-acked messages in JMS and finally to storage problems on the JMS server side.

It would be great to modify the nifi-jms-processor package to reliably cope with higher-traffic JMS sources. Ideas from my side would be / include:

1) make synchronous / asynchronous delivery configurable
2) add configuration option for custom ACKing modes (i.e. non-standard JMS modes like individual ACKing or NO_ACK)
3) add configuration option for JMS message selectors

From other projects, I found the JMS communication options in this spark-jms-receiver package
  https://github.com/tbfenet/spark-jms-receiver
very helpful (in which situations are synchronous / asynchronous approaches reliable, how to buffer messages before acking them, how/when to ack, ...). Here's also a java port of a subset:
  https://github.com/bernhardschaefer/spark-jms-receiver-java

For any mentioned option I'm also happy to help!

Best & thanks for a great piece of software,
  Dominik



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)