[jira] [Created] (NIFI-8085) Use poll(Duration) in ConsumeKafka_2_x processors

["Peter Turcsanyi (Jira)" <ji...@apache.org>]

Peter Turcsanyi created NIFI-8085:
-------------------------------------

             Summary: Use poll(Duration) in ConsumeKafka_2_x processors 
                 Key: NIFI-8085
                 URL: https://issues.apache.org/jira/browse/NIFI-8085
             Project: Apache NiFi
          Issue Type: Improvement
            Reporter: Peter Turcsanyi
            Assignee: Peter Turcsanyi


The currently used {{KafkaConsumer.poll(long timeout)}} method is deprecated in 2.x. The method waits for the metadata response indefinitely until we get it (so in case of an unavailable/stopped kafka cluster until the connection gets recovered). It means the {{poll(long)}} acquires and holds a thread exclusively for this period which can lead to starvation of other processors for timer driven threads.

Use poll(Duration timeout) instead which is not simply an overloaded version of the method but has different semantics (https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior).
If can return from metadata fetch after the timeout expires and does not occupy a thread continuously and makes the processor able to yield.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)