[GitHub] [nifi] goosalex commented on pull request #4410: NIFI-7642: KafkaConsumers - improve batching into FlowFiles under high latency conditions

[GitBox <gi...@apache.org>]

goosalex commented on pull request #4410:
URL: https://github.com/apache/nifi/pull/4410#issuecomment-670169916


   Thanks @pvillard31 for having a look at it. We were noticing too small Flowfiles compared to the "maxPollRecords"  when handling production Flows ( up to over 250'000msg/s ), on a secured Kafka cluster in a non trivial network layout. 
   The fixed 10ms timeout per single poll roundtrip led to undeterministic polling results, reporting "no more records" after  ~200 messages on average, never mind "maxPollRecords" and "maxWatingMilllis" beeing very high. 
   In a simple, unsecured Lab setting, we could not reproduce the issue.
   
   As I understand it, "maxWaitmillis" is the maximum time a Nifi-KafkaConsumer-Processor can use up to fill a FlowFile, whereas x in kafkaConsumer.poll(x) is the timeout for a single Kafka-transaction.
   
   Perhaps we should not put the user supplied-maWaitMillis directly here, but also a lower bound of the previously hard-coded 10ms ? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org