You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Paul Grey (Jira)" <ji...@apache.org> on 2022/08/05 18:16:00 UTC
[jira] [Updated] (NIFI-10199) PublishKafkaRecord should not flood the logs when a batch times out
[ https://issues.apache.org/jira/browse/NIFI-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Grey updated NIFI-10199:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
https://github.com/apache/nifi/pull/6185
> PublishKafkaRecord should not flood the logs when a batch times out
> -------------------------------------------------------------------
>
> Key: NIFI-10199
> URL: https://issues.apache.org/jira/browse/NIFI-10199
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Daniel Urban
> Assignee: Daniel Urban
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, PublishKafkaRecord reports errors on the record level. This means that when an error occurs on the batch level, the same error is reported for each record of said batch.
> For example, in case of a delivery timeout, a transactional Kafka producer logs a message like this:
> {code:java}
> 2022-06-29 13:16:20,913 INFO org.apache.kafka.clients.producer.internals.TransactionManager: [Producer clientId=producer-XYZ, transactionalId=XYZ] Transiting to abortable error state due to org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for TOPIC-01:302002 ms has passed since batch creation {code}
> Then, following this, we get N lines, where N is the number of records in the batch. Also note that the TimeoutException is printed twice in this message.
> {code:java}
> 2022-06-29 13:16:20,913 ERROR org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_6: PublishKafkaRecord_2_6[id=91ba351d-96ad-1d0d-934c-cfd07e0bacd5] Failed to send StandardFlowFileRecord[uuid=349b4d38-451d-4e62-8ea9-636f7c666fc1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1656453186665-7724448, container=default, section=416], offset=0, length=132021252],offset=0,name=NAME,size=132021252] to Kafka: org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation
> org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation{code}
> Experienced this with PublishKafkaRecord_2_6, but other versions might be affected as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)