You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/08/05 18:14:00 UTC
[jira] [Commented] (NIFI-10199) PublishKafkaRecord should not flood the logs when a batch times out
[ https://issues.apache.org/jira/browse/NIFI-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17575995#comment-17575995 ]
ASF subversion and git services commented on NIFI-10199:
--------------------------------------------------------
Commit 387ae7d32318c1cd765c1794aff70f4b80a40b28 in nifi's branch refs/heads/main from Daniel Urban
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=387ae7d323 ]
NIFI-10199: PublishKafka InFlightMessageTracker should not log batch level errors on each record
Currently, when a batch level error occurs (e.g. delivery timeout), the same error is logged for each record of the batch, needlessly flooding the logs.
InFlightMessageTracker now only logs an exception when a flow file encounters it for the first time.
This closes #6185
Signed-off-by: Paul Grey <gr...@apache.org>
> PublishKafkaRecord should not flood the logs when a batch times out
> -------------------------------------------------------------------
>
> Key: NIFI-10199
> URL: https://issues.apache.org/jira/browse/NIFI-10199
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Daniel Urban
> Assignee: Daniel Urban
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, PublishKafkaRecord reports errors on the record level. This means that when an error occurs on the batch level, the same error is reported for each record of said batch.
> For example, in case of a delivery timeout, a transactional Kafka producer logs a message like this:
> {code:java}
> 2022-06-29 13:16:20,913 INFO org.apache.kafka.clients.producer.internals.TransactionManager: [Producer clientId=producer-XYZ, transactionalId=XYZ] Transiting to abortable error state due to org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for TOPIC-01:302002 ms has passed since batch creation {code}
> Then, following this, we get N lines, where N is the number of records in the batch. Also note that the TimeoutException is printed twice in this message.
> {code:java}
> 2022-06-29 13:16:20,913 ERROR org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_6: PublishKafkaRecord_2_6[id=91ba351d-96ad-1d0d-934c-cfd07e0bacd5] Failed to send StandardFlowFileRecord[uuid=349b4d38-451d-4e62-8ea9-636f7c666fc1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1656453186665-7724448, container=default, section=416], offset=0, length=132021252],offset=0,name=NAME,size=132021252] to Kafka: org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation
> org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation{code}
> Experienced this with PublishKafkaRecord_2_6, but other versions might be affected as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)