[jira] [Commented] (KAFKA-14011) Reduce retention.ms from default 7 days to 1 day (Make it configurable) for the dead letter topic created for error handling via sink connector

["Jordan Moore (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-14011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17556337#comment-17556337 ] 

Jordan Moore commented on KAFKA-14011:
--------------------------------------

"topic.creation" configs are only available for source connectors, for topics that source connectors will write to, not dead letter topics.

If you are allowing the broker to auto-create topics with defaults, then it's recommended that you disable this, and create all necessary topics ahead of time, with the topic settings you need. 

For already existing topics, you can alter their configs. 

Both operations can be done with kafka-topics script 



> Reduce retention.ms from default 7 days to 1 day (Make it configurable) for the dead letter topic created for error handling via sink connector
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-14011
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14011
>             Project: Kafka
>          Issue Type: Improvement
>          Components: config, KafkaConnect
>    Affects Versions: 3.0.1
>            Reporter: Souvik
>            Priority: Major
>
> We are creating a sink connector along with the error handling mechanism , so if in case if there is bad record it is routed to error queue , The properties used while creating the sink connector are as following
>  
>  'errors.tolerance'='all',
>  'errors.deadletterqueue.topic.name' = 'error_dlq',
>  'errors.deadletterqueue.topic.replication.factor'= -1,
>  'errors.log.include.messages' = true,
>  'errors.log.enable' = true,
>  'errors.deadletterqueue.context.headers.enable' = true
> *now is there any way we can configure the retention.ms for the dead letter queue topic i.e from default 7 days to 1 day , or is there any way we can change the compaction policy by providing any property while creating the source connector*
> *we have tried the following properties in the connector*
> {code:java}
> 'topic.creation.default.retention.ms'='3000',
> 'admin.topic.creation.default.retention.ms'='3000',
> 'admin.retention.ms' = '3000',
> 'admin.topic.retention.ms' = '3000',
> 'admin.topic.creation.retention.ms' = '3000',
> 'error.topic.creation.default.retention.ms'='3000',
> 'error.deadletterqueue.topic.creation.default.retention.ms'='3000',
> 'error.deadletterqueue.topic.retention.ms'='3000',
> 'admin.topic.creation.default.retention.ms' = 3000, {code}
> but it did not work.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)