You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Mickael Maison (Jira)" <ji...@apache.org> on 2023/02/23 16:36:00 UTC
[jira] [Updated] (KAFKA-14746) Throwing in Connector.taskConfigs generates a lot of logs
[ https://issues.apache.org/jira/browse/KAFKA-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mickael Maison updated KAFKA-14746:
-----------------------------------
Description:
If a Connector throws in its taskConfigs() method, the runtime ends up retrying using DistributedHerder.RECONFIGURE_CONNECTOR_TASKS_BACKOFF_MS which is a fixed value (250ms). For each retry, the runtime prints the connector configuration and the enriched configuration so this can quickly generate a lot of logs.
There is some value in throwing in taskConfigs() as it allows to fail fast in case the connector is given bad credentials. For example this is what some of the Debezium connectors do: https://github.com/debezium/debezium/blob/main/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerConnector.java#L56-L69
The way Connectors are expected to work today is to instead always create tasks and let each task fail in case the configuration is wrong. We should document that and make it clear in the javadoc that throwing in taskConfigs is not recommended.
was:
If a Connector throws in its taskConfigs() method, the runtime ends up retrying using DistributedHerder.RECONFIGURE_CONNECTOR_TASKS_BACKOFF_MS which is a fixed value (250ms). For each retry, the runtime prints the connector configuration and the enriched configuration so this can quickly generate a lot of logs.
There is some value in throwing in taskConfigs() as it allows to fail fast in case the connector is given bad credentials for example.
The way Connectors are expected to work today is to instead always create tasks and let each task fail in case the configuration is wrong. We should document that and make it clear in the javadoc that throwing in taskConfigs is not recommended.
> Throwing in Connector.taskConfigs generates a lot of logs
> ---------------------------------------------------------
>
> Key: KAFKA-14746
> URL: https://issues.apache.org/jira/browse/KAFKA-14746
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Mickael Maison
> Priority: Major
>
> If a Connector throws in its taskConfigs() method, the runtime ends up retrying using DistributedHerder.RECONFIGURE_CONNECTOR_TASKS_BACKOFF_MS which is a fixed value (250ms). For each retry, the runtime prints the connector configuration and the enriched configuration so this can quickly generate a lot of logs.
> There is some value in throwing in taskConfigs() as it allows to fail fast in case the connector is given bad credentials. For example this is what some of the Debezium connectors do: https://github.com/debezium/debezium/blob/main/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerConnector.java#L56-L69
> The way Connectors are expected to work today is to instead always create tasks and let each task fail in case the configuration is wrong. We should document that and make it clear in the javadoc that throwing in taskConfigs is not recommended.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)