You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Chris Egerton (Jira)" <ji...@apache.org> on 2023/03/27 23:37:00 UTC
[jira] [Updated] (KAFKA-14843) Connector plugins config endpoint does not include Common configs
[ https://issues.apache.org/jira/browse/KAFKA-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Egerton updated KAFKA-14843:
----------------------------------
Description:
Connector plugins GET config endpoint introduced in [https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions] allows to get plugin configuration from the rest endpoint.
This configuration only includes the plugin configuration, but not the base configuration of the Sink/Source Connector.
For instance, when validating the configuration of a plugin, _all_ configs are returned:
```
curl -s $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config | jq -r '.[].name' | sort -u | wc -l
21
curl -s $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config/validate -XPUT -H 'Content-type: application/json' --data "\{\"connector.class\": \"io.aiven.kafka.connect.http.HttpSinkConnector\", \"topics\": \"example-topic-name\"}" | jq -r '.configs[].definition.name' | sort -u | wc -l
39
```
and the missing configs are all from base config:
```
diff validate.txt config.txt
6,14d5
< config.action.reload
< connector.class
< errors.deadletterqueue.context.headers.enable
< errors.deadletterqueue.topic.name
< errors.deadletterqueue.topic.replication.factor
< errors.log.enable
< errors.log.include.messages
< errors.retry.delay.max.ms
< errors.retry.timeout
16d6
< header.converter
24d13
< key.converter
26d14
< name
33d20
< predicates
35,39d21
< tasks.max
< topics
< topics.regex
< transforms
< value.converter
```
Would be great to get the base configs from the same endpoint as well, so we could rely on it instead of using the validate endpoint to get all configs.
was:
Connector plugins GET config endpoint introduced in [https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions|https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions,] allows to get plugin configuration from the rest endpoint.
This configuration only includes the plugin configuration, but not the base configuration of the Sink/Source Connector.
For instance, when validating the configuration of a plugin, _all_ configs are returned:
```
curl -s $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config | jq -r '.[].name' | sort -u | wc -l
21
curl -s $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config/validate -XPUT -H 'Content-type: application/json' --data "\{\"connector.class\": \"io.aiven.kafka.connect.http.HttpSinkConnector\", \"topics\": \"example-topic-name\"}" | jq -r '.configs[].definition.name' | sort -u | wc -l
39
```
and the missing configs are all from base config:
```
diff validate.txt config.txt
6,14d5
< config.action.reload
< connector.class
< errors.deadletterqueue.context.headers.enable
< errors.deadletterqueue.topic.name
< errors.deadletterqueue.topic.replication.factor
< errors.log.enable
< errors.log.include.messages
< errors.retry.delay.max.ms
< errors.retry.timeout
16d6
< header.converter
24d13
< key.converter
26d14
< name
33d20
< predicates
35,39d21
< tasks.max
< topics
< topics.regex
< transforms
< value.converter
```
Would be great to get the base configs from the same endpoint as well, so we could rely on it instead of using the validate endpoint to get all configs.
> Connector plugins config endpoint does not include Common configs
> -----------------------------------------------------------------
>
> Key: KAFKA-14843
> URL: https://issues.apache.org/jira/browse/KAFKA-14843
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 3.2.0
> Reporter: Jorge Esteban Quilcate Otoya
> Assignee: Jorge Esteban Quilcate Otoya
> Priority: Major
>
> Connector plugins GET config endpoint introduced in [https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions] allows to get plugin configuration from the rest endpoint.
> This configuration only includes the plugin configuration, but not the base configuration of the Sink/Source Connector.
> For instance, when validating the configuration of a plugin, _all_ configs are returned:
> ```
> curl -s $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config | jq -r '.[].name' | sort -u | wc -l
> 21
> curl -s $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config/validate -XPUT -H 'Content-type: application/json' --data "\{\"connector.class\": \"io.aiven.kafka.connect.http.HttpSinkConnector\", \"topics\": \"example-topic-name\"}" | jq -r '.configs[].definition.name' | sort -u | wc -l
> 39
> ```
> and the missing configs are all from base config:
> ```
> diff validate.txt config.txt
> 6,14d5
> < config.action.reload
> < connector.class
> < errors.deadletterqueue.context.headers.enable
> < errors.deadletterqueue.topic.name
> < errors.deadletterqueue.topic.replication.factor
> < errors.log.enable
> < errors.log.include.messages
> < errors.retry.delay.max.ms
> < errors.retry.timeout
> 16d6
> < header.converter
> 24d13
> < key.converter
> 26d14
> < name
> 33d20
> < predicates
> 35,39d21
> < tasks.max
> < topics
> < topics.regex
> < transforms
> < value.converter
> ```
> Would be great to get the base configs from the same endpoint as well, so we could rely on it instead of using the validate endpoint to get all configs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)