Regarding secrets or external configurations with other datatypes besides Strings

[Yeikel Santana <em...@yeikel.com>]

Hello all,

From what I've seen so far, with https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations, Kafka provides a mechanism to externalize secrets, which was later extended with https://cwiki.apache.org/confluence/display/KAFKA/KIP-887%3A+Add+ConfigProvider+to+make+use+of+environment+variables, and possibly others.

The challenge that I am facing at the moment is that ConfigData is configured as a Map<String, String>, whereas my values are of other types that are not Strings, such as floats and integers.

To give a more concrete example, some kafka-connectors allow me to configure values such as timeouts that I'd like to control per environment using externalization systems. Since these values are numeric, they fail to be read with parsing exceptions. 

Is the expectation that connectors handle this on their end, or what options do we have to externalize values so that they are read back using a different type other than Strings? 

Thank you!

Re: Regarding secrets or external configurations with other datatypes besides Strings

[Yash Mayya <ya...@gmail.com>]

Hi Yeikel,

Connector configurations are represented as String key-value pairs
throughout Kafka Connect, including during the creation of connectors
(regardless of whether or not any config providers are configured),
configuration updates etc. The Javadoc for the Connector class [1] and the
connector development guide [2] also demonstrate this constraint. Connector
plugin implementations should be handling any required parsing on their end
(ideally by leveraging the ConfigDef class [3]). Do let me know in case you
have any further questions!

Thanks,
Yash

[1] -
https://kafka.apache.org/35/javadoc/org/apache/kafka/connect/connector/Connector.html

[2] - https://kafka.apache.org/documentation/#connect_developing
[3] -
https://kafka.apache.org/35/javadoc/org/apache/kafka/common/config/ConfigDef.html


On Mon, Sep 18, 2023 at 7:20 PM Yeikel Santana <em...@yeikel.com> wrote:

> Hello all,
>
> From what I've seen so far, with
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations,
> Kafka provides a mechanism to externalize secrets, which was later extended
> with
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-887%3A+Add+ConfigProvider+to+make+use+of+environment+variables,
> and possibly others.
>
> The challenge that I am facing at the moment is that ConfigData is
> configured as a Map<String, String>, whereas my values are of other types
> that are not Strings, such as floats and integers.
>
> To give a more concrete example, some kafka-connectors allow me to
> configure values such as timeouts that I'd like to control per environment
> using externalization systems. Since these values are numeric, they fail to
> be read with parsing exceptions.
>
> Is the expectation that connectors handle this on their end, or what
> options do we have to externalize values so that they are read back using a
> different type other than Strings?
>
> Thank you!