You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "lihaosky (via GitHub)" <gi...@apache.org> on 2026/05/01 17:59:27 UTC
[PR] [FLINK-39590] connection catalog manager changes [flink]
lihaosky opened a new pull request, #28085:
URL: https://github.com/apache/flink/pull/28085
## What is the purpose of the change
*(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)*
## Brief change log
*(for example:)*
- *The TaskInfo is stored in the blob store on job creation time as a persistent artifact*
- *Deployments RPC transmits only the blob storage reference*
- *TaskManagers retrieve the TaskInfo from the blob cache*
## Verifying this change
Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing).
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (100MB)*
- *Extended integration test for recovery after master (JobManager) failure*
- *Added test that validates that TaskInfo is transferred only once across recoveries*
- *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no)
- The serializers: (yes / no / don't know)
- The runtime per-record code paths (performance sensitive): (yes / no / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
- The S3 file system connector: (yes / no / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name and Version]
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "lihaosky (via GitHub)" <gi...@apache.org>.
lihaosky commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3244073166
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java:
##########
@@ -1791,6 +1836,328 @@ public ResolvedCatalogModel resolveCatalogModel(CatalogModel model) {
return ResolvedCatalogModel.of(model, resolvedInputSchema, resolvedOutputSchema);
}
+ // ------ connections ------
+
+ /**
+ * Get a connection from the catalog with the given object identifier.
+ *
+ * @param objectIdentifier The fully qualified path of the connection.
+ * @return The requested connection wrapped in Optional.
+ */
+ public Optional<CatalogConnection> getConnection(ObjectIdentifier objectIdentifier) {
+ CatalogConnection temporaryConnection = temporaryConnections.get(objectIdentifier);
+ if (temporaryConnection != null) {
+ return Optional.of(temporaryConnection);
+ }
+
+ Optional<Catalog> catalog = getCatalog(objectIdentifier.getCatalogName());
+ if (catalog.isPresent()) {
+ try {
+ return Optional.of(catalog.get().getConnection(objectIdentifier.toObjectPath()));
+ } catch (ConnectionNotExistException | UnsupportedOperationException e) {
+ // ConnectionNotExistException: connection does not exist in this catalog.
+ // UnsupportedOperationException: catalog does not support connections.
+ return Optional.empty();
+ }
+ } else {
+ return Optional.empty();
+ }
+ }
+
+ /**
+ * List all connections in the given catalog and database.
+ *
+ * @param catalogName The name of the catalog.
+ * @param databaseName The name of the database.
+ * @return A set of connection names.
+ */
+ public Set<String> listConnections(String catalogName, String databaseName) {
+ Catalog catalog = getCatalogOrError(catalogName);
+ try {
+ Set<String> connections = new HashSet<>(catalog.listConnections(databaseName));
+
+ // Add temporary connections for this catalog and database
+ temporaryConnections.keySet().stream()
+ .filter(
+ identifier ->
+ identifier.getCatalogName().equals(catalogName)
+ && identifier.getDatabaseName().equals(databaseName))
+ .map(ObjectIdentifier::getObjectName)
+ .forEach(connections::add);
+
+ return connections;
+ } catch (DatabaseNotExistException e) {
+ throw new ValidationException(
+ String.format(
+ "Database %s does not exist in catalog %s.", databaseName, catalogName),
+ e);
+ } catch (CatalogException e) {
+ throw new TableException(
+ String.format(
+ "Failed to list connections in catalog %s and database %s.",
+ catalogName, databaseName),
+ e);
+ }
+ }
+
+ /**
+ * Create a permanent connection in the given fully qualified path.
+ *
+ * <p>If a {@link ConnectionFactory} and {@link WritableSecretStore} are configured, sensitive
+ * fields are extracted from the connection and stored in the secret store before persisting the
+ * non-sensitive {@link CatalogConnection} to the catalog.
+ *
+ * @param connection The connection with all options including sensitive fields.
+ * @param objectIdentifier The fully qualified path where to create the connection.
+ * @param ignoreIfExists If false exception will be thrown if the connection already exists.
+ */
+ public void createConnection(
+ SensitiveConnection connection,
+ ObjectIdentifier objectIdentifier,
+ boolean ignoreIfExists) {
+ if (connectionFactory == null || writableSecretStore == null) {
+ throw new ValidationException(
+ "ConnectionFactory and WritableSecretStore must be configured to create connections.");
+ }
+ if (getConnection(objectIdentifier).isPresent()) {
+ if (ignoreIfExists) {
+ return;
+ }
+ throw new ValidationException(
+ String.format(
+ "Connection with identifier '%s' already exists.",
+ objectIdentifier.asSummaryString()));
+ }
+ final CatalogConnection catalogConnection =
+ connectionFactory.createConnection(connection, writableSecretStore);
+ boolean persisted = false;
+ try {
+ execute(
+ (catalog, path) -> {
+ catalog.createConnection(path, catalogConnection, ignoreIfExists);
+ catalogModificationListeners.forEach(
+ listener ->
+ listener.onEvent(
+ CreateConnectionEvent.createEvent(
+ CatalogContext.createContext(
+ objectIdentifier.getCatalogName(),
+ catalog),
+ objectIdentifier,
+ catalogConnection,
+ ignoreIfExists,
+ false)));
+ },
+ objectIdentifier,
+ ignoreIfExists,
+ "CreateConnection");
+ persisted = true;
+ } finally {
+ if (!persisted) {
+ tryDeleteSecrets(
+ catalogConnection,
+ writableSecretStore,
+ "rollback createConnection " + objectIdentifier);
+ }
+ }
+ }
+
+ /**
+ * Create a temporary connection in the given fully qualified path.
+ *
+ * @param connection The connection with all options including sensitive fields.
+ * @param objectIdentifier The fully qualified path where to create the connection.
+ * @param ignoreIfExists If false exception will be thrown if the connection already exists.
+ */
+ public void createTemporaryConnection(
+ SensitiveConnection connection,
+ ObjectIdentifier objectIdentifier,
+ boolean ignoreIfExists) {
+ if (connectionFactory == null) {
Review Comment:
I actually fixed it in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3282531053
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/ConnectionFactory.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+
+/**
+ * Factory for creating and resolving connections, handling the encryption and decryption of
+ * sensitive connection fields.
+ *
+ * <p>A {@code ConnectionFactory} is responsible for:
+ *
+ * <ul>
+ * <li>Extracting sensitive fields from a {@link SensitiveConnection} and storing them in a {@link
+ * WritableSecretStore}, returning a {@link CatalogConnection} that is safe to persist in a
+ * catalog.
+ * <li>Resolving a {@link CatalogConnection} from a catalog by retrieving its secrets from a
+ * {@link ReadableSecretStore} and returning a complete {@link SensitiveConnection}.
+ * </ul>
+ *
+ * @see org.apache.flink.table.factories.DefaultConnectionFactory
+ */
+@PublicEvolving
+public interface ConnectionFactory extends Factory {
+
+ /**
+ * The option key that identifies the connection type in a {@link SensitiveConnection}'s or
+ * {@link CatalogConnection}'s options. The value of this option is matched against the {@link
+ * #factoryIdentifier()} of registered {@code ConnectionFactory} implementations to discover the
+ * factory that handles a given connection (see FLIP-529).
+ */
+ String CONNECTION_TYPE_KEY = "type";
Review Comment:
What I meant was just a nicer declaration of this variable. Similar how we do it in `org.apache.flink.table.factories.FactoryUtil#PROVIDER`. An option with a type `string` a default value and description. The behavior of this PR would not change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr merged PR #28085:
URL: https://github.com/apache/flink/pull/28085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3281928185
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
Review Comment:
keep docs short and avoid duplicate code, have one source of truth (the field below)
##########
flink-table/flink-table-common/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory:
##########
@@ -14,3 +14,4 @@
# limitations under the License.
org.apache.flink.table.module.CoreModuleFactory
+org.apache.flink.table.factories.DefaultConnectionFactory
Review Comment:
```suggestion
org.apache.flink.table.catalog.DefaultConnectionFactory
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/ConnectionFactory.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+
+/**
+ * Factory for creating and resolving connections, handling the encryption and decryption of
+ * sensitive connection fields.
+ *
+ * <p>A {@code ConnectionFactory} is responsible for:
+ *
+ * <ul>
+ * <li>Extracting sensitive fields from a {@link SensitiveConnection} and storing them in a {@link
+ * WritableSecretStore}, returning a {@link CatalogConnection} that is safe to persist in a
+ * catalog.
+ * <li>Resolving a {@link CatalogConnection} from a catalog by retrieving its secrets from a
+ * {@link ReadableSecretStore} and returning a complete {@link SensitiveConnection}.
+ * </ul>
+ *
+ * @see org.apache.flink.table.factories.DefaultConnectionFactory
+ */
+@PublicEvolving
+public interface ConnectionFactory extends Factory {
+
+ /**
+ * The option key that identifies the connection type in a {@link SensitiveConnection}'s or
+ * {@link CatalogConnection}'s options. The value of this option is matched against the {@link
+ * #factoryIdentifier()} of registered {@code ConnectionFactory} implementations to discover the
+ * factory that handles a given connection (see FLIP-529).
+ */
+ String CONNECTION_TYPE_KEY = "type";
Review Comment:
We could make this a ConfigOption instead with a `DefaultConnectionFactory.IDENTIFIER` as default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3281703070
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
+
+ private static final Set<String> SENSITIVE_FIELD_NAMES =
+ Collections.unmodifiableSet(
+ new HashSet<>(
+ Arrays.asList(
+ "password",
Review Comment:
A comment would definitely help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3227874405
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
+
+ private static final Set<String> SENSITIVE_FIELD_NAMES =
+ Collections.unmodifiableSet(
+ new HashSet<>(
+ Arrays.asList(
+ "password",
Review Comment:
Could you explain how this list was created?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/exceptions/ConnectionAlreadyExistException.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog.exceptions;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.catalog.ObjectPath;
+
+/** Exception for trying to create a connection that already exists. */
+@PublicEvolving
+public class ConnectionAlreadyExistException extends Exception {
+
+ private static final String MSG = "Connection %s already exists in Catalog %s.";
Review Comment:
can we wrap all template parameters into single quotes, this helps downstream projects to clean sensitive content
```suggestion
private static final String MSG = "Connection '%s' already exists in catalog '%s'.";
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
+
+ private static final Set<String> SENSITIVE_FIELD_NAMES =
+ Collections.unmodifiableSet(
+ new HashSet<>(
+ Arrays.asList(
+ "password",
+ "secret",
+ "fs.azure.account.key",
+ "apikey",
+ "api-key",
+ "auth-params",
+ "service-key",
+ "token",
+ "basic-auth",
+ "jaas.config",
+ "http-headers")));
+
+ @Override
+ public String factoryIdentifier() {
+ return "default";
+ }
+
+ @Override
+ public Set<ConfigOption<?>> requiredOptions() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Set<ConfigOption<?>> optionalOptions() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public CatalogConnection createConnection(
+ SensitiveConnection connection, WritableSecretStore secretStore) {
+ Map<String, String> allOptions = connection.getOptions();
+
+ if (allOptions.containsKey(SECRET_REFERENCE_KEY)) {
+ throw new ValidationException(
+ String.format(
+ "Connection option '%s' is reserved and cannot be set by users.",
+ SECRET_REFERENCE_KEY));
+ }
+
+ Map<String, String> sensitiveOptions =
+ allOptions.entrySet().stream()
+ .filter(e -> SENSITIVE_FIELD_NAMES.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ Map<String, String> nonSensitiveOptions =
+ allOptions.entrySet().stream()
+ .filter(e -> !SENSITIVE_FIELD_NAMES.contains(e.getKey()))
+ .collect(
+ Collectors.toMap(
+ Map.Entry::getKey,
+ Map.Entry::getValue,
+ (a, b) -> a,
+ HashMap::new));
+
+ if (!sensitiveOptions.isEmpty()) {
+ final String secretId;
+ try {
+ secretId = secretStore.storeSecret(sensitiveOptions);
+ } catch (SecretException e) {
+ throw e;
+ } catch (RuntimeException e) {
+ throw new SecretException("Failed to store connection secret.", e);
+ }
+ nonSensitiveOptions.put(SECRET_REFERENCE_KEY, secretId);
Review Comment:
we should add a test that also verifies how secretStore is called.
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
Review Comment:
nit: because in Flink options are usually separated by `-`. But I'm fine with the surrounding `__`
```suggestion
public static final String SECRET_REFERENCE_KEY = "__flink.encrypted-secret-key__";
```
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
+
+ private static final Set<String> SENSITIVE_FIELD_NAMES =
+ Collections.unmodifiableSet(
+ new HashSet<>(
+ Arrays.asList(
+ "password",
+ "secret",
+ "fs.azure.account.key",
+ "apikey",
+ "api-key",
+ "auth-params",
+ "service-key",
+ "token",
+ "basic-auth",
+ "jaas.config",
+ "http-headers")));
+
+ @Override
+ public String factoryIdentifier() {
+ return "default";
+ }
+
+ @Override
+ public Set<ConfigOption<?>> requiredOptions() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Set<ConfigOption<?>> optionalOptions() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public CatalogConnection createConnection(
+ SensitiveConnection connection, WritableSecretStore secretStore) {
+ Map<String, String> allOptions = connection.getOptions();
+
+ if (allOptions.containsKey(SECRET_REFERENCE_KEY)) {
+ throw new ValidationException(
+ String.format(
+ "Connection option '%s' is reserved and cannot be set by users.",
+ SECRET_REFERENCE_KEY));
+ }
+
+ Map<String, String> sensitiveOptions =
+ allOptions.entrySet().stream()
+ .filter(e -> SENSITIVE_FIELD_NAMES.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ Map<String, String> nonSensitiveOptions =
+ allOptions.entrySet().stream()
+ .filter(e -> !SENSITIVE_FIELD_NAMES.contains(e.getKey()))
+ .collect(
+ Collectors.toMap(
+ Map.Entry::getKey,
+ Map.Entry::getValue,
+ (a, b) -> a,
+ HashMap::new));
+
+ if (!sensitiveOptions.isEmpty()) {
+ final String secretId;
+ try {
+ secretId = secretStore.storeSecret(sensitiveOptions);
+ } catch (SecretException e) {
+ throw e;
+ } catch (RuntimeException e) {
+ throw new SecretException("Failed to store connection secret.", e);
+ }
+ nonSensitiveOptions.put(SECRET_REFERENCE_KEY, secretId);
Review Comment:
is there any test that verifies the behavior of this class? I don't see any references of SECRET_REFERENCE_KEY out side of this class
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java:
##########
@@ -1791,6 +1836,328 @@ public ResolvedCatalogModel resolveCatalogModel(CatalogModel model) {
return ResolvedCatalogModel.of(model, resolvedInputSchema, resolvedOutputSchema);
}
+ // ------ connections ------
+
+ /**
+ * Get a connection from the catalog with the given object identifier.
+ *
+ * @param objectIdentifier The fully qualified path of the connection.
+ * @return The requested connection wrapped in Optional.
+ */
+ public Optional<CatalogConnection> getConnection(ObjectIdentifier objectIdentifier) {
+ CatalogConnection temporaryConnection = temporaryConnections.get(objectIdentifier);
+ if (temporaryConnection != null) {
+ return Optional.of(temporaryConnection);
+ }
+
+ Optional<Catalog> catalog = getCatalog(objectIdentifier.getCatalogName());
+ if (catalog.isPresent()) {
+ try {
+ return Optional.of(catalog.get().getConnection(objectIdentifier.toObjectPath()));
+ } catch (ConnectionNotExistException | UnsupportedOperationException e) {
+ // ConnectionNotExistException: connection does not exist in this catalog.
+ // UnsupportedOperationException: catalog does not support connections.
+ return Optional.empty();
+ }
+ } else {
+ return Optional.empty();
+ }
+ }
+
+ /**
+ * List all connections in the given catalog and database.
+ *
+ * @param catalogName The name of the catalog.
+ * @param databaseName The name of the database.
+ * @return A set of connection names.
+ */
+ public Set<String> listConnections(String catalogName, String databaseName) {
+ Catalog catalog = getCatalogOrError(catalogName);
+ try {
+ Set<String> connections = new HashSet<>(catalog.listConnections(databaseName));
+
+ // Add temporary connections for this catalog and database
+ temporaryConnections.keySet().stream()
+ .filter(
+ identifier ->
+ identifier.getCatalogName().equals(catalogName)
+ && identifier.getDatabaseName().equals(databaseName))
+ .map(ObjectIdentifier::getObjectName)
+ .forEach(connections::add);
+
+ return connections;
+ } catch (DatabaseNotExistException e) {
+ throw new ValidationException(
+ String.format(
+ "Database %s does not exist in catalog %s.", databaseName, catalogName),
+ e);
+ } catch (CatalogException e) {
+ throw new TableException(
+ String.format(
+ "Failed to list connections in catalog %s and database %s.",
+ catalogName, databaseName),
+ e);
+ }
+ }
+
+ /**
+ * Create a permanent connection in the given fully qualified path.
+ *
+ * <p>If a {@link ConnectionFactory} and {@link WritableSecretStore} are configured, sensitive
+ * fields are extracted from the connection and stored in the secret store before persisting the
+ * non-sensitive {@link CatalogConnection} to the catalog.
+ *
+ * @param connection The connection with all options including sensitive fields.
+ * @param objectIdentifier The fully qualified path where to create the connection.
+ * @param ignoreIfExists If false exception will be thrown if the connection already exists.
+ */
+ public void createConnection(
+ SensitiveConnection connection,
+ ObjectIdentifier objectIdentifier,
+ boolean ignoreIfExists) {
+ if (connectionFactory == null || writableSecretStore == null) {
+ throw new ValidationException(
+ "ConnectionFactory and WritableSecretStore must be configured to create connections.");
+ }
+ if (getConnection(objectIdentifier).isPresent()) {
+ if (ignoreIfExists) {
+ return;
+ }
+ throw new ValidationException(
+ String.format(
+ "Connection with identifier '%s' already exists.",
+ objectIdentifier.asSummaryString()));
+ }
+ final CatalogConnection catalogConnection =
+ connectionFactory.createConnection(connection, writableSecretStore);
+ boolean persisted = false;
+ try {
+ execute(
+ (catalog, path) -> {
+ catalog.createConnection(path, catalogConnection, ignoreIfExists);
+ catalogModificationListeners.forEach(
+ listener ->
+ listener.onEvent(
+ CreateConnectionEvent.createEvent(
+ CatalogContext.createContext(
+ objectIdentifier.getCatalogName(),
+ catalog),
+ objectIdentifier,
+ catalogConnection,
+ ignoreIfExists,
+ false)));
+ },
+ objectIdentifier,
+ ignoreIfExists,
+ "CreateConnection");
+ persisted = true;
+ } finally {
+ if (!persisted) {
+ tryDeleteSecrets(
+ catalogConnection,
+ writableSecretStore,
+ "rollback createConnection " + objectIdentifier);
+ }
+ }
+ }
+
+ /**
+ * Create a temporary connection in the given fully qualified path.
+ *
+ * @param connection The connection with all options including sensitive fields.
+ * @param objectIdentifier The fully qualified path where to create the connection.
+ * @param ignoreIfExists If false exception will be thrown if the connection already exists.
+ */
+ public void createTemporaryConnection(
+ SensitiveConnection connection,
+ ObjectIdentifier objectIdentifier,
+ boolean ignoreIfExists) {
+ if (connectionFactory == null) {
Review Comment:
I thought we use Java Service providers and discover a connection factory. If no factory can be found, we can fall back to the default one configured in EnvironmentSettings and from there passed to CatalogManager. Wdyt?
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/exceptions/ConnectionAlreadyExistException.java:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.catalog.exceptions;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.catalog.ObjectPath;
+
+/** Exception for trying to create a connection that already exists. */
+@PublicEvolving
+public class ConnectionAlreadyExistException extends Exception {
+
+ private static final String MSG = "Connection %s already exists in Catalog %s.";
Review Comment:
please revisit other messages in this PR
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
+
+ private static final Set<String> SENSITIVE_FIELD_NAMES =
+ Collections.unmodifiableSet(
+ new HashSet<>(
+ Arrays.asList(
+ "password",
Review Comment:
Shouldn't we start smaller and let this list evolve?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "lihaosky (via GitHub)" <gi...@apache.org>.
lihaosky commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3236339101
##########
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java:
##########
@@ -1791,6 +1836,328 @@ public ResolvedCatalogModel resolveCatalogModel(CatalogModel model) {
return ResolvedCatalogModel.of(model, resolvedInputSchema, resolvedOutputSchema);
}
+ // ------ connections ------
+
+ /**
+ * Get a connection from the catalog with the given object identifier.
+ *
+ * @param objectIdentifier The fully qualified path of the connection.
+ * @return The requested connection wrapped in Optional.
+ */
+ public Optional<CatalogConnection> getConnection(ObjectIdentifier objectIdentifier) {
+ CatalogConnection temporaryConnection = temporaryConnections.get(objectIdentifier);
+ if (temporaryConnection != null) {
+ return Optional.of(temporaryConnection);
+ }
+
+ Optional<Catalog> catalog = getCatalog(objectIdentifier.getCatalogName());
+ if (catalog.isPresent()) {
+ try {
+ return Optional.of(catalog.get().getConnection(objectIdentifier.toObjectPath()));
+ } catch (ConnectionNotExistException | UnsupportedOperationException e) {
+ // ConnectionNotExistException: connection does not exist in this catalog.
+ // UnsupportedOperationException: catalog does not support connections.
+ return Optional.empty();
+ }
+ } else {
+ return Optional.empty();
+ }
+ }
+
+ /**
+ * List all connections in the given catalog and database.
+ *
+ * @param catalogName The name of the catalog.
+ * @param databaseName The name of the database.
+ * @return A set of connection names.
+ */
+ public Set<String> listConnections(String catalogName, String databaseName) {
+ Catalog catalog = getCatalogOrError(catalogName);
+ try {
+ Set<String> connections = new HashSet<>(catalog.listConnections(databaseName));
+
+ // Add temporary connections for this catalog and database
+ temporaryConnections.keySet().stream()
+ .filter(
+ identifier ->
+ identifier.getCatalogName().equals(catalogName)
+ && identifier.getDatabaseName().equals(databaseName))
+ .map(ObjectIdentifier::getObjectName)
+ .forEach(connections::add);
+
+ return connections;
+ } catch (DatabaseNotExistException e) {
+ throw new ValidationException(
+ String.format(
+ "Database %s does not exist in catalog %s.", databaseName, catalogName),
+ e);
+ } catch (CatalogException e) {
+ throw new TableException(
+ String.format(
+ "Failed to list connections in catalog %s and database %s.",
+ catalogName, databaseName),
+ e);
+ }
+ }
+
+ /**
+ * Create a permanent connection in the given fully qualified path.
+ *
+ * <p>If a {@link ConnectionFactory} and {@link WritableSecretStore} are configured, sensitive
+ * fields are extracted from the connection and stored in the secret store before persisting the
+ * non-sensitive {@link CatalogConnection} to the catalog.
+ *
+ * @param connection The connection with all options including sensitive fields.
+ * @param objectIdentifier The fully qualified path where to create the connection.
+ * @param ignoreIfExists If false exception will be thrown if the connection already exists.
+ */
+ public void createConnection(
+ SensitiveConnection connection,
+ ObjectIdentifier objectIdentifier,
+ boolean ignoreIfExists) {
+ if (connectionFactory == null || writableSecretStore == null) {
+ throw new ValidationException(
+ "ConnectionFactory and WritableSecretStore must be configured to create connections.");
+ }
+ if (getConnection(objectIdentifier).isPresent()) {
+ if (ignoreIfExists) {
+ return;
+ }
+ throw new ValidationException(
+ String.format(
+ "Connection with identifier '%s' already exists.",
+ objectIdentifier.asSummaryString()));
+ }
+ final CatalogConnection catalogConnection =
+ connectionFactory.createConnection(connection, writableSecretStore);
+ boolean persisted = false;
+ try {
+ execute(
+ (catalog, path) -> {
+ catalog.createConnection(path, catalogConnection, ignoreIfExists);
+ catalogModificationListeners.forEach(
+ listener ->
+ listener.onEvent(
+ CreateConnectionEvent.createEvent(
+ CatalogContext.createContext(
+ objectIdentifier.getCatalogName(),
+ catalog),
+ objectIdentifier,
+ catalogConnection,
+ ignoreIfExists,
+ false)));
+ },
+ objectIdentifier,
+ ignoreIfExists,
+ "CreateConnection");
+ persisted = true;
+ } finally {
+ if (!persisted) {
+ tryDeleteSecrets(
+ catalogConnection,
+ writableSecretStore,
+ "rollback createConnection " + objectIdentifier);
+ }
+ }
+ }
+
+ /**
+ * Create a temporary connection in the given fully qualified path.
+ *
+ * @param connection The connection with all options including sensitive fields.
+ * @param objectIdentifier The fully qualified path where to create the connection.
+ * @param ignoreIfExists If false exception will be thrown if the connection already exists.
+ */
+ public void createTemporaryConnection(
+ SensitiveConnection connection,
+ ObjectIdentifier objectIdentifier,
+ boolean ignoreIfExists) {
+ if (connectionFactory == null) {
Review Comment:
You are right. ConnectionFactory should be discovered by SPI. Can I fix it in a separate PR? This PR is already large. Created https://issues.apache.org/jira/browse/FLINK-39678 to track
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
+
+ private static final Set<String> SENSITIVE_FIELD_NAMES =
+ Collections.unmodifiableSet(
+ new HashSet<>(
+ Arrays.asList(
+ "password",
+ "secret",
+ "fs.azure.account.key",
+ "apikey",
+ "api-key",
+ "auth-params",
+ "service-key",
+ "token",
+ "basic-auth",
+ "jaas.config",
+ "http-headers")));
+
+ @Override
+ public String factoryIdentifier() {
+ return "default";
+ }
+
+ @Override
+ public Set<ConfigOption<?>> requiredOptions() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public Set<ConfigOption<?>> optionalOptions() {
+ return Collections.emptySet();
+ }
+
+ @Override
+ public CatalogConnection createConnection(
+ SensitiveConnection connection, WritableSecretStore secretStore) {
+ Map<String, String> allOptions = connection.getOptions();
+
+ if (allOptions.containsKey(SECRET_REFERENCE_KEY)) {
+ throw new ValidationException(
+ String.format(
+ "Connection option '%s' is reserved and cannot be set by users.",
+ SECRET_REFERENCE_KEY));
+ }
+
+ Map<String, String> sensitiveOptions =
+ allOptions.entrySet().stream()
+ .filter(e -> SENSITIVE_FIELD_NAMES.contains(e.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+
+ Map<String, String> nonSensitiveOptions =
+ allOptions.entrySet().stream()
+ .filter(e -> !SENSITIVE_FIELD_NAMES.contains(e.getKey()))
+ .collect(
+ Collectors.toMap(
+ Map.Entry::getKey,
+ Map.Entry::getValue,
+ (a, b) -> a,
+ HashMap::new));
+
+ if (!sensitiveOptions.isEmpty()) {
+ final String secretId;
+ try {
+ secretId = secretStore.storeSecret(sensitiveOptions);
+ } catch (SecretException e) {
+ throw e;
+ } catch (RuntimeException e) {
+ throw new SecretException("Failed to store connection secret.", e);
+ }
+ nonSensitiveOptions.put(SECRET_REFERENCE_KEY, secretId);
Review Comment:
Let me add tests. `SECRET_REFERENCE_KEY` is not used out side of this class. Can convert it to private
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/DefaultConnectionFactory.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.table.api.ValidationException;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+import org.apache.flink.table.secret.exceptions.SecretNotFoundException;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Default implementation of {@link ConnectionFactory} that identifies sensitive fields by a
+ * predefined whitelist of field names.
+ *
+ * <p>During {@link #createConnection}, sensitive fields are extracted from the connection options
+ * and stored as a single secret in the {@link WritableSecretStore}. A reference key ({@value
+ * #SECRET_REFERENCE_KEY}) pointing to the stored secret is added to the returned {@link
+ * CatalogConnection}.
+ *
+ * <p>During {@link #resolveConnection}, the secret reference is used to retrieve the sensitive
+ * fields from the {@link ReadableSecretStore} and merge them back into the options.
+ *
+ * <p>The following field names are treated as sensitive by default: {@code password}, {@code
+ * secret}, {@code fs.azure.account.key}, {@code apikey}, {@code api-key}, {@code auth-params},
+ * {@code service-key}, {@code token}, {@code basic-auth}, {@code jaas.config}, {@code
+ * http-headers}.
+ */
+@Internal
+public class DefaultConnectionFactory implements ConnectionFactory {
+
+ /**
+ * Reserved option key used to store the reference to secrets in the secret store. The
+ * surrounding double underscores make collision with user-supplied option names unlikely; user
+ * options containing this key will be rejected at create-time.
+ */
+ public static final String SECRET_REFERENCE_KEY = "__flink.encrypted_secret_key__";
+
+ private static final Set<String> SENSITIVE_FIELD_NAMES =
+ Collections.unmodifiableSet(
+ new HashSet<>(
+ Arrays.asList(
+ "password",
Review Comment:
It's from the list here: https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L49. I can add a comment. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "lihaosky (via GitHub)" <gi...@apache.org>.
lihaosky commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3282456896
##########
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/ConnectionFactory.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.factories;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.table.catalog.CatalogConnection;
+import org.apache.flink.table.catalog.SensitiveConnection;
+import org.apache.flink.table.secret.ReadableSecretStore;
+import org.apache.flink.table.secret.WritableSecretStore;
+import org.apache.flink.table.secret.exceptions.SecretException;
+
+/**
+ * Factory for creating and resolving connections, handling the encryption and decryption of
+ * sensitive connection fields.
+ *
+ * <p>A {@code ConnectionFactory} is responsible for:
+ *
+ * <ul>
+ * <li>Extracting sensitive fields from a {@link SensitiveConnection} and storing them in a {@link
+ * WritableSecretStore}, returning a {@link CatalogConnection} that is safe to persist in a
+ * catalog.
+ * <li>Resolving a {@link CatalogConnection} from a catalog by retrieving its secrets from a
+ * {@link ReadableSecretStore} and returning a complete {@link SensitiveConnection}.
+ * </ul>
+ *
+ * @see org.apache.flink.table.factories.DefaultConnectionFactory
+ */
+@PublicEvolving
+public interface ConnectionFactory extends Factory {
+
+ /**
+ * The option key that identifies the connection type in a {@link SensitiveConnection}'s or
+ * {@link CatalogConnection}'s options. The value of this option is matched against the {@link
+ * #factoryIdentifier()} of registered {@code ConnectionFactory} implementations to discover the
+ * factory that handles a given connection (see FLIP-529).
+ */
+ String CONNECTION_TYPE_KEY = "type";
Review Comment:
But that will defeat the purpose of putting the type in Connection options during connection creation. And you can only have one Factory type for all connection? In the FLIP, type is an option in connection option during creation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3289210379
##########
flink-table/flink-table-common/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory:
##########
@@ -14,3 +14,4 @@
# limitations under the License.
org.apache.flink.table.module.CoreModuleFactory
+org.apache.flink.table.factories.DefaultConnectionFactory
Review Comment:
I'm very sorry about this comment. I posted it but wanted to delete it again. The `org.apache.flink.table.factories` was the right package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "lihaosky (via GitHub)" <gi...@apache.org>.
lihaosky commented on code in PR #28085:
URL: https://github.com/apache/flink/pull/28085#discussion_r3289410555
##########
flink-table/flink-table-common/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory:
##########
@@ -14,3 +14,4 @@
# limitations under the License.
org.apache.flink.table.module.CoreModuleFactory
+org.apache.flink.table.factories.DefaultConnectionFactory
Review Comment:
No problem. Moved it back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-39590] connection catalog manager changes [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28085:
URL: https://github.com/apache/flink/pull/28085#issuecomment-4360829501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16ecf0da2aa3c8b412b74abf3fd8033808a65feb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "16ecf0da2aa3c8b412b74abf3fd8033808a65feb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16ecf0da2aa3c8b412b74abf3fd8033808a65feb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org