You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "krutoileshii (via GitHub)" <gi...@apache.org> on 2026/01/09 02:14:56 UTC
[I] [Feature][Seatinnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
krutoileshii opened a new issue, #10309:
URL: https://github.com/apache/seatunnel/issues/10309
### Search before asking
- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
#### Background
Many organizations already rely on centralized secret management systems (key vaults) such as **Azure Key Vault**, **AWS Secrets Manager**, **HashiCorp Vault**, or **GCP Secret Manager** to securely store and rotate credentials. Native support for these systems would significantly improve SeaTunnel’s security posture and enterprise adoption.
#### Proposed Feature
Introduce **native support for external key vaults** in SeaTunnel configuration resolution, starting with **Azure Key Vault**, and designed to be extensible for other providers.
##### Key Capabilities
1. **Secret Reference Syntax in Configs**
Allow configuration values to reference secrets stored in a key vault, for example:
```hocon
password = ${keyvault:azure:my-key-vault/db-password}
```
2. **Pluggable Vault Provider Architecture**
* Define a generic `SecretProvider` interface.
* Implement initial providers:
* Azure Key Vault
* (Future) AWS Secrets Manager
* (Future) HashiCorp Vault
* (Future) GCP Secret Manager
3. **Authentication Support**
For Azure Key Vault:
* Managed Identity
* Service Principal (Client ID / Secret)
* Default Azure Credential chain
4. **Runtime Secret Resolution**
* Secrets resolved at runtime before job execution.
* No secrets persisted in logs, metrics, or checkpoints.
5. **Backward Compatibility**
* Existing configs continue to work unchanged.
* Vault integration is opt-in.
#### Example Use Case
```hocon
env {
execution.parallelism = 2
}
sink {
Jdbc {
url = "jdbc:sqlserver://mydb.database.windows.net:1433"
user = "seatunnel_user"
password = ${keyvault:azure:prod-kv/sql-password}
}
}
```
#### Benefits
* Improved security and compliance (SOC 2, ISO 27001, HIPAA, etc.)
* Centralized secret management and rotation
* Reduced operational risk
* Stronger enterprise and cloud-native adoption of SeaTunnel
#### Additional Considerations
* Caching and TTL for secret lookups
* Failure behavior (e.g., fail-fast if secret is unavailable)
* Clear documentation and examples
* Masking secrets in logs and error messages
#### Open Questions
* Should secret resolution occur at parse time or execution time?
* Should secrets be refreshable for long-running jobs?
* Preferred syntax and naming convention
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Seatunnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
Posted by "krutoileshii (via GitHub)" <gi...@apache.org>.
krutoileshii commented on issue #10309:
URL: https://github.com/apache/seatunnel/issues/10309#issuecomment-3746180057
Before jumping into one for for key vault, I'm trying to write one for simply AES encrypting them with a key stored on the server/client (In Config File).
What would be the best way to do this? Would it make sense to read it from seatunnel.yaml or instead just have a separate config for the encryption key?
Figured if i get that working first, the key-vault would be second.
A couple things I am trying to do with that right now:
1. Encryption string stored in config file or ENV Var
2. Same key could be used across all nodes to share easily.
Once I can get it working properly with AES, I would probably use same type of setup for KeyVault.
Now given that this is a plugin, what would be the best way to get it included by default into the project? I think it would be beneficial to have at least basic encryption in place beyond the base64.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Seatinnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10309:
URL: https://github.com/apache/seatunnel/issues/10309#issuecomment-3726850146
@krutoileshii
Good suggestion, would you like to implement this feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Seatunnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
Posted by "krutoileshii (via GitHub)" <gi...@apache.org>.
krutoileshii commented on issue #10309:
URL: https://github.com/apache/seatunnel/issues/10309#issuecomment-3727401030
Another thought:
Plug into variable substitution, but need to prevent client from trying to sub as it could fail since client shouldn't need keyvault kreds. Only workers and server need them.
https://seatunnel.apache.org/docs/2.3.12/concept/config#config-variable-substitution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Seatunnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10309:
URL: https://github.com/apache/seatunnel/issues/10309#issuecomment-4414923642
Thanks @ryanvflannery for picking this up.
I rechecked the current config path before replying. The important point is that SeaTunnel already has a concrete extension point here: `ConfigShade` + `ConfigShadeUtils`.
Today the current behavior is still parse-time decryption, not worker-only late resolution:
- CLI / file loading goes through `ConfigBuilder`, which calls `ConfigShadeUtils.decryptConfig(...)` while loading the config.
- REST submission also calls the same `decryptConfig(...)` path on the server before job submission.
So for a first implementation, I strongly recommend keeping the scope aligned with the current contract instead of introducing a new generic runtime secret-resolution model in the same PR.
A good first scope would be:
1. implement one provider via `ConfigShade`;
2. use `shade.identifier` + `shade.properties` for provider selection and provider-specific settings;
3. keep decryption limited to the current sensitive-option model (`password`, `token`, etc., plus explicit `shade.options` when needed);
4. keep it opt-in and backward-compatible.
A few boundaries are worth keeping explicit:
- server-local / worker-local key material outside the submitted job config is not part of the current `ConfigShade` contract today, so that should be treated as a follow-up design topic rather than bundled into the first PR;
- a built-in AES implementation in core and external key-vault integration are related, but they are not the same scope. I would avoid mixing both in the first patch;
- if we eventually want "client does not need vault credentials, only server/workers do", that is a larger architectural change than the current SPI path, because the config is decrypted where it is parsed today.
For this particular issue, I would keep the PR centered on an external provider path first. If you want to add a built-in AES implementation in core later, please treat that as a separate follow-up issue/PR so the key-management discussion stays clear.
If you open a focused draft PR around the existing `ConfigShade` SPI, that should be a good starting point. Happy to review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Seatunnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
Posted by "ryanvflannery (via GitHub)" <gi...@apache.org>.
ryanvflannery commented on issue #10309:
URL: https://github.com/apache/seatunnel/issues/10309#issuecomment-4412179425
Hi, I’ll be working on this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Seatunnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10309:
URL: https://github.com/apache/seatunnel/issues/10309#issuecomment-3732507174
Hi @krutoileshii,
Thanks for the proposal! This is a great feature and aligns well with our enterprise security needs.
We already have an SPI interface designed for configuration encryption/decryption called `ConfigShade`. I believe this is exactly the extension point you are looking for, and it would be the best place to implement the Key Vault integration without modifying the core config loading logic.
If we don't let client do this variable substitution, the change would be **quite significant** (architectural level).
Current Architecture:
In the CLI submission mode (`seatunnel.sh`), the Client acts as the **Driver**. It needs to:
1. Parse the config.
2. **Connect to the datasource to infer Schema/Catalog.** (This is the blocking point).
3. Build the Logical DAG.
4. Send the DAG to the Master.
If we prevent the Client from substituting variables (decrypting secrets), the Client will fail to connect to the database to fetch metadata (Schema), and the job submission will fail immediately at the client side.
To achieve "Client-side encryption, Server-side decryption", we would effectively need to move the entire DAG building process (including Schema inference) from the Client to the Master node (similar to "Application Mode" in other compute engines). This is a much larger scope than just adding KeyVault support.
For now, I recommend sticking to the standard SPI approach where both Client and Server have access to the decryption provider. If the user submits jobs via **REST API**, the parsing happens on the Server, so the Client doesn't need credentials in that specific case.
Here is a brief guide on how to get started:
### 1. Implement the Interface
You need to create a new class that implements `org.apache.seatunnel.api.configuration.ConfigShade`.
* **`getIdentifier()`**: Return a unique identifier for your provider, e.g., `"azure-kv"`.
* **`decrypt(String content)`**: This is where the core logic goes. The `content` parameter will receive the value from your config file (e.g., `${keyvault:azure:my-key-vault/db-password}`).
* Your implementation should check if the content matches your expected pattern.
* If it matches, connect to Azure Key Vault (using the Azure SDK), fetch the secret, and return the actual plaintext value.
* If it doesn't match, you can return the content as is.
* **`encrypt(String content)`**: (Optional) You can implement this if you want to support converting a plaintext value into a Key Vault reference (e.g., uploading it to the Vault), or just return the original `content` if this flow isn't needed.
### 2. Configuration & Usage
Users can select your implementation in their job configuration (e.g., v2.batch.config.template or their job file) under the `env` block.
```hocon
env {
# This tells SeaTunnel to use your SPI implementation
shade.identifier = "azure-kv"
# You can pass additional properties to your Init method via shade.props
shade.props {
vault.url = "https://my-vault.vault.azure.net/"
# ...
}
}
sink {
Jdbc {
password = "${keyvault:azure:my-key-vault/db-password}"
# ...
}
}
```
### 3. Code References
* **Interface**: `org.apache.seatunnel.api.configuration.ConfigShade` (in seatunnel-api module).
* **Invocation Logic**: The logic that loads and calls this SPI is in `org.apache.seatunnel.core.starter.utils.ConfigShadeUtils`.
* **Example**: You can check the inner class `Base64ConfigShade` in ConfigShadeUtils.java for a simple reference implementation.
By using this SPI, `ConfigShadeUtils` will automatically identify sensitive fields (like `password`, `username`) and call your `decrypt` method for them during the job startup, so you don't need to manually intercept configurations.
Let me know if you have any further questions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Seatinnel] Introduce support for key vaults such as Azure KeyVault in configs [seatunnel]
Posted by "krutoileshii (via GitHub)" <gi...@apache.org>.
krutoileshii commented on issue #10309:
URL: https://github.com/apache/seatunnel/issues/10309#issuecomment-3726873669
my java skills are pretty slim. if someone can point where to start, I could take a look.
basically the way I see is intercept configs and replace there
so worker,Master, and job submission for client.
Not sure where else this would have to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org