You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "richpodraza (via GitHub)" <gi...@apache.org> on 2023/02/07 17:06:49 UTC
[GitHub] [beam] richpodraza opened a new issue, #25361: [Feature Request]: Support ElasticsearchIO ConnectionConfiguration (or its fields) dynamically, e.g. with ValueProviders
richpodraza opened a new issue, #25361:
URL: https://github.com/apache/beam/issues/25361
### What would you like to happen?
I would like to be able to dynamically provide ElasticsearchIO the host, port, username, and password (or the entire ConnectionConfiguration), similar to this issue: https://github.com/apache/beam/pull/25024
In the issue above the need was for a token that changes frequently. In my case, authentication is via username and password provided by GCP Secret Manager, and I'd like to be able to query for those credentials similar to what is described here: https://henrysuryawirawan.com/posts/dataflow-secret-manager/
Additionally, if host and port could be supplied dynamically (e.g. through ValueProviders or some other means), then I could have a single compiled template that works in different execution environments (dev, prod, etc.) and whenever these values change.
Other Apache Beam sdk connectors support similar ValueProviders, such as RedisConnectionConfiguration, BigQueryIO, PubsubIO
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] richpodraza closed issue #25361: [Feature Request]: Support ElasticsearchIO ConnectionConfiguration (or its fields) dynamically, e.g. with ValueProviders
Posted by "richpodraza (via GitHub)" <gi...@apache.org>.
richpodraza closed issue #25361: [Feature Request]: Support ElasticsearchIO ConnectionConfiguration (or its fields) dynamically, e.g. with ValueProviders
URL: https://github.com/apache/beam/issues/25361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #25361: [Feature Request]: Support ElasticsearchIO ConnectionConfiguration (or its fields) dynamically, e.g. with ValueProviders
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #25361:
URL: https://github.com/apache/beam/issues/25361#issuecomment-1421494286
Are you trying to use an existing Google provided template or are you trying to define your own ?
You should not have to use ValueProiders if you use Flex templates: https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] richpodraza commented on issue #25361: [Feature Request]: Support ElasticsearchIO ConnectionConfiguration (or its fields) dynamically, e.g. with ValueProviders
Posted by "richpodraza (via GitHub)" <gi...@apache.org>.
richpodraza commented on issue #25361:
URL: https://github.com/apache/beam/issues/25361#issuecomment-1425857331
@chamikaramj I am defining my own and had indeed not heard of Flex Templates. Thanks for the tip, this is what I will pursue!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org