You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 15:29:40 UTC

[GitHub] [beam] damccorm opened a new issue, #20123: KafkaIO attempting to make connection when uploading via Dataflow Runner

damccorm opened a new issue, #20123:
URL: https://github.com/apache/beam/issues/20123

   Given our Kafka is running in GCP, within a VPC that is not accessible outside of the VPC.
   
   When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD worker, it attempts to make a connection to the broker. Our broker isn't available outside the VPC, so this process fails. We can remedy this by running a pod from within the VPC to upload, but it causes an awkward step within our CI/CD process.
   
   If there is a more elegant solution to this, such as KafkaIO taking in a ValueProvider for bootstrap servers.
   
   Imported from Jira [BEAM-9525](https://issues.apache.org/jira/browse/BEAM-9525). Original Jira may contain additional context.
   Reported by: rossr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org