[GitHub] [beam] GoosvandenBekerom commented on issue #25326: [Feature Request]: PubSubIO.Write doesn't work in Batch pipelines in Go SDK

["GoosvandenBekerom (via GitHub)" <gi...@apache.org>]

GoosvandenBekerom commented on issue #25326:
URL: https://github.com/apache/beam/issues/25326#issuecomment-1639891413

   We have a bunch of different teams in our company doing this same `BigQuery -> transform -> Pubsub` batch pipeline on dataflow for (initial) loading data into systems, most teams are writing their own version of this using the java sdk.
   
   I've been trying to set up a generic dataflow flex-template using the go sdk to standardize this, but getting into this missing sink issue. @lostluck I was looking into your workaround suggested here https://stackoverflow.com/questions/69651665/go-apache-beam-gcp-dataflow-could-not-find-the-sink-for-pubsub-check-that-th but I don't feel confident enough with beam to know what would be sane batch sizes etc.
   
   From the thread linked above I get that this feature is still under discussion, so I figured I would add my use case here to make you aware.
   
   If there is anything I can do to help, let me know.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org