You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Taher Koitawala <ta...@gmail.com> on 2019/10/30 11:06:49 UTC
Streaming data from Pubsub to Spanner with Beam dataflow pipeline
Hi All,
My current use-case is to write data from Pubsub to Spanner using
a streaming pipeline. I do see that Beam does have a SpannerIO to write.
However, pubsub being streaming and Spanner being RDBMS like, it
would be helpful to you guys can tell me if this will be performant enough
or not. If someone has already tried this out and can give me a few
caveats, then that would be really awesome.
Regards,
Taher Koitawala
Re: Streaming data from Pubsub to Spanner with Beam dataflow pipeline
Posted by Kenneth Knowles <ke...@apache.org>.
Moving to user@beam.apache.org, the best mailing list for questions like
this.
Yes, this kind of workload is a core use case for Beam. If you have a
problem, please write to this user list with details.
Kenn
On Wed, Oct 30, 2019 at 4:07 AM Taher Koitawala <ta...@gmail.com> wrote:
> Hi All,
> My current use-case is to write data from Pubsub to Spanner
> using a streaming pipeline. I do see that Beam does have a SpannerIO to
> write.
>
> However, pubsub being streaming and Spanner being RDBMS like, it
> would be helpful to you guys can tell me if this will be performant enough
> or not. If someone has already tried this out and can give me a few
> caveats, then that would be really awesome.
>
>
> Regards,
> Taher Koitawala
>
Re: Streaming data from Pubsub to Spanner with Beam dataflow pipeline
Posted by Kenneth Knowles <ke...@apache.org>.
Moving to user@beam.apache.org, the best mailing list for questions like
this.
Yes, this kind of workload is a core use case for Beam. If you have a
problem, please write to this user list with details.
Kenn
On Wed, Oct 30, 2019 at 4:07 AM Taher Koitawala <ta...@gmail.com> wrote:
> Hi All,
> My current use-case is to write data from Pubsub to Spanner
> using a streaming pipeline. I do see that Beam does have a SpannerIO to
> write.
>
> However, pubsub being streaming and Spanner being RDBMS like, it
> would be helpful to you guys can tell me if this will be performant enough
> or not. If someone has already tried this out and can give me a few
> caveats, then that would be really awesome.
>
>
> Regards,
> Taher Koitawala
>