[Question] Writing custom streaming input sources for Dataflow

[Taiga Ishida <ta...@gmail.com>]

Hello Beam Team, 

I am trying to write a custom streaming pubsub input source that requires authenticating with google cloud given the supplied credentials (the credentials may be different than the google cloud environment the pipeline is running in). My input source works if I run the pipeline on Dataflow in batch mode but hangs indefinitely when I run in streaming mode. I’m wondering if there is a special requirement when writing custom streaming input sources? 

Thank you,

Taiga

Re: [Question] Writing custom streaming input sources for Dataflow

[Luke Cwik <lc...@google.com>]

Are you writing a splittable DoFn?
Are you using Dataflow prime?
What kind of credential are you using that is valid for the lifetime of the
streaming job?

On Tue, Jan 25, 2022 at 7:03 AM Taiga Ishida <ta...@gmail.com>
wrote:

> Hello Beam Team,
>
> I am trying to write a custom streaming pubsub input source that requires
> authenticating with google cloud given the supplied credentials (the
> credentials may be different than the google cloud environment the pipeline
> is running in). My input source works if I run the pipeline on Dataflow in
> batch mode but hangs indefinitely when I run in streaming mode. I’m
> wondering if there is a special requirement when writing custom streaming
> input sources?
>
> Thank you,
>
> Taiga