You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Mátyás Manninger <mm...@lunar.app> on 2022/03/29 12:45:20 UTC
Problem with main pipeline triggering with side input while testing locally
Dear Users,
I am looking for some help on this issue I am having when testing a side
input pattern locally. Here is my StackOverflow question for reference:
https://stackoverflow.com/questions/71648731/fixedwindow-apparently-not-firing-in-streaming-python-beam-pipeline
As mentioned there I am trying to achieve the first side input pattern
mentioned here: https://beam.apache.org/documentation/patterns/side-inputs/.
This does not have a python code sample, is this because it's not supposed
to work in python?
I am very thankful for any help, tips, pointers :)
Best regards,
Matyas Manninger
Re: Problem with main pipeline triggering with side input while testing locally
Posted by Robert Bradshaw <ro...@google.com>.
In general, batch is much better supported on the Python local runner
than streaming. You could try developing/running your pipeline with a
fixed input in batch mode. You can also use TestStream which has
better control than reading from PubSub. (One can still use windows
and timestamps in batch mode, but not sources like PubSub.)
On Wed, Mar 30, 2022 at 8:17 AM Mátyás Manninger <mm...@lunar.app> wrote:
>
> I figured out that the problem is with the local runner. Once I deploy the pipeline to Dataflow everything works as expected. Does anyone have an idea what might be the issue when running locally? It's tedious to debug a complicated pipeline if I can't do it locally.
>
> Thanks,
> Matyas
>
> On Tue, 29 Mar 2022 at 14:45, Mátyás Manninger <mm...@lunar.app> wrote:
>>
>> Dear Users,
>>
>> I am looking for some help on this issue I am having when testing a side input pattern locally. Here is my StackOverflow question for reference: https://stackoverflow.com/questions/71648731/fixedwindow-apparently-not-firing-in-streaming-python-beam-pipeline
>> As mentioned there I am trying to achieve the first side input pattern mentioned here: https://beam.apache.org/documentation/patterns/side-inputs/. This does not have a python code sample, is this because it's not supposed to work in python?
>> I am very thankful for any help, tips, pointers :)
>>
>> Best regards,
>> Matyas Manninger
Re: Problem with main pipeline triggering with side input while testing locally
Posted by Mátyás Manninger <mm...@lunar.app>.
I figured out that the problem is with the local runner. Once I deploy the
pipeline to Dataflow everything works as expected. Does anyone have an idea
what might be the issue when running locally? It's tedious to debug a
complicated pipeline if I can't do it locally.
Thanks,
Matyas
On Tue, 29 Mar 2022 at 14:45, Mátyás Manninger <mm...@lunar.app> wrote:
> Dear Users,
>
> I am looking for some help on this issue I am having when testing a side
> input pattern locally. Here is my StackOverflow question for reference:
> https://stackoverflow.com/questions/71648731/fixedwindow-apparently-not-firing-in-streaming-python-beam-pipeline
> As mentioned there I am trying to achieve the first side input pattern
> mentioned here:
> https://beam.apache.org/documentation/patterns/side-inputs/. This does
> not have a python code sample, is this because it's not supposed to work in
> python?
> I am very thankful for any help, tips, pointers :)
>
> Best regards,
> Matyas Manninger
>