You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Yu Watanabe <yu...@gmail.com> on 2019/09/08 13:23:27 UTC
How to buffer events using spark portable runner ?
Hello .
I would like to ask question related to timely processing as stated in
below page.
https://beam.apache.org/blog/2017/08/28/timely-processing.html
Python version: 3.7.4
apache beam version: 2.15.0
I currently use timely processing to first buffer events and send *bulk
requests *to elasticsearch. The source of data is bounded source and I use
DirectRunner for runner.
To have more memory resource , I am considering to move to process the
pipeline on apache spark using portable runner. However, according to
compatibility matrix,
*Timers *is not supported on apache spark.
https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-when
Is there anyway in portable runner that you can do similar processing
as *timely
processing* ?
This is my first time using portable runner and I appreciate if I can get
help with this.
Best Regards,
Yu Watanabe
--
Yu Watanabe
Weekend Freelancer who loves to challenge building data platform
yu.w.tennis@gmail.com
[image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
Twitter icon] <https://twitter.com/yuwtennis>
Re: How to buffer events using spark portable runner ?
Posted by Yu Watanabe <yu...@gmail.com>.
Lukasaz
Thank you for the reply.
I will try apache flink.
Thanks,
Yu
On Sun, Sep 8, 2019 at 11:59 PM Lukasz Cwik <lc...@google.com> wrote:
> Try using Apache Flink.
>
> On Sun, Sep 8, 2019 at 6:23 AM Yu Watanabe <yu...@gmail.com> wrote:
>
>> Hello .
>>
>> I would like to ask question related to timely processing as stated in
>> below page.
>>
>> https://beam.apache.org/blog/2017/08/28/timely-processing.html
>>
>> Python version: 3.7.4
>> apache beam version: 2.15.0
>>
>> I currently use timely processing to first buffer events and send *bulk
>> requests *to elasticsearch. The source of data is bounded source and I
>> use DirectRunner for runner.
>>
>> To have more memory resource , I am considering to move to process the
>> pipeline on apache spark using portable runner. However, according to
>> compatibility matrix,
>> *Timers *is not supported on apache spark.
>>
>>
>> https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-when
>>
>> Is there anyway in portable runner that you can do similar processing as *timely
>> processing* ?
>> This is my first time using portable runner and I appreciate if I can get
>> help with this.
>>
>> Best Regards,
>> Yu Watanabe
>>
>> --
>> Yu Watanabe
>> Weekend Freelancer who loves to challenge building data platform
>> yu.w.tennis@gmail.com
>> [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
>> Twitter icon] <https://twitter.com/yuwtennis>
>>
>
--
Yu Watanabe
Weekend Freelancer who loves to challenge building data platform
yu.w.tennis@gmail.com
[image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
Twitter icon] <https://twitter.com/yuwtennis>
Re: How to buffer events using spark portable runner ?
Posted by Lukasz Cwik <lc...@google.com>.
Try using Apache Flink.
On Sun, Sep 8, 2019 at 6:23 AM Yu Watanabe <yu...@gmail.com> wrote:
> Hello .
>
> I would like to ask question related to timely processing as stated in
> below page.
>
> https://beam.apache.org/blog/2017/08/28/timely-processing.html
>
> Python version: 3.7.4
> apache beam version: 2.15.0
>
> I currently use timely processing to first buffer events and send *bulk
> requests *to elasticsearch. The source of data is bounded source and I
> use DirectRunner for runner.
>
> To have more memory resource , I am considering to move to process the
> pipeline on apache spark using portable runner. However, according to
> compatibility matrix,
> *Timers *is not supported on apache spark.
>
>
> https://beam.apache.org/documentation/runners/capability-matrix/#cap-summary-when
>
> Is there anyway in portable runner that you can do similar processing as *timely
> processing* ?
> This is my first time using portable runner and I appreciate if I can get
> help with this.
>
> Best Regards,
> Yu Watanabe
>
> --
> Yu Watanabe
> Weekend Freelancer who loves to challenge building data platform
> yu.w.tennis@gmail.com
> [image: LinkedIn icon] <https://www.linkedin.com/in/yuwatanabe1> [image:
> Twitter icon] <https://twitter.com/yuwtennis>
>