You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Etienne Chauchot <ec...@apache.org> on 2019/09/12 00:10:03 UTC
Re: Pointers on Contributing to Structured Streaming Spark Runner
Hi Rahul,Sure, and great ! Thanks for proposing !If you want details, here is the presentation I did 30 mins ago at the
apachecon. You will find the video on youtube shortly but in the meantime, here is my presentation slides.
And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
https://github.com/apache/beam/tree/spark-runner_structured-streaming
BestEtienne
Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
> Hi Etienne,
>
> I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works in Progress.
> I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
>
> I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming Spark Runner.
>
> Can you please point me in the right direction?
>
> Thanks,
> Rahul
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Etienne Chauchot <ec...@apache.org>.
Hi Rui,Thanks for proposing to contribute to this new runner !
Here are the pointers:- SS runner branch: https://github.com/apache/beam/tree/spark-runner_structured-streaming- spark
design doc for multiple watermarks support:
https://docs.google.com/document/d/1IAH9UQJPUiUCLd7H6dazRK2k1szDX38SnM6GVNZYvUo/edit#t# . There is also a good
discussion in this Spark PR branch : https://github.com/apache/spark/pull/23576
As Alexey mentioned in this thread, the SS runner feature branch will be merged into master when the runner is in good
shape. I think we will not wait for the streaming part as it requires a deep change in the spark core + impl of the
streaming part of the Beam runner, so it would take too long. IMHO we need to get batch mode of the new runner in a
stable state (encoders ongoing work, fix bad perf of the 2 nexmark queries, ...) before merging.
Best,Etienne
Le mercredi 11 septembre 2019 à 18:02 -0700, Rui Wang a écrit :
> Hello Etienne,
> Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-aggregations
> in its streaming mode but design is ongoing. Do you have a link or something else to their design discussion/doc?
>
>
> -Rui
> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org> wrote:
> > Hi Rahul,Sure, and great ! Thanks for proposing !If you want details, here is the presentation I did 30 mins ago at
> > the apachecon. You will find the video on youtube shortly but in the meantime, here is my presentation slides.
> > And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
> > https://github.com/apache/beam/tree/spark-runner_structured-streaming
> > BestEtienne
> > Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
> > > Hi Etienne,
> > >
> > > I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works in
> > > Progress.
> > > I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
> > >
> > > I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming Spark
> > > Runner.
> > >
> > > Can you please point me in the right direction?
> > >
> > > Thanks,
> > > Rahul
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Rui Wang <ru...@google.com>.
+1 on merging the runner into master, which make it more discoverable and
easy to contribute (I am also interested in contributing).
-Rui
On Tue, Sep 17, 2019 at 3:36 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Hi Xinyu,
>
> Great to hear that you wish to contribute into new Spark runner! We used
> to have the sync meetings about all Spark runners in general every two
> weeks, so feel free to let know us if you want to participate too.
>
> Also, as one of the contributors for Structural Streaming Spark runner
> (yes, we need to find a shorter way how to call it =), I agree that it’s a
> good time to merge it into master (even if it’s not 100% ready). Then we
> can create a roadmap with Jiras tasks and push code in normal PR-based way,
> so it will be easier to discover new changes and track the work progress.
> We only need to prevent users that it’s still under development and not
> ready to use in production.
>
> Scheme part is still quite vague and hazy, so it’s a good topic for
> separate discussion. I believe that it would be much effective in terms of
> performance if we will be able to have a strong relation between Beam and
> Spark schemes in the end.
>
> On 13 Sep 2019, at 21:16, Xinyu Liu <xi...@gmail.com> wrote:
>
> Hi, Etienne,
>
> The slides are very informative! Thanks for sharing the details about how
> the Beam API are mapped into Spark Structural Streaming. We (LinkedIn) are
> also interested in trying the new SparkRunner to run Beam pipeine in batch,
> and contribute to it too. From my understanding, seems the functionality on
> batch side is mostly complete and covers quite a large percentage of the
> tests (a few missing pieces like state and timer in ParDo and SDF). If so,
> is it possible to merge the new runner sooner into master so it's much
> easier for us to pull it in (we have an internal fork) and contribute back?
>
> Also curious about the scheme part in the runner. Seems we can leverage
> the schema-aware work in PCollection and translate from Beam schema to
> Spark, so it can be optimized in the planner layer. It will be great to
> hear back your plans on that.
>
> Congrats on this great work!
> Thanks,
> Xinyu
>
> On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
>
>> Hello Etienne,
>>
>> Your slide mentioned that streaming mode development is blocked because
>> Spark lacks supporting multiple-aggregations in its streaming mode but
>> design is ongoing. Do you have a link or something else to their design
>> discussion/doc?
>>
>>
>> -Rui
>>
>> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org>
>> wrote:
>>
>>> Hi Rahul,
>>> Sure, and great ! Thanks for proposing !
>>> If you want details, here is the presentation I did 30 mins ago at the
>>> apachecon. You will find the video on youtube shortly but in the meantime,
>>> here is my presentation slides.
>>>
>>> And here is the structured streaming branch. I'll be happy to review
>>> your PRs, thanks !
>>>
>>> <https://github.com/apache/beam/tree/spark-runner_structured-streaming>
>>> https://github.com/apache/beam/tree/spark-runner_structured-streaming
>>>
>>> Best
>>> Etienne
>>>
>>> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>>>
>>> Hi Etienne,
>>>
>>> I came to know about the work going on in Structured Streaming Spark
>>> Runner from Apache Beam Wiki - Works in Progress.
>>> I have contributed to BeamSql earlier. And I am working on supporting
>>> PCollectionView in BeamSql.
>>>
>>> I would love to understand the Runner's side of Apache Beam and
>>> contribute to the Structured Streaming Spark Runner.
>>>
>>> Can you please point me in the right direction?
>>>
>>> Thanks,
>>> Rahul
>>>
>>>
>
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Alexey Romanenko <ar...@gmail.com>.
Hi Xinyu,
Great to hear that you wish to contribute into new Spark runner! We used to have the sync meetings about all Spark runners in general every two weeks, so feel free to let know us if you want to participate too.
Also, as one of the contributors for Structural Streaming Spark runner (yes, we need to find a shorter way how to call it =), I agree that it’s a good time to merge it into master (even if it’s not 100% ready). Then we can create a roadmap with Jiras tasks and push code in normal PR-based way, so it will be easier to discover new changes and track the work progress. We only need to prevent users that it’s still under development and not ready to use in production.
Scheme part is still quite vague and hazy, so it’s a good topic for separate discussion. I believe that it would be much effective in terms of performance if we will be able to have a strong relation between Beam and Spark schemes in the end.
> On 13 Sep 2019, at 21:16, Xinyu Liu <xi...@gmail.com> wrote:
>
> Hi, Etienne,
>
> The slides are very informative! Thanks for sharing the details about how the Beam API are mapped into Spark Structural Streaming. We (LinkedIn) are also interested in trying the new SparkRunner to run Beam pipeine in batch, and contribute to it too. From my understanding, seems the functionality on batch side is mostly complete and covers quite a large percentage of the tests (a few missing pieces like state and timer in ParDo and SDF). If so, is it possible to merge the new runner sooner into master so it's much easier for us to pull it in (we have an internal fork) and contribute back?
>
> Also curious about the scheme part in the runner. Seems we can leverage the schema-aware work in PCollection and translate from Beam schema to Spark, so it can be optimized in the planner layer. It will be great to hear back your plans on that.
>
> Congrats on this great work!
> Thanks,
> Xinyu
>
> On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ruwang@google.com <ma...@google.com>> wrote:
> Hello Etienne,
>
> Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-aggregations in its streaming mode but design is ongoing. Do you have a link or something else to their design discussion/doc?
>
>
> -Rui
>
> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <echauchot@apache.org <ma...@apache.org>> wrote:
> Hi Rahul,
> Sure, and great ! Thanks for proposing !
> If you want details, here is the presentation I did 30 mins ago at the apachecon. You will find the video on youtube shortly but in the meantime, here is my presentation slides.
>
> And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
>
> <https://github.com/apache/beam/tree/spark-runner_structured-streaming>https://github.com/apache/beam/tree/spark-runner_structured-streaming <https://github.com/apache/beam/tree/spark-runner_structured-streaming>
>
> Best
> Etienne
>
> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>> Hi Etienne,
>>
>> I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works in Progress.
>> I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
>>
>> I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming Spark Runner.
>>
>> Can you please point me in the right direction?
>>
>> Thanks,
>> Rahul
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Alexey Romanenko <ar...@gmail.com>.
I don’t see any updates on my calendar. Does it work for others?
> On 19 Sep 2019, at 17:16, Ismaël Mejía <ie...@gmail.com> wrote:
>
> 25/09 looks ok. I just updated the meeting invitation to the new
> date.I will prepare a mini agenda in the shared minute document in the
> meantime.
> I cannot see the old invitees, can someone please confirm me they see
> the date updated.
> Thanks,
> Ismaël
>
> On Thu, Sep 19, 2019 at 2:13 PM Etienne Chauchot <ec...@apache.org> wrote:
>>
>> Hi Rahul and Xinyu,
>> I just added you to the list of guests in the meeting. Time is 5pm GMT +2.
>> That being said, for some reason last meeting scheduled was 08/28. Ismael initially created the meeting, I do not have the rights to add a new date. Ismael can you add a date ? I suggest 09/25. WDYT ?
>>
>> Best
>> Etienne
>>
>> Le jeudi 19 septembre 2019 à 00:49 +0530, rahul patwari a écrit :
>>
>> Hi,
>>
>> I would love to join the call.
>> Can you also share the meeting invitation with me?
>>
>> Thanks,
>> Rahul
>>
>> On Wed 18 Sep, 2019, 11:48 PM Xinyu Liu, <xi...@gmail.com> wrote:
>>
>> Alexey and Etienne: I'm very happy to join the sync-up meeting. Please forward the meeting info to me. I am based in California, US and hopefully the time will work :).
>>
>> Thanks,
>> Xinyu
>>
>> On Wed, Sep 18, 2019 at 6:39 AM Etienne Chauchot <ec...@apache.org> wrote:
>>
>> Hi Xinyu,
>>
>> Thanks for offering help ! My comments are inline:
>>
>> Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit :
>>
>> Hi, Etienne,
>>
>> The slides are very informative! Thanks for sharing the details about how the Beam API are mapped into Spark Structural Streaming.
>>
>>
>> Thanks !
>>
>> We (LinkedIn) are also interested in trying the new SparkRunner to run Beam pipeine in batch, and contribute to it too. From my understanding, seems the functionality on batch side is mostly complete and covers quite a large percentage of the tests (a few missing pieces like state and timer in ParDo and SDF).
>>
>>
>> Correct, it passes 89% of the tests, but there is more than SDF, state and timer missing, there is also ongoing encoders work that I would like to commit/push before merging.
>>
>> If so, is it possible to merge the new runner sooner into master so it's much easier for us to pull it in (we have an internal fork) and contribute back?
>>
>>
>> Sure, see my other mail on this thread. As Alexey mentioned, please join the sync meeting we have, the more the merrier !
>>
>>
>> Also curious about the scheme part in the runner. Seems we can leverage the schema-aware work in PCollection and translate from Beam schema to Spark, so it can be optimized in the planner layer. It will be great to hear back your plans on that.
>>
>>
>> Well, it is not designed yet but, if you remember my talk, we need to store beam windowing information with the data itself, so ending up having a dataset<WindowedValue> . One lead that was discussed is to store it as a Spark schema such as this:
>>
>> 1. field1: binary data for beam windowing information (cannot be mapped to fields because beam windowing info is complex structure)
>>
>> 2. fields of data as defined in the Beam schema if there is one
>>
>>
>> Congrats on this great work!
>>
>> Thanks !
>>
>> Best,
>>
>> Etienne
>>
>> Thanks,
>> Xinyu
>>
>> On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
>>
>> Hello Etienne,
>>
>> Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-aggregations in its streaming mode but design is ongoing. Do you have a link or something else to their design discussion/doc?
>>
>>
>> -Rui
>>
>> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org> wrote:
>>
>> Hi Rahul,
>> Sure, and great ! Thanks for proposing !
>> If you want details, here is the presentation I did 30 mins ago at the apachecon. You will find the video on youtube shortly but in the meantime, here is my presentation slides.
>>
>> And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
>>
>> https://github.com/apache/beam/tree/spark-runner_structured-streaming
>>
>> Best
>> Etienne
>>
>> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>>
>> Hi Etienne,
>>
>> I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works in Progress.
>> I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
>>
>> I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming Spark Runner.
>>
>> Can you please point me in the right direction?
>>
>> Thanks,
>> Rahul
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
25/09 looks ok. I just updated the meeting invitation to the new
date.I will prepare a mini agenda in the shared minute document in the
meantime.
I cannot see the old invitees, can someone please confirm me they see
the date updated.
Thanks,
Ismaël
On Thu, Sep 19, 2019 at 2:13 PM Etienne Chauchot <ec...@apache.org> wrote:
>
> Hi Rahul and Xinyu,
> I just added you to the list of guests in the meeting. Time is 5pm GMT +2.
> That being said, for some reason last meeting scheduled was 08/28. Ismael initially created the meeting, I do not have the rights to add a new date. Ismael can you add a date ? I suggest 09/25. WDYT ?
>
> Best
> Etienne
>
> Le jeudi 19 septembre 2019 à 00:49 +0530, rahul patwari a écrit :
>
> Hi,
>
> I would love to join the call.
> Can you also share the meeting invitation with me?
>
> Thanks,
> Rahul
>
> On Wed 18 Sep, 2019, 11:48 PM Xinyu Liu, <xi...@gmail.com> wrote:
>
> Alexey and Etienne: I'm very happy to join the sync-up meeting. Please forward the meeting info to me. I am based in California, US and hopefully the time will work :).
>
> Thanks,
> Xinyu
>
> On Wed, Sep 18, 2019 at 6:39 AM Etienne Chauchot <ec...@apache.org> wrote:
>
> Hi Xinyu,
>
> Thanks for offering help ! My comments are inline:
>
> Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit :
>
> Hi, Etienne,
>
> The slides are very informative! Thanks for sharing the details about how the Beam API are mapped into Spark Structural Streaming.
>
>
> Thanks !
>
> We (LinkedIn) are also interested in trying the new SparkRunner to run Beam pipeine in batch, and contribute to it too. From my understanding, seems the functionality on batch side is mostly complete and covers quite a large percentage of the tests (a few missing pieces like state and timer in ParDo and SDF).
>
>
> Correct, it passes 89% of the tests, but there is more than SDF, state and timer missing, there is also ongoing encoders work that I would like to commit/push before merging.
>
> If so, is it possible to merge the new runner sooner into master so it's much easier for us to pull it in (we have an internal fork) and contribute back?
>
>
> Sure, see my other mail on this thread. As Alexey mentioned, please join the sync meeting we have, the more the merrier !
>
>
> Also curious about the scheme part in the runner. Seems we can leverage the schema-aware work in PCollection and translate from Beam schema to Spark, so it can be optimized in the planner layer. It will be great to hear back your plans on that.
>
>
> Well, it is not designed yet but, if you remember my talk, we need to store beam windowing information with the data itself, so ending up having a dataset<WindowedValue> . One lead that was discussed is to store it as a Spark schema such as this:
>
> 1. field1: binary data for beam windowing information (cannot be mapped to fields because beam windowing info is complex structure)
>
> 2. fields of data as defined in the Beam schema if there is one
>
>
> Congrats on this great work!
>
> Thanks !
>
> Best,
>
> Etienne
>
> Thanks,
> Xinyu
>
> On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
>
> Hello Etienne,
>
> Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-aggregations in its streaming mode but design is ongoing. Do you have a link or something else to their design discussion/doc?
>
>
> -Rui
>
> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org> wrote:
>
> Hi Rahul,
> Sure, and great ! Thanks for proposing !
> If you want details, here is the presentation I did 30 mins ago at the apachecon. You will find the video on youtube shortly but in the meantime, here is my presentation slides.
>
> And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
>
> https://github.com/apache/beam/tree/spark-runner_structured-streaming
>
> Best
> Etienne
>
> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>
> Hi Etienne,
>
> I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works in Progress.
> I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
>
> I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming Spark Runner.
>
> Can you please point me in the right direction?
>
> Thanks,
> Rahul
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Vishwas Bm <bm...@gmail.com>.
Hi Ismael,
Can you also add me to this meeting. I would also like to contribute.
Regards,
Vishwas
On 2019/09/19 12:13:40, Etienne Chauchot <ec...@apache.org> wrote:
> Hi Rahul and Xinyu,I just added you to the list of guests in the meeting. Time is 5pm GMT +2. That being said, for some
> reason last meeting scheduled was 08/28. Ismael initially created the meeting, I do not have the rights to add a new
> date. Ismael can you add a date ? I suggest 09/25. WDYT ?
> BestEtienne
> Le jeudi 19 septembre 2019 à 00:49 +0530, rahul patwari a écrit :
> > Hi,
> > I would love to join the call.
> > Can you also share the meeting invitation with me?
> >
> > Thanks,
> > Rahul
> > On Wed 18 Sep, 2019, 11:48 PM Xinyu Liu, <xi...@gmail.com> wrote:
> > > Alexey and Etienne: I'm very happy to join the sync-up meeting. Please forward the meeting info to me. I am based in
> > > California, US and hopefully the time will work :).
> > > Thanks,
> > > Xinyu
> > > On Wed, Sep 18, 2019 at 6:39 AM Etienne Chauchot <ec...@apache.org> wrote:
> > > > Hi Xinyu,
> > > > Thanks for offering help ! My comments are inline:
> > > > Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit :
> > > > > Hi, Etienne,
> > > > > The slides are very informative! Thanks for sharing the details about how the Beam API are mapped into Spark
> > > > > Structural Streaming.
> > > >
> > > > Thanks !
> > > > > We (LinkedIn) are also interested in trying the new SparkRunner to run Beam pipeine in batch, and contribute to
> > > > > it too. From my understanding, seems the functionality on batch side is mostly complete and covers quite a large
> > > > > percentage of the tests (a few missing pieces like state and timer in ParDo and SDF).
> > > >
> > > > Correct, it passes 89% of the tests, but there is more than SDF, state and timer missing, there is also ongoing
> > > > encoders work that I would like to commit/push before merging.
> > > > > If so, is it possible to merge the new runner sooner into master so it's much easier for us to pull it in (we
> > > > > have an internal fork) and contribute back?
> > > >
> > > > Sure, see my other mail on this thread. As Alexey mentioned, please join the sync meeting we have, the more the
> > > > merrier !
> > > > > Also curious about the scheme part in the runner. Seems we can leverage the schema-aware work in PCollection and
> > > > > translate from Beam schema to Spark, so it can be optimized in the planner layer. It will be great to hear back
> > > > > your plans on that.
> > > >
> > > > Well, it is not designed yet but, if you remember my talk, we need to store beam windowing information with the
> > > > data itself, so ending up having a dataset<WindowedValue> . One lead that was discussed is to store it as a Spark
> > > > schema such as this:
> > > > 1. field1: binary data for beam windowing information (cannot be mapped to fields because beam windowing info is
> > > > complex structure)
> > > > 2. fields of data as defined in the Beam schema if there is one
> > > >
> > > > > Congrats on this great work!
> > > > Thanks !
> > > > Best,
> > > > Etienne
> > > > > Thanks,
> > > > > Xinyu
> > > > > On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
> > > > > > Hello Etienne,
> > > > > > Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-
> > > > > > aggregations in its streaming mode but design is ongoing. Do you have a link or something else to their design
> > > > > > discussion/doc?
> > > > > >
> > > > > >
> > > > > > -Rui
> > > > > > On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org> wrote:
> > > > > > > Hi Rahul,Sure, and great ! Thanks for proposing !If you want details, here is the presentation I did 30 mins
> > > > > > > ago at the apachecon. You will find the video on youtube shortly but in the meantime, here is my
> > > > > > > presentation slides.
> > > > > > > And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
> > > > > > > https://github.com/apache/beam/tree/spark-runner_structured-streaming
> > > > > > > BestEtienne
> > > > > > > Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
> > > > > > > > Hi Etienne,
> > > > > > > >
> > > > > > > > I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works
> > > > > > > > in Progress.
> > > > > > > > I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
> > > > > > > >
> > > > > > > > I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming
> > > > > > > > Spark Runner.
> > > > > > > >
> > > > > > > > Can you please point me in the right direction?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Rahul
>
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Etienne Chauchot <ec...@apache.org>.
Hi Rahul and Xinyu,I just added you to the list of guests in the meeting. Time is 5pm GMT +2. That being said, for some
reason last meeting scheduled was 08/28. Ismael initially created the meeting, I do not have the rights to add a new
date. Ismael can you add a date ? I suggest 09/25. WDYT ?
BestEtienne
Le jeudi 19 septembre 2019 à 00:49 +0530, rahul patwari a écrit :
> Hi,
> I would love to join the call.
> Can you also share the meeting invitation with me?
>
> Thanks,
> Rahul
> On Wed 18 Sep, 2019, 11:48 PM Xinyu Liu, <xi...@gmail.com> wrote:
> > Alexey and Etienne: I'm very happy to join the sync-up meeting. Please forward the meeting info to me. I am based in
> > California, US and hopefully the time will work :).
> > Thanks,
> > Xinyu
> > On Wed, Sep 18, 2019 at 6:39 AM Etienne Chauchot <ec...@apache.org> wrote:
> > > Hi Xinyu,
> > > Thanks for offering help ! My comments are inline:
> > > Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit :
> > > > Hi, Etienne,
> > > > The slides are very informative! Thanks for sharing the details about how the Beam API are mapped into Spark
> > > > Structural Streaming.
> > >
> > > Thanks !
> > > > We (LinkedIn) are also interested in trying the new SparkRunner to run Beam pipeine in batch, and contribute to
> > > > it too. From my understanding, seems the functionality on batch side is mostly complete and covers quite a large
> > > > percentage of the tests (a few missing pieces like state and timer in ParDo and SDF).
> > >
> > > Correct, it passes 89% of the tests, but there is more than SDF, state and timer missing, there is also ongoing
> > > encoders work that I would like to commit/push before merging.
> > > > If so, is it possible to merge the new runner sooner into master so it's much easier for us to pull it in (we
> > > > have an internal fork) and contribute back?
> > >
> > > Sure, see my other mail on this thread. As Alexey mentioned, please join the sync meeting we have, the more the
> > > merrier !
> > > > Also curious about the scheme part in the runner. Seems we can leverage the schema-aware work in PCollection and
> > > > translate from Beam schema to Spark, so it can be optimized in the planner layer. It will be great to hear back
> > > > your plans on that.
> > >
> > > Well, it is not designed yet but, if you remember my talk, we need to store beam windowing information with the
> > > data itself, so ending up having a dataset<WindowedValue> . One lead that was discussed is to store it as a Spark
> > > schema such as this:
> > > 1. field1: binary data for beam windowing information (cannot be mapped to fields because beam windowing info is
> > > complex structure)
> > > 2. fields of data as defined in the Beam schema if there is one
> > >
> > > > Congrats on this great work!
> > > Thanks !
> > > Best,
> > > Etienne
> > > > Thanks,
> > > > Xinyu
> > > > On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
> > > > > Hello Etienne,
> > > > > Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-
> > > > > aggregations in its streaming mode but design is ongoing. Do you have a link or something else to their design
> > > > > discussion/doc?
> > > > >
> > > > >
> > > > > -Rui
> > > > > On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org> wrote:
> > > > > > Hi Rahul,Sure, and great ! Thanks for proposing !If you want details, here is the presentation I did 30 mins
> > > > > > ago at the apachecon. You will find the video on youtube shortly but in the meantime, here is my
> > > > > > presentation slides.
> > > > > > And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
> > > > > > https://github.com/apache/beam/tree/spark-runner_structured-streaming
> > > > > > BestEtienne
> > > > > > Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
> > > > > > > Hi Etienne,
> > > > > > >
> > > > > > > I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works
> > > > > > > in Progress.
> > > > > > > I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
> > > > > > >
> > > > > > > I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming
> > > > > > > Spark Runner.
> > > > > > >
> > > > > > > Can you please point me in the right direction?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Rahul
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by rahul patwari <ra...@gmail.com>.
Hi,
I would love to join the call.
Can you also share the meeting invitation with me?
Thanks,
Rahul
On Wed 18 Sep, 2019, 11:48 PM Xinyu Liu, <xi...@gmail.com> wrote:
> Alexey and Etienne: I'm very happy to join the sync-up meeting. Please
> forward the meeting info to me. I am based in California, US and hopefully
> the time will work :).
>
> Thanks,
> Xinyu
>
> On Wed, Sep 18, 2019 at 6:39 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
>> Hi Xinyu,
>>
>> Thanks for offering help ! My comments are inline:
>>
>> Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit :
>>
>> Hi, Etienne,
>>
>> The slides are very informative! Thanks for sharing the details about how
>> the Beam API are mapped into Spark Structural Streaming.
>>
>>
>> Thanks !
>>
>> We (LinkedIn) are also interested in trying the new SparkRunner to run
>> Beam pipeine in batch, and contribute to it too. From my understanding,
>> seems the functionality on batch side is mostly complete and covers quite a
>> large percentage of the tests (a few missing pieces like state and timer in
>> ParDo and SDF).
>>
>>
>> Correct, it passes 89% of the tests, but there is more than SDF, state
>> and timer missing, there is also ongoing encoders work that I would like to
>> commit/push before merging.
>>
>> If so, is it possible to merge the new runner sooner into master so it's
>> much easier for us to pull it in (we have an internal fork) and contribute
>> back?
>>
>>
>> Sure, see my other mail on this thread. As Alexey mentioned, please join
>> the sync meeting we have, the more the merrier !
>>
>>
>> Also curious about the scheme part in the runner. Seems we can leverage
>> the schema-aware work in PCollection and translate from Beam schema to
>> Spark, so it can be optimized in the planner layer. It will be great to
>> hear back your plans on that.
>>
>>
>> Well, it is not designed yet but, if you remember my talk, we need to
>> store beam windowing information with the data itself, so ending up having
>> a dataset<WindowedValue> . One lead that was discussed is to store it as a
>> Spark schema such as this:
>>
>> 1. field1: binary data for beam windowing information (cannot be mapped
>> to fields because beam windowing info is complex structure)
>>
>> 2. fields of data as defined in the Beam schema if there is one
>>
>>
>> Congrats on this great work!
>>
>> Thanks !
>>
>> Best,
>>
>> Etienne
>>
>> Thanks,
>> Xinyu
>>
>> On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
>>
>> Hello Etienne,
>>
>> Your slide mentioned that streaming mode development is blocked because
>> Spark lacks supporting multiple-aggregations in its streaming mode but
>> design is ongoing. Do you have a link or something else to their design
>> discussion/doc?
>>
>>
>> -Rui
>>
>> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org>
>> wrote:
>>
>> Hi Rahul,
>> Sure, and great ! Thanks for proposing !
>> If you want details, here is the presentation I did 30 mins ago at the
>> apachecon. You will find the video on youtube shortly but in the meantime,
>> here is my presentation slides.
>>
>> And here is the structured streaming branch. I'll be happy to review your
>> PRs, thanks !
>>
>> <https://github.com/apache/beam/tree/spark-runner_structured-streaming>
>> https://github.com/apache/beam/tree/spark-runner_structured-streaming
>>
>> Best
>> Etienne
>>
>> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>>
>> Hi Etienne,
>>
>> I came to know about the work going on in Structured Streaming Spark
>> Runner from Apache Beam Wiki - Works in Progress.
>> I have contributed to BeamSql earlier. And I am working on supporting
>> PCollectionView in BeamSql.
>>
>> I would love to understand the Runner's side of Apache Beam and
>> contribute to the Structured Streaming Spark Runner.
>>
>> Can you please point me in the right direction?
>>
>> Thanks,
>> Rahul
>>
>>
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Xinyu Liu <xi...@gmail.com>.
Alexey and Etienne: I'm very happy to join the sync-up meeting. Please
forward the meeting info to me. I am based in California, US and hopefully
the time will work :).
Thanks,
Xinyu
On Wed, Sep 18, 2019 at 6:39 AM Etienne Chauchot <ec...@apache.org>
wrote:
> Hi Xinyu,
>
> Thanks for offering help ! My comments are inline:
>
> Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit :
>
> Hi, Etienne,
>
> The slides are very informative! Thanks for sharing the details about how
> the Beam API are mapped into Spark Structural Streaming.
>
>
> Thanks !
>
> We (LinkedIn) are also interested in trying the new SparkRunner to run
> Beam pipeine in batch, and contribute to it too. From my understanding,
> seems the functionality on batch side is mostly complete and covers quite a
> large percentage of the tests (a few missing pieces like state and timer in
> ParDo and SDF).
>
>
> Correct, it passes 89% of the tests, but there is more than SDF, state and
> timer missing, there is also ongoing encoders work that I would like to
> commit/push before merging.
>
> If so, is it possible to merge the new runner sooner into master so it's
> much easier for us to pull it in (we have an internal fork) and contribute
> back?
>
>
> Sure, see my other mail on this thread. As Alexey mentioned, please join
> the sync meeting we have, the more the merrier !
>
>
> Also curious about the scheme part in the runner. Seems we can leverage
> the schema-aware work in PCollection and translate from Beam schema to
> Spark, so it can be optimized in the planner layer. It will be great to
> hear back your plans on that.
>
>
> Well, it is not designed yet but, if you remember my talk, we need to
> store beam windowing information with the data itself, so ending up having
> a dataset<WindowedValue> . One lead that was discussed is to store it as a
> Spark schema such as this:
>
> 1. field1: binary data for beam windowing information (cannot be mapped to
> fields because beam windowing info is complex structure)
>
> 2. fields of data as defined in the Beam schema if there is one
>
>
> Congrats on this great work!
>
> Thanks !
>
> Best,
>
> Etienne
>
> Thanks,
> Xinyu
>
> On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
>
> Hello Etienne,
>
> Your slide mentioned that streaming mode development is blocked because
> Spark lacks supporting multiple-aggregations in its streaming mode but
> design is ongoing. Do you have a link or something else to their design
> discussion/doc?
>
>
> -Rui
>
> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org>
> wrote:
>
> Hi Rahul,
> Sure, and great ! Thanks for proposing !
> If you want details, here is the presentation I did 30 mins ago at the
> apachecon. You will find the video on youtube shortly but in the meantime,
> here is my presentation slides.
>
> And here is the structured streaming branch. I'll be happy to review your
> PRs, thanks !
>
> <https://github.com/apache/beam/tree/spark-runner_structured-streaming>
> https://github.com/apache/beam/tree/spark-runner_structured-streaming
>
> Best
> Etienne
>
> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>
> Hi Etienne,
>
> I came to know about the work going on in Structured Streaming Spark
> Runner from Apache Beam Wiki - Works in Progress.
> I have contributed to BeamSql earlier. And I am working on supporting
> PCollectionView in BeamSql.
>
> I would love to understand the Runner's side of Apache Beam and
> contribute to the Structured Streaming Spark Runner.
>
> Can you please point me in the right direction?
>
> Thanks,
> Rahul
>
>
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Etienne Chauchot <ec...@apache.org>.
Hi Xinyu,
Thanks for offering help ! My comments are inline:
Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit :
> Hi, Etienne,
> The slides are very informative! Thanks for sharing the details about how the Beam API are mapped into Spark
> Structural Streaming.
Thanks !
> We (LinkedIn) are also interested in trying the new SparkRunner to run Beam pipeine in batch, and contribute to it
> too. From my understanding, seems the functionality on batch side is mostly complete and covers quite a large
> percentage of the tests (a few missing pieces like state and timer in ParDo and SDF).
Correct, it passes 89% of the tests, but there is more than SDF, state and timer missing, there is also ongoing encoders
work that I would like to commit/push before merging.
> If so, is it possible to merge the new runner sooner into master so it's much easier for us to pull it in (we have an
> internal fork) and contribute back?
Sure, see my other mail on this thread. As Alexey mentioned, please join the sync meeting we have, the more the merrier
!
> Also curious about the scheme part in the runner. Seems we can leverage the schema-aware work in PCollection and
> translate from Beam schema to Spark, so it can be optimized in the planner layer. It will be great to hear back your
> plans on that.
Well, it is not designed yet but, if you remember my talk, we need to store beam windowing information with the data
itself, so ending up having a dataset<WindowedValue> . One lead that was discussed is to store it as a Spark schema such
as this:
1. field1: binary data for beam windowing information (cannot be mapped to fields because beam windowing info is
complex structure)
2. fields of data as defined in the Beam schema if there is one
> Congrats on this great work!
Thanks !
Best,
Etienne
> Thanks,
> Xinyu
> On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
> > Hello Etienne,
> > Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-aggregations
> > in its streaming mode but design is ongoing. Do you have a link or something else to their design discussion/doc?
> >
> >
> > -Rui
> > On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org> wrote:
> > > Hi Rahul,Sure, and great ! Thanks for proposing !If you want details, here is the presentation I did 30 mins ago
> > > at the apachecon. You will find the video on youtube shortly but in the meantime, here is my presentation slides.
> > > And here is the structured streaming branch. I'll be happy to review your PRs, thanks !
> > > https://github.com/apache/beam/tree/spark-runner_structured-streaming
> > > BestEtienne
> > > Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
> > > > Hi Etienne,
> > > >
> > > > I came to know about the work going on in Structured Streaming Spark Runner from Apache Beam Wiki - Works in
> > > > Progress.
> > > > I have contributed to BeamSql earlier. And I am working on supporting PCollectionView in BeamSql.
> > > >
> > > > I would love to understand the Runner's side of Apache Beam and contribute to the Structured Streaming Spark
> > > > Runner.
> > > >
> > > > Can you please point me in the right direction?
> > > >
> > > > Thanks,
> > > > Rahul
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Xinyu Liu <xi...@gmail.com>.
Hi, Etienne,
The slides are very informative! Thanks for sharing the details about how
the Beam API are mapped into Spark Structural Streaming. We (LinkedIn) are
also interested in trying the new SparkRunner to run Beam pipeine in batch,
and contribute to it too. From my understanding, seems the functionality on
batch side is mostly complete and covers quite a large percentage of the
tests (a few missing pieces like state and timer in ParDo and SDF). If so,
is it possible to merge the new runner sooner into master so it's much
easier for us to pull it in (we have an internal fork) and contribute back?
Also curious about the scheme part in the runner. Seems we can leverage the
schema-aware work in PCollection and translate from Beam schema to Spark,
so it can be optimized in the planner layer. It will be great to hear back
your plans on that.
Congrats on this great work!
Thanks,
Xinyu
On Wed, Sep 11, 2019 at 6:02 PM Rui Wang <ru...@google.com> wrote:
> Hello Etienne,
>
> Your slide mentioned that streaming mode development is blocked because
> Spark lacks supporting multiple-aggregations in its streaming mode but
> design is ongoing. Do you have a link or something else to their design
> discussion/doc?
>
>
> -Rui
>
> On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org>
> wrote:
>
>> Hi Rahul,
>> Sure, and great ! Thanks for proposing !
>> If you want details, here is the presentation I did 30 mins ago at the
>> apachecon. You will find the video on youtube shortly but in the meantime,
>> here is my presentation slides.
>>
>> And here is the structured streaming branch. I'll be happy to review your
>> PRs, thanks !
>>
>> <https://github.com/apache/beam/tree/spark-runner_structured-streaming>
>> https://github.com/apache/beam/tree/spark-runner_structured-streaming
>>
>> Best
>> Etienne
>>
>> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>>
>> Hi Etienne,
>>
>> I came to know about the work going on in Structured Streaming Spark
>> Runner from Apache Beam Wiki - Works in Progress.
>> I have contributed to BeamSql earlier. And I am working on supporting
>> PCollectionView in BeamSql.
>>
>> I would love to understand the Runner's side of Apache Beam and
>> contribute to the Structured Streaming Spark Runner.
>>
>> Can you please point me in the right direction?
>>
>> Thanks,
>> Rahul
>>
>>
Re: Pointers on Contributing to Structured Streaming Spark Runner
Posted by Rui Wang <ru...@google.com>.
Hello Etienne,
Your slide mentioned that streaming mode development is blocked because
Spark lacks supporting multiple-aggregations in its streaming mode but
design is ongoing. Do you have a link or something else to their design
discussion/doc?
-Rui
On Wed, Sep 11, 2019 at 5:10 PM Etienne Chauchot <ec...@apache.org>
wrote:
> Hi Rahul,
> Sure, and great ! Thanks for proposing !
> If you want details, here is the presentation I did 30 mins ago at the
> apachecon. You will find the video on youtube shortly but in the meantime,
> here is my presentation slides.
>
> And here is the structured streaming branch. I'll be happy to review your
> PRs, thanks !
>
> <https://github.com/apache/beam/tree/spark-runner_structured-streaming>
> https://github.com/apache/beam/tree/spark-runner_structured-streaming
>
> Best
> Etienne
>
> Le mercredi 11 septembre 2019 à 16:37 +0530, rahul patwari a écrit :
>
> Hi Etienne,
>
> I came to know about the work going on in Structured Streaming Spark
> Runner from Apache Beam Wiki - Works in Progress.
> I have contributed to BeamSql earlier. And I am working on supporting
> PCollectionView in BeamSql.
>
> I would love to understand the Runner's side of Apache Beam and
> contribute to the Structured Streaming Spark Runner.
>
> Can you please point me in the right direction?
>
> Thanks,
> Rahul
>
>