You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Xinyu Liu <xi...@gmail.com> on 2018/10/10 17:22:36 UTC
Beam Samza Runner status update
Hi, All,
It's been over four months since we added the Samza Runner to Beam, and
we've been making a lot of progress after that. Here I would like to update
your guys and share some really good news happening here at LinkedIn:
1) First Beam job in production @LInkedIn!
After a few rounds of testing and benchmarking, we finally rolled out our
first Beam job here! The job uses quite a few features, such as event time,
fixed/session windowing, early triggering, and stateful processing. Our
first customer is very happy and they highly appraise the easy-to-use Beam
API as well as powerful processing model. Due to the limited resources
here, we put our full trust in the work you guys are doing, and we didn't
run into any surprises. We see extremely attention to details as well as
non-compromise in any user experience everywhere in the code base. We would
like to thank everyone in the Beam community to contribute to such an
amazing framework!
2) A portable Samza Runner prototype
We are also starting the work in making Samza Runner portable. So far we
just got the python word count example working using portable Samza Runner.
Please look out for the PR for this very soon :). Again, this work is not
possible without the great Beam portability framework, and the developers
like Luke and Ahmet, just to name a few, behind it. The ReferenceRunner has
been extremely useful to us to figure out what's needed and how it works.
Kudos to Thomas Groh, Ben Sidhom and all the others who makes this
available to us. And to Kenn, your fuse work rocks.
3) More contributors in Samza Runner
The runner has been Chris and my personal project for a while and now it's
not the case. We got Hai Lu and Boris Shkolnik from Samza team to
contribute. Hai has been focusing on the portability work as mentioned in
#2, and Boris will work mostly on supporting our use cases. We will send
more emails discussing our use cases, like the "Update state after firing"
email I sent out earlier.
Finally, a shout-out to our very own Chris Pettitt. Without you, none of
the above won't happen!
Thanks,
Xinyu
Re: Beam Samza Runner status update
Posted by Lukasz Cwik <lc...@google.com>.
I'm excited to hear about making Samza capable of running non-Java
pipelines.
On Fri, Oct 12, 2018 at 9:46 AM Xinyu Liu <xi...@gmail.com> wrote:
> @Max: absolutely we should work together! FlinkRunner has been our best
> reference since the start of our SamzaRunner, and the previous work in
> Flink portable runner has been extremely valuable to us too. We haven't got
> to the point of portable stateful processing yet. Our next step is to hook
> up a streaming source, i.e. Kafka, and test out streaming capabilities such
> as watermarks, windowing and triggers. For us, reading from Kafka will
> happen on the Java side (we have quite a lot of extensions of Kafka in
> LinkedIn), so we will try to create some internal Python API and do the
> translation right now. On the other hand, we are following up with
> BEAM-2937 for the optimization of Combine in portable runner. Previously we
> run into some state problem without this being resolved. Anyway, look
> forward to syncing up with you more!
>
> Thanks,
> Xinyu
>
>
>
> On Fri, Oct 12, 2018 at 1:40 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> Thanks for the updating, Xinyu and Hai! Great to see another Running
>> emerging :)
>>
>> I'm on the FlinkRunner. Looking forward to working together with you to
>> make the Beam Runners even better. Particularly, we should sync on the
>> portability, as some things are still to be fleshed out. In Flink, we
>> are starting to integrate portable State.
>>
>> Best,
>> Max
>>
>> On 11.10.18 05:14, Jesse Anderson wrote:
>> > Interesting
>> >
>> > On Wed, Oct 10, 2018, 3:49 PM Kenneth Knowles <kenn@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > Welcome, Hai!
>> >
>> > On Wed, Oct 10, 2018 at 3:46 PM Hai Lu <lhaiesp@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> > Hi, all
>> >
>> > This is Hai from LinkedIn. As Xinyu mentioned, I have been
>> > working on portable API for Samza runner and made some solid
>> > progress. It's been a very smooth process (although not
>> > effortless for sure) and I'm really grateful for the great
>> > platform that you all have built. I'm very impressed. Bravo!
>> >
>> > Excited to work with everyone on Beam. Do expect more questions
>> > from me down the road.
>> >
>> > Thanks,
>> > Hai
>> >
>> > On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles
>> > <kenn@apache.org <ma...@apache.org>> wrote:
>> >
>> > Clarification: Thomas Groh wrote the fuser, not me!
>> >
>> > Thanks for the sharing all this. Really cool.
>> >
>> > Kenn
>> >
>> > On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <
>> ruwang@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > Thanks for sharing! it's so exciting to hear that Beam
>> > is being used on Samza in production @LinkedIn! Your
>> > feedback will be helpful to Beam community!
>> >
>> > Besides, Beam supports SQL right now and hopefully Beam
>> > community could also receive feedback on BeamSQL
>> > <
>> https://beam.apache.org/documentation/dsls/sql/overview/> in
>> > the future.
>> >
>> > -Rui
>> >
>> > On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré
>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>> >
>> > Thanks for sharing and congrats for this great work
>> !
>> >
>> > Regards
>> > JB
>> > Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us
>> > <mailto:%3Ca>@gmail.com <http://gmail.com>
>> > target=_blank>xinyuliu.us
>> > <http://xinyuliu.us>@gmail.com <http://gmail.com>>
>> a
>> > écrit:
>> >
>> > Hi, All,
>> >
>> > It's been over four months since we added the
>> > Samza Runner to Beam, and we've been making a
>> > lot of progress after that. Here I would like to
>> > update your guys and share some really good news
>> > happening here at LinkedIn:
>> >
>> > 1) First Beam job in production @LInkedIn!
>> > After a few rounds of testing and benchmarking,
>> > we finally rolled out our first Beam job here!
>> > The job uses quite a few features, such as event
>> > time, fixed/session windowing, early triggering,
>> > and stateful processing. Our first customer is
>> > very happy and they highly appraise the
>> > easy-to-use Beam API as well as powerful
>> > processing model. Due to the limited resources
>> > here, we put our full trust in the work you guys
>> > are doing, and we didn't run into any surprises.
>> > We see extremely attention to details as well as
>> > non-compromise in any user experience everywhere
>> > in the code base. We would like to thank
>> > everyone in the Beam community to contribute to
>> > such an amazing framework!
>> >
>> > 2) A portable Samza Runner prototype
>> > We are also starting the work in making Samza
>> > Runner portable. So far we just got the python
>> > word count example working using portable Samza
>> > Runner. Please look out for the PR for this very
>> > soon :). Again, this work is not possible
>> > without the great Beam portability framework,
>> > and the developers like Luke and Ahmet, just to
>> > name a few, behind it. The ReferenceRunner has
>> > been extremely useful to us to figure out what's
>> > needed and how it works. Kudos to Thomas Groh,
>> > Ben Sidhom and all the others who makes this
>> > available to us. And to Kenn, your fuse work
>> rocks.
>> >
>> > 3) More contributors in Samza Runner
>> > The runner has been Chris and my personal
>> > project for a while and now it's not the case.
>> > We got Hai Lu and Boris Shkolnik from Samza team
>> > to contribute. Hai has been focusing on the
>> > portability work as mentioned in #2, and Boris
>> > will work mostly on supporting our use cases. We
>> > will send more emails discussing our use cases,
>> > like the "Update state after firing" email I
>> > sent out earlier.
>> >
>> > Finally, a shout-out to our very own Chris
>> > Pettitt. Without you, none of the above won't
>> > happen!
>> >
>> > Thanks,
>> > Xinyu
>> >
>>
>
Re: Beam Samza Runner status update
Posted by Xinyu Liu <xi...@gmail.com>.
@Max: absolutely we should work together! FlinkRunner has been our best
reference since the start of our SamzaRunner, and the previous work in
Flink portable runner has been extremely valuable to us too. We haven't got
to the point of portable stateful processing yet. Our next step is to hook
up a streaming source, i.e. Kafka, and test out streaming capabilities such
as watermarks, windowing and triggers. For us, reading from Kafka will
happen on the Java side (we have quite a lot of extensions of Kafka in
LinkedIn), so we will try to create some internal Python API and do the
translation right now. On the other hand, we are following up with
BEAM-2937 for the optimization of Combine in portable runner. Previously we
run into some state problem without this being resolved. Anyway, look
forward to syncing up with you more!
Thanks,
Xinyu
On Fri, Oct 12, 2018 at 1:40 AM Maximilian Michels <mx...@apache.org> wrote:
> Thanks for the updating, Xinyu and Hai! Great to see another Running
> emerging :)
>
> I'm on the FlinkRunner. Looking forward to working together with you to
> make the Beam Runners even better. Particularly, we should sync on the
> portability, as some things are still to be fleshed out. In Flink, we
> are starting to integrate portable State.
>
> Best,
> Max
>
> On 11.10.18 05:14, Jesse Anderson wrote:
> > Interesting
> >
> > On Wed, Oct 10, 2018, 3:49 PM Kenneth Knowles <kenn@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Welcome, Hai!
> >
> > On Wed, Oct 10, 2018 at 3:46 PM Hai Lu <lhaiesp@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Hi, all
> >
> > This is Hai from LinkedIn. As Xinyu mentioned, I have been
> > working on portable API for Samza runner and made some solid
> > progress. It's been a very smooth process (although not
> > effortless for sure) and I'm really grateful for the great
> > platform that you all have built. I'm very impressed. Bravo!
> >
> > Excited to work with everyone on Beam. Do expect more questions
> > from me down the road.
> >
> > Thanks,
> > Hai
> >
> > On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles
> > <kenn@apache.org <ma...@apache.org>> wrote:
> >
> > Clarification: Thomas Groh wrote the fuser, not me!
> >
> > Thanks for the sharing all this. Really cool.
> >
> > Kenn
> >
> > On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <ruwang@google.com
> > <ma...@google.com>> wrote:
> >
> > Thanks for sharing! it's so exciting to hear that Beam
> > is being used on Samza in production @LinkedIn! Your
> > feedback will be helpful to Beam community!
> >
> > Besides, Beam supports SQL right now and hopefully Beam
> > community could also receive feedback on BeamSQL
> > <
> https://beam.apache.org/documentation/dsls/sql/overview/> in
> > the future.
> >
> > -Rui
> >
> > On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > Thanks for sharing and congrats for this great work !
> >
> > Regards
> > JB
> > Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us
> > <mailto:%3Ca>@gmail.com <http://gmail.com>
> > target=_blank>xinyuliu.us
> > <http://xinyuliu.us>@gmail.com <http://gmail.com>> a
> > écrit:
> >
> > Hi, All,
> >
> > It's been over four months since we added the
> > Samza Runner to Beam, and we've been making a
> > lot of progress after that. Here I would like to
> > update your guys and share some really good news
> > happening here at LinkedIn:
> >
> > 1) First Beam job in production @LInkedIn!
> > After a few rounds of testing and benchmarking,
> > we finally rolled out our first Beam job here!
> > The job uses quite a few features, such as event
> > time, fixed/session windowing, early triggering,
> > and stateful processing. Our first customer is
> > very happy and they highly appraise the
> > easy-to-use Beam API as well as powerful
> > processing model. Due to the limited resources
> > here, we put our full trust in the work you guys
> > are doing, and we didn't run into any surprises.
> > We see extremely attention to details as well as
> > non-compromise in any user experience everywhere
> > in the code base. We would like to thank
> > everyone in the Beam community to contribute to
> > such an amazing framework!
> >
> > 2) A portable Samza Runner prototype
> > We are also starting the work in making Samza
> > Runner portable. So far we just got the python
> > word count example working using portable Samza
> > Runner. Please look out for the PR for this very
> > soon :). Again, this work is not possible
> > without the great Beam portability framework,
> > and the developers like Luke and Ahmet, just to
> > name a few, behind it. The ReferenceRunner has
> > been extremely useful to us to figure out what's
> > needed and how it works. Kudos to Thomas Groh,
> > Ben Sidhom and all the others who makes this
> > available to us. And to Kenn, your fuse work
> rocks.
> >
> > 3) More contributors in Samza Runner
> > The runner has been Chris and my personal
> > project for a while and now it's not the case.
> > We got Hai Lu and Boris Shkolnik from Samza team
> > to contribute. Hai has been focusing on the
> > portability work as mentioned in #2, and Boris
> > will work mostly on supporting our use cases. We
> > will send more emails discussing our use cases,
> > like the "Update state after firing" email I
> > sent out earlier.
> >
> > Finally, a shout-out to our very own Chris
> > Pettitt. Without you, none of the above won't
> > happen!
> >
> > Thanks,
> > Xinyu
> >
>
Re: Beam Samza Runner status update
Posted by Maximilian Michels <mx...@apache.org>.
Thanks for the updating, Xinyu and Hai! Great to see another Running
emerging :)
I'm on the FlinkRunner. Looking forward to working together with you to
make the Beam Runners even better. Particularly, we should sync on the
portability, as some things are still to be fleshed out. In Flink, we
are starting to integrate portable State.
Best,
Max
On 11.10.18 05:14, Jesse Anderson wrote:
> Interesting
>
> On Wed, Oct 10, 2018, 3:49 PM Kenneth Knowles <kenn@apache.org
> <ma...@apache.org>> wrote:
>
> Welcome, Hai!
>
> On Wed, Oct 10, 2018 at 3:46 PM Hai Lu <lhaiesp@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi, all
>
> This is Hai from LinkedIn. As Xinyu mentioned, I have been
> working on portable API for Samza runner and made some solid
> progress. It's been a very smooth process (although not
> effortless for sure) and I'm really grateful for the great
> platform that you all have built. I'm very impressed. Bravo!
>
> Excited to work with everyone on Beam. Do expect more questions
> from me down the road.
>
> Thanks,
> Hai
>
> On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles
> <kenn@apache.org <ma...@apache.org>> wrote:
>
> Clarification: Thomas Groh wrote the fuser, not me!
>
> Thanks for the sharing all this. Really cool.
>
> Kenn
>
> On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <ruwang@google.com
> <ma...@google.com>> wrote:
>
> Thanks for sharing! it's so exciting to hear that Beam
> is being used on Samza in production @LinkedIn! Your
> feedback will be helpful to Beam community!
>
> Besides, Beam supports SQL right now and hopefully Beam
> community could also receive feedback on BeamSQL
> <https://beam.apache.org/documentation/dsls/sql/overview/> in
> the future.
>
> -Rui
>
> On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>
> Thanks for sharing and congrats for this great work !
>
> Regards
> JB
> Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us
> <mailto:%3Ca>@gmail.com <http://gmail.com>
> target=_blank>xinyuliu.us
> <http://xinyuliu.us>@gmail.com <http://gmail.com>> a
> écrit:
>
> Hi, All,
>
> It's been over four months since we added the
> Samza Runner to Beam, and we've been making a
> lot of progress after that. Here I would like to
> update your guys and share some really good news
> happening here at LinkedIn:
>
> 1) First Beam job in production @LInkedIn!
> After a few rounds of testing and benchmarking,
> we finally rolled out our first Beam job here!
> The job uses quite a few features, such as event
> time, fixed/session windowing, early triggering,
> and stateful processing. Our first customer is
> very happy and they highly appraise the
> easy-to-use Beam API as well as powerful
> processing model. Due to the limited resources
> here, we put our full trust in the work you guys
> are doing, and we didn't run into any surprises.
> We see extremely attention to details as well as
> non-compromise in any user experience everywhere
> in the code base. We would like to thank
> everyone in the Beam community to contribute to
> such an amazing framework!
>
> 2) A portable Samza Runner prototype
> We are also starting the work in making Samza
> Runner portable. So far we just got the python
> word count example working using portable Samza
> Runner. Please look out for the PR for this very
> soon :). Again, this work is not possible
> without the great Beam portability framework,
> and the developers like Luke and Ahmet, just to
> name a few, behind it. The ReferenceRunner has
> been extremely useful to us to figure out what's
> needed and how it works. Kudos to Thomas Groh,
> Ben Sidhom and all the others who makes this
> available to us. And to Kenn, your fuse work rocks.
>
> 3) More contributors in Samza Runner
> The runner has been Chris and my personal
> project for a while and now it's not the case.
> We got Hai Lu and Boris Shkolnik from Samza team
> to contribute. Hai has been focusing on the
> portability work as mentioned in #2, and Boris
> will work mostly on supporting our use cases. We
> will send more emails discussing our use cases,
> like the "Update state after firing" email I
> sent out earlier.
>
> Finally, a shout-out to our very own Chris
> Pettitt. Without you, none of the above won't
> happen!
>
> Thanks,
> Xinyu
>
Re: Beam Samza Runner status update
Posted by Jesse Anderson <je...@bigdatainstitute.io>.
Interesting
On Wed, Oct 10, 2018, 3:49 PM Kenneth Knowles <ke...@apache.org> wrote:
> Welcome, Hai!
>
> On Wed, Oct 10, 2018 at 3:46 PM Hai Lu <lh...@gmail.com> wrote:
>
>> Hi, all
>>
>> This is Hai from LinkedIn. As Xinyu mentioned, I have been working on
>> portable API for Samza runner and made some solid progress. It's been a
>> very smooth process (although not effortless for sure) and I'm really
>> grateful for the great platform that you all have built. I'm very
>> impressed. Bravo!
>>
>> Excited to work with everyone on Beam. Do expect more questions from me
>> down the road.
>>
>> Thanks,
>> Hai
>>
>> On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> Clarification: Thomas Groh wrote the fuser, not me!
>>>
>>> Thanks for the sharing all this. Really cool.
>>>
>>> Kenn
>>>
>>> On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <ru...@google.com> wrote:
>>>
>>>> Thanks for sharing! it's so exciting to hear that Beam is being used on
>>>> Samza in production @LinkedIn! Your feedback will be helpful to Beam
>>>> community!
>>>>
>>>> Besides, Beam supports SQL right now and hopefully Beam community could
>>>> also receive feedback on BeamSQL
>>>> <https://beam.apache.org/documentation/dsls/sql/overview/> in the
>>>> future.
>>>>
>>>> -Rui
>>>>
>>>> On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> Thanks for sharing and congrats for this great work !
>>>>>
>>>>> Regards
>>>>> JB
>>>>> Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us <%3Ca>@gmail.com
>>>>> target=_blank>xinyuliu.us@gmail.com> a écrit:
>>>>>>
>>>>>> Hi, All,
>>>>>>
>>>>>> It's been over four months since we added the Samza Runner to Beam,
>>>>>> and we've been making a lot of progress after that. Here I would like to
>>>>>> update your guys and share some really good news happening here at LinkedIn:
>>>>>>
>>>>>> 1) First Beam job in production @LInkedIn!
>>>>>> After a few rounds of testing and benchmarking, we finally rolled out
>>>>>> our first Beam job here! The job uses quite a few features, such as event
>>>>>> time, fixed/session windowing, early triggering, and stateful processing.
>>>>>> Our first customer is very happy and they highly appraise the easy-to-use
>>>>>> Beam API as well as powerful processing model. Due to the limited resources
>>>>>> here, we put our full trust in the work you guys are doing, and we didn't
>>>>>> run into any surprises. We see extremely attention to details as well as
>>>>>> non-compromise in any user experience everywhere in the code base. We would
>>>>>> like to thank everyone in the Beam community to contribute to such an
>>>>>> amazing framework!
>>>>>>
>>>>>> 2) A portable Samza Runner prototype
>>>>>> We are also starting the work in making Samza Runner portable. So far
>>>>>> we just got the python word count example working using portable Samza
>>>>>> Runner. Please look out for the PR for this very soon :). Again, this work
>>>>>> is not possible without the great Beam portability framework, and the
>>>>>> developers like Luke and Ahmet, just to name a few, behind it. The
>>>>>> ReferenceRunner has been extremely useful to us to figure out what's needed
>>>>>> and how it works. Kudos to Thomas Groh, Ben Sidhom and all the others who
>>>>>> makes this available to us. And to Kenn, your fuse work rocks.
>>>>>>
>>>>>> 3) More contributors in Samza Runner
>>>>>> The runner has been Chris and my personal project for a while and now
>>>>>> it's not the case. We got Hai Lu and Boris Shkolnik from Samza team to
>>>>>> contribute. Hai has been focusing on the portability work as mentioned in
>>>>>> #2, and Boris will work mostly on supporting our use cases. We will send
>>>>>> more emails discussing our use cases, like the "Update state after firing"
>>>>>> email I sent out earlier.
>>>>>>
>>>>>> Finally, a shout-out to our very own Chris Pettitt. Without you, none
>>>>>> of the above won't happen!
>>>>>>
>>>>>> Thanks,
>>>>>> Xinyu
>>>>>>
>>>>>
Re: Beam Samza Runner status update
Posted by Kenneth Knowles <ke...@apache.org>.
Welcome, Hai!
On Wed, Oct 10, 2018 at 3:46 PM Hai Lu <lh...@gmail.com> wrote:
> Hi, all
>
> This is Hai from LinkedIn. As Xinyu mentioned, I have been working on
> portable API for Samza runner and made some solid progress. It's been a
> very smooth process (although not effortless for sure) and I'm really
> grateful for the great platform that you all have built. I'm very
> impressed. Bravo!
>
> Excited to work with everyone on Beam. Do expect more questions from me
> down the road.
>
> Thanks,
> Hai
>
> On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Clarification: Thomas Groh wrote the fuser, not me!
>>
>> Thanks for the sharing all this. Really cool.
>>
>> Kenn
>>
>> On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <ru...@google.com> wrote:
>>
>>> Thanks for sharing! it's so exciting to hear that Beam is being used on
>>> Samza in production @LinkedIn! Your feedback will be helpful to Beam
>>> community!
>>>
>>> Besides, Beam supports SQL right now and hopefully Beam community could
>>> also receive feedback on BeamSQL
>>> <https://beam.apache.org/documentation/dsls/sql/overview/> in the
>>> future.
>>>
>>> -Rui
>>>
>>> On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Thanks for sharing and congrats for this great work !
>>>>
>>>> Regards
>>>> JB
>>>> Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us <%3Ca>@gmail.com
>>>> target=_blank>xinyuliu.us@gmail.com> a écrit:
>>>>>
>>>>> Hi, All,
>>>>>
>>>>> It's been over four months since we added the Samza Runner to Beam,
>>>>> and we've been making a lot of progress after that. Here I would like to
>>>>> update your guys and share some really good news happening here at LinkedIn:
>>>>>
>>>>> 1) First Beam job in production @LInkedIn!
>>>>> After a few rounds of testing and benchmarking, we finally rolled out
>>>>> our first Beam job here! The job uses quite a few features, such as event
>>>>> time, fixed/session windowing, early triggering, and stateful processing.
>>>>> Our first customer is very happy and they highly appraise the easy-to-use
>>>>> Beam API as well as powerful processing model. Due to the limited resources
>>>>> here, we put our full trust in the work you guys are doing, and we didn't
>>>>> run into any surprises. We see extremely attention to details as well as
>>>>> non-compromise in any user experience everywhere in the code base. We would
>>>>> like to thank everyone in the Beam community to contribute to such an
>>>>> amazing framework!
>>>>>
>>>>> 2) A portable Samza Runner prototype
>>>>> We are also starting the work in making Samza Runner portable. So far
>>>>> we just got the python word count example working using portable Samza
>>>>> Runner. Please look out for the PR for this very soon :). Again, this work
>>>>> is not possible without the great Beam portability framework, and the
>>>>> developers like Luke and Ahmet, just to name a few, behind it. The
>>>>> ReferenceRunner has been extremely useful to us to figure out what's needed
>>>>> and how it works. Kudos to Thomas Groh, Ben Sidhom and all the others who
>>>>> makes this available to us. And to Kenn, your fuse work rocks.
>>>>>
>>>>> 3) More contributors in Samza Runner
>>>>> The runner has been Chris and my personal project for a while and now
>>>>> it's not the case. We got Hai Lu and Boris Shkolnik from Samza team to
>>>>> contribute. Hai has been focusing on the portability work as mentioned in
>>>>> #2, and Boris will work mostly on supporting our use cases. We will send
>>>>> more emails discussing our use cases, like the "Update state after firing"
>>>>> email I sent out earlier.
>>>>>
>>>>> Finally, a shout-out to our very own Chris Pettitt. Without you, none
>>>>> of the above won't happen!
>>>>>
>>>>> Thanks,
>>>>> Xinyu
>>>>>
>>>>
Re: Beam Samza Runner status update
Posted by Hai Lu <lh...@gmail.com>.
Hi, all
This is Hai from LinkedIn. As Xinyu mentioned, I have been working on
portable API for Samza runner and made some solid progress. It's been a
very smooth process (although not effortless for sure) and I'm really
grateful for the great platform that you all have built. I'm very
impressed. Bravo!
Excited to work with everyone on Beam. Do expect more questions from me
down the road.
Thanks,
Hai
On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles <ke...@apache.org> wrote:
> Clarification: Thomas Groh wrote the fuser, not me!
>
> Thanks for the sharing all this. Really cool.
>
> Kenn
>
> On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <ru...@google.com> wrote:
>
>> Thanks for sharing! it's so exciting to hear that Beam is being used on
>> Samza in production @LinkedIn! Your feedback will be helpful to Beam
>> community!
>>
>> Besides, Beam supports SQL right now and hopefully Beam community could
>> also receive feedback on BeamSQL
>> <https://beam.apache.org/documentation/dsls/sql/overview/> in the future.
>>
>> -Rui
>>
>> On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Thanks for sharing and congrats for this great work !
>>>
>>> Regards
>>> JB
>>> Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us <%3Ca>@gmail.com
>>> target=_blank>xinyuliu.us@gmail.com> a écrit:
>>>>
>>>> Hi, All,
>>>>
>>>> It's been over four months since we added the Samza Runner to Beam, and
>>>> we've been making a lot of progress after that. Here I would like to update
>>>> your guys and share some really good news happening here at LinkedIn:
>>>>
>>>> 1) First Beam job in production @LInkedIn!
>>>> After a few rounds of testing and benchmarking, we finally rolled out
>>>> our first Beam job here! The job uses quite a few features, such as event
>>>> time, fixed/session windowing, early triggering, and stateful processing.
>>>> Our first customer is very happy and they highly appraise the easy-to-use
>>>> Beam API as well as powerful processing model. Due to the limited resources
>>>> here, we put our full trust in the work you guys are doing, and we didn't
>>>> run into any surprises. We see extremely attention to details as well as
>>>> non-compromise in any user experience everywhere in the code base. We would
>>>> like to thank everyone in the Beam community to contribute to such an
>>>> amazing framework!
>>>>
>>>> 2) A portable Samza Runner prototype
>>>> We are also starting the work in making Samza Runner portable. So far
>>>> we just got the python word count example working using portable Samza
>>>> Runner. Please look out for the PR for this very soon :). Again, this work
>>>> is not possible without the great Beam portability framework, and the
>>>> developers like Luke and Ahmet, just to name a few, behind it. The
>>>> ReferenceRunner has been extremely useful to us to figure out what's needed
>>>> and how it works. Kudos to Thomas Groh, Ben Sidhom and all the others who
>>>> makes this available to us. And to Kenn, your fuse work rocks.
>>>>
>>>> 3) More contributors in Samza Runner
>>>> The runner has been Chris and my personal project for a while and now
>>>> it's not the case. We got Hai Lu and Boris Shkolnik from Samza team to
>>>> contribute. Hai has been focusing on the portability work as mentioned in
>>>> #2, and Boris will work mostly on supporting our use cases. We will send
>>>> more emails discussing our use cases, like the "Update state after firing"
>>>> email I sent out earlier.
>>>>
>>>> Finally, a shout-out to our very own Chris Pettitt. Without you, none
>>>> of the above won't happen!
>>>>
>>>> Thanks,
>>>> Xinyu
>>>>
>>>
Re: Beam Samza Runner status update
Posted by Kenneth Knowles <ke...@apache.org>.
Clarification: Thomas Groh wrote the fuser, not me!
Thanks for the sharing all this. Really cool.
Kenn
On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <ru...@google.com> wrote:
> Thanks for sharing! it's so exciting to hear that Beam is being used on
> Samza in production @LinkedIn! Your feedback will be helpful to Beam
> community!
>
> Besides, Beam supports SQL right now and hopefully Beam community could
> also receive feedback on BeamSQL
> <https://beam.apache.org/documentation/dsls/sql/overview/> in the future.
>
> -Rui
>
> On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Thanks for sharing and congrats for this great work !
>>
>> Regards
>> JB
>> Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us <%3Ca>@gmail.com
>> target=_blank>xinyuliu.us@gmail.com> a écrit:
>>>
>>> Hi, All,
>>>
>>> It's been over four months since we added the Samza Runner to Beam, and
>>> we've been making a lot of progress after that. Here I would like to update
>>> your guys and share some really good news happening here at LinkedIn:
>>>
>>> 1) First Beam job in production @LInkedIn!
>>> After a few rounds of testing and benchmarking, we finally rolled out
>>> our first Beam job here! The job uses quite a few features, such as event
>>> time, fixed/session windowing, early triggering, and stateful processing.
>>> Our first customer is very happy and they highly appraise the easy-to-use
>>> Beam API as well as powerful processing model. Due to the limited resources
>>> here, we put our full trust in the work you guys are doing, and we didn't
>>> run into any surprises. We see extremely attention to details as well as
>>> non-compromise in any user experience everywhere in the code base. We would
>>> like to thank everyone in the Beam community to contribute to such an
>>> amazing framework!
>>>
>>> 2) A portable Samza Runner prototype
>>> We are also starting the work in making Samza Runner portable. So far we
>>> just got the python word count example working using portable Samza Runner.
>>> Please look out for the PR for this very soon :). Again, this work is not
>>> possible without the great Beam portability framework, and the developers
>>> like Luke and Ahmet, just to name a few, behind it. The ReferenceRunner has
>>> been extremely useful to us to figure out what's needed and how it works.
>>> Kudos to Thomas Groh, Ben Sidhom and all the others who makes this
>>> available to us. And to Kenn, your fuse work rocks.
>>>
>>> 3) More contributors in Samza Runner
>>> The runner has been Chris and my personal project for a while and now
>>> it's not the case. We got Hai Lu and Boris Shkolnik from Samza team to
>>> contribute. Hai has been focusing on the portability work as mentioned in
>>> #2, and Boris will work mostly on supporting our use cases. We will send
>>> more emails discussing our use cases, like the "Update state after firing"
>>> email I sent out earlier.
>>>
>>> Finally, a shout-out to our very own Chris Pettitt. Without you, none of
>>> the above won't happen!
>>>
>>> Thanks,
>>> Xinyu
>>>
>>
Re: Beam Samza Runner status update
Posted by Rui Wang <ru...@google.com>.
Thanks for sharing! it's so exciting to hear that Beam is being used on
Samza in production @LinkedIn! Your feedback will be helpful to Beam
community!
Besides, Beam supports SQL right now and hopefully Beam community could
also receive feedback on BeamSQL
<https://beam.apache.org/documentation/dsls/sql/overview/> in the future.
-Rui
On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Thanks for sharing and congrats for this great work !
>
> Regards
> JB
> Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us <%3Ca>@gmail.com
> target=_blank>xinyuliu.us@gmail.com> a écrit:
>>
>> Hi, All,
>>
>> It's been over four months since we added the Samza Runner to Beam, and
>> we've been making a lot of progress after that. Here I would like to update
>> your guys and share some really good news happening here at LinkedIn:
>>
>> 1) First Beam job in production @LInkedIn!
>> After a few rounds of testing and benchmarking, we finally rolled out our
>> first Beam job here! The job uses quite a few features, such as event time,
>> fixed/session windowing, early triggering, and stateful processing. Our
>> first customer is very happy and they highly appraise the easy-to-use Beam
>> API as well as powerful processing model. Due to the limited resources
>> here, we put our full trust in the work you guys are doing, and we didn't
>> run into any surprises. We see extremely attention to details as well as
>> non-compromise in any user experience everywhere in the code base. We would
>> like to thank everyone in the Beam community to contribute to such an
>> amazing framework!
>>
>> 2) A portable Samza Runner prototype
>> We are also starting the work in making Samza Runner portable. So far we
>> just got the python word count example working using portable Samza Runner.
>> Please look out for the PR for this very soon :). Again, this work is not
>> possible without the great Beam portability framework, and the developers
>> like Luke and Ahmet, just to name a few, behind it. The ReferenceRunner has
>> been extremely useful to us to figure out what's needed and how it works.
>> Kudos to Thomas Groh, Ben Sidhom and all the others who makes this
>> available to us. And to Kenn, your fuse work rocks.
>>
>> 3) More contributors in Samza Runner
>> The runner has been Chris and my personal project for a while and now
>> it's not the case. We got Hai Lu and Boris Shkolnik from Samza team to
>> contribute. Hai has been focusing on the portability work as mentioned in
>> #2, and Boris will work mostly on supporting our use cases. We will send
>> more emails discussing our use cases, like the "Update state after firing"
>> email I sent out earlier.
>>
>> Finally, a shout-out to our very own Chris Pettitt. Without you, none of
>> the above won't happen!
>>
>> Thanks,
>> Xinyu
>>
>
Re: Beam Samza Runner status update
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks for sharing and congrats for this great work !
Regards
JB
Le 10 oct. 2018 à 20:23, à 20:23, Xinyu Liu <xi...@gmail.com> a écrit:
>Hi, All,
>
>It's been over four months since we added the Samza Runner to Beam, and
>we've been making a lot of progress after that. Here I would like to
>update
>your guys and share some really good news happening here at LinkedIn:
>
>1) First Beam job in production @LInkedIn!
>After a few rounds of testing and benchmarking, we finally rolled out
>our
>first Beam job here! The job uses quite a few features, such as event
>time,
>fixed/session windowing, early triggering, and stateful processing. Our
>first customer is very happy and they highly appraise the easy-to-use
>Beam
>API as well as powerful processing model. Due to the limited resources
>here, we put our full trust in the work you guys are doing, and we
>didn't
>run into any surprises. We see extremely attention to details as well
>as
>non-compromise in any user experience everywhere in the code base. We
>would
>like to thank everyone in the Beam community to contribute to such an
>amazing framework!
>
>2) A portable Samza Runner prototype
>We are also starting the work in making Samza Runner portable. So far
>we
>just got the python word count example working using portable Samza
>Runner.
>Please look out for the PR for this very soon :). Again, this work is
>not
>possible without the great Beam portability framework, and the
>developers
>like Luke and Ahmet, just to name a few, behind it. The ReferenceRunner
>has
>been extremely useful to us to figure out what's needed and how it
>works.
>Kudos to Thomas Groh, Ben Sidhom and all the others who makes this
>available to us. And to Kenn, your fuse work rocks.
>
>3) More contributors in Samza Runner
>The runner has been Chris and my personal project for a while and now
>it's
>not the case. We got Hai Lu and Boris Shkolnik from Samza team to
>contribute. Hai has been focusing on the portability work as mentioned
>in
>#2, and Boris will work mostly on supporting our use cases. We will
>send
>more emails discussing our use cases, like the "Update state after
>firing"
>email I sent out earlier.
>
>Finally, a shout-out to our very own Chris Pettitt. Without you, none
>of
>the above won't happen!
>
>Thanks,
>Xinyu