You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by ������ <sh...@gmail.com> on 2019/08/30 08:56:12 UTC
How to Implement a Runner for C++ Streaming Processing Engine���
I have a streaming processing engine implemented in C++. Do you have any good plans and suggestions to implement a runner?
I looked at the Beam Runner implementation guide and felt that I couldn't implement a Runner for a C++ streaming data processing engine. Because I can't let the C++ streaming data processing engine execute beam pipeline if I implement Runner in java.
Re: How to Implement a Runner for C++ Streaming Processing Engine���
Posted by ������ <sh...@gmail.com>.
On 2019/08/30 15:37:21, Lukasz Cwik <lc...@google.com> wrote:
> Sorry, meant to say that some re-usable parts exist in Python and Go (not
> C++).
>
> On Fri, Aug 30, 2019 at 8:36 AM Lukasz Cwik <lc...@google.com> wrote:
>
> > There is an ongoing portability effort which is attempting to enable any
> > runner execute any Beam SDK written in any language.
> >
> > A good starting point would be to take a look at:
> > https://beam.apache.org/roadmap/portability/
> > http://s.apache.org/beam-runner-api
> > http://s.apache.org/beam-fn-api
> >
> > The idea behind portability is that you can create a language agnostic
> > representation of the pipeline in proto and use gRPC based services to
> > launch, manage and execute that pipeline. Since Proto and gPRC both have
> > C++ bindings, you should be able to integrate your execution engine but one
> > issue you'll run into is that most of the code that allows one to run
> > portable pipelines is written in Java and some re-usable parts in exist in
> > Python and C++.
> >
> > On Fri, Aug 30, 2019 at 7:40 AM 邵凯 <sh...@gmail.com> wrote:
> >
> >> I have a streaming processing engine implemented in C++. Do you have any
> >> good plans and suggestions to implement a runner?
> >> I looked at the Beam Runner implementation guide and felt that I couldn't
> >> implement a Runner for a C++ streaming data processing engine. Because I
> >> can't let the C++ streaming data processing engine execute beam pipeline if
> >> I implement Runner in java.
> >>
> >
> Thank you very much .
Re: How to Implement a Runner for C++ Streaming Processing Engine？
Posted by Lukasz Cwik <lc...@google.com>.
Sorry, meant to say that some re-usable parts exist in Python and Go (not
C++).
On Fri, Aug 30, 2019 at 8:36 AM Lukasz Cwik <lc...@google.com> wrote:
> There is an ongoing portability effort which is attempting to enable any
> runner execute any Beam SDK written in any language.
>
> A good starting point would be to take a look at:
> https://beam.apache.org/roadmap/portability/
> http://s.apache.org/beam-runner-api
> http://s.apache.org/beam-fn-api
>
> The idea behind portability is that you can create a language agnostic
> representation of the pipeline in proto and use gRPC based services to
> launch, manage and execute that pipeline. Since Proto and gPRC both have
> C++ bindings, you should be able to integrate your execution engine but one
> issue you'll run into is that most of the code that allows one to run
> portable pipelines is written in Java and some re-usable parts in exist in
> Python and C++.
>
> On Fri, Aug 30, 2019 at 7:40 AM 邵凯 <sh...@gmail.com> wrote:
>
>> I have a streaming processing engine implemented in C++. Do you have any
>> good plans and suggestions to implement a runner?
>> I looked at the Beam Runner implementation guide and felt that I couldn't
>> implement a Runner for a C++ streaming data processing engine. Because I
>> can't let the C++ streaming data processing engine execute beam pipeline if
>> I implement Runner in java.
>>
>
Re: How to Implement a Runner for C++ Streaming Processing Engine？
Posted by Lukasz Cwik <lc...@google.com>.
There is an ongoing portability effort which is attempting to enable any
runner execute any Beam SDK written in any language.
A good starting point would be to take a look at:
https://beam.apache.org/roadmap/portability/
http://s.apache.org/beam-runner-api
http://s.apache.org/beam-fn-api
The idea behind portability is that you can create a language agnostic
representation of the pipeline in proto and use gRPC based services to
launch, manage and execute that pipeline. Since Proto and gPRC both have
C++ bindings, you should be able to integrate your execution engine but one
issue you'll run into is that most of the code that allows one to run
portable pipelines is written in Java and some re-usable parts in exist in
Python and C++.
On Fri, Aug 30, 2019 at 7:40 AM 邵凯 <sh...@gmail.com> wrote:
> I have a streaming processing engine implemented in C++. Do you have any
> good plans and suggestions to implement a runner?
> I looked at the Beam Runner implementation guide and felt that I couldn't
> implement a Runner for a C++ streaming data processing engine. Because I
> can't let the C++ streaming data processing engine execute beam pipeline if
> I implement Runner in java.
>