You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Manu Zhang <ow...@gmail.com> on 2016/04/07 06:24:16 UTC
Gearpump Runner Design Document
Hi all,
I've drafted a design document
<https://drive.google.com/open?id=1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g>for
Gearpump Runner (https://issues.apache.org/jira/browse/BEAM-79). Please
kindly offer your comment.
Thanks,
Manu
Re: Gearpump Runner Design Document
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
I went through the doc and added some comments.
On Fri, Apr 8, 2016 at 7:32 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Manu,
>
> awesome !
>
> Thanks for sharing this !
>
> I will take a look.
>
> Regards
> JB
>
>
> On 04/08/2016 03:38 AM, Manu Zhang wrote:
>
>> Hi JB,
>>
>> - it seems the Gearpump Serializer (based on Kryo) can be expressed as a
>>
>>> Coder. You mean because you have only binary "input", whereas the Coder
>>> needs a specific type. Right ?
>>>
>>
>>
>> Yes. I'm thinking about adding a typed serializer interface to Gearpump
>> and
>> the default serializer based on Kryo could be type of "Any" in Scala.
>> Then
>> it's trivial to add support for Coder.
>>
>>
>> - Any plan for the side inputs (I don't know if we can do it easily in
>>
>>> Gearpump) ?
>>>
>>
>>
>> Certainly, it's in our plan. Off the top of my head, Gearpump is built on
>> Akka Actors and side inputs could be actor states which we can query with
>> messages. Any good ideas about how a do it in a streaming engine (not
>> specifically on Gearpump) ?
>>
>> The code is at
>> https://github.com/manuzhang/incubator-beam/tree/gearpump_runner. Nothing
>> is working yet and I still need to fix styles, add docs and tests. The
>> focus now is to get a working example first.
>>
>> I also created an umbrella jira (
>> https://issues.apache.org/jira/browse/GEARPUMP-21) to track status and
>> related issues at Gearpump side and notify the Gearpump community.
>>
>> Regards,
>> Manu
>>
>> On Thu, Apr 7, 2016 at 1:45 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>> Hi Manu,
>>>
>>> I replied in the Jira.
>>>
>>> It sounds good to me.
>>>
>>> Just a question about the gaps:
>>> - it seems the Gearpump Serializer (based on Kryo) can be expressed as a
>>> Coder. You mean because you have only binary "input", whereas the Coder
>>> needs a specific type. Right ?
>>> - Any plan for the side inputs (I don't know if we can do it easily in
>>> Gearpump) ?
>>>
>>> Is it possible to see the code in progress ?
>>>
>>> Thanks !
>>>
>>> Regards
>>> JB
>>>
>>> On 04/07/2016 06:24 AM, Manu Zhang wrote:
>>>
>>> Hi all,
>>>>
>>>> I've drafted a design document
>>>> <
>>>>
>>>> https://drive.google.com/open?id=1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g
>>>>
>>>>> for
>>>>>
>>>> Gearpump Runner (https://issues.apache.org/jira/browse/BEAM-79). Please
>>>> kindly offer your comment.
>>>>
>>>>
>>>> Thanks,
>>>> Manu
>>>>
>>>>
>>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Gearpump Runner Design Document
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Manu,
awesome !
Thanks for sharing this !
I will take a look.
Regards
JB
On 04/08/2016 03:38 AM, Manu Zhang wrote:
> Hi JB,
>
> - it seems the Gearpump Serializer (based on Kryo) can be expressed as a
>> Coder. You mean because you have only binary "input", whereas the Coder
>> needs a specific type. Right ?
>
>
> Yes. I'm thinking about adding a typed serializer interface to Gearpump and
> the default serializer based on Kryo could be type of "Any" in Scala. Then
> it's trivial to add support for Coder.
>
>
> - Any plan for the side inputs (I don't know if we can do it easily in
>> Gearpump) ?
>
>
> Certainly, it's in our plan. Off the top of my head, Gearpump is built on
> Akka Actors and side inputs could be actor states which we can query with
> messages. Any good ideas about how a do it in a streaming engine (not
> specifically on Gearpump) ?
>
> The code is at
> https://github.com/manuzhang/incubator-beam/tree/gearpump_runner. Nothing
> is working yet and I still need to fix styles, add docs and tests. The
> focus now is to get a working example first.
>
> I also created an umbrella jira (
> https://issues.apache.org/jira/browse/GEARPUMP-21) to track status and
> related issues at Gearpump side and notify the Gearpump community.
>
> Regards,
> Manu
>
> On Thu, Apr 7, 2016 at 1:45 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Manu,
>>
>> I replied in the Jira.
>>
>> It sounds good to me.
>>
>> Just a question about the gaps:
>> - it seems the Gearpump Serializer (based on Kryo) can be expressed as a
>> Coder. You mean because you have only binary "input", whereas the Coder
>> needs a specific type. Right ?
>> - Any plan for the side inputs (I don't know if we can do it easily in
>> Gearpump) ?
>>
>> Is it possible to see the code in progress ?
>>
>> Thanks !
>>
>> Regards
>> JB
>>
>> On 04/07/2016 06:24 AM, Manu Zhang wrote:
>>
>>> Hi all,
>>>
>>> I've drafted a design document
>>> <
>>> https://drive.google.com/open?id=1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g
>>>> for
>>> Gearpump Runner (https://issues.apache.org/jira/browse/BEAM-79). Please
>>> kindly offer your comment.
>>>
>>>
>>> Thanks,
>>> Manu
>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Gearpump Runner Design Document
Posted by Manu Zhang <ow...@gmail.com>.
Hi JB,
- it seems the Gearpump Serializer (based on Kryo) can be expressed as a
> Coder. You mean because you have only binary "input", whereas the Coder
> needs a specific type. Right ?
Yes. I'm thinking about adding a typed serializer interface to Gearpump and
the default serializer based on Kryo could be type of "Any" in Scala. Then
it's trivial to add support for Coder.
- Any plan for the side inputs (I don't know if we can do it easily in
> Gearpump) ?
Certainly, it's in our plan. Off the top of my head, Gearpump is built on
Akka Actors and side inputs could be actor states which we can query with
messages. Any good ideas about how a do it in a streaming engine (not
specifically on Gearpump) ?
The code is at
https://github.com/manuzhang/incubator-beam/tree/gearpump_runner. Nothing
is working yet and I still need to fix styles, add docs and tests. The
focus now is to get a working example first.
I also created an umbrella jira (
https://issues.apache.org/jira/browse/GEARPUMP-21) to track status and
related issues at Gearpump side and notify the Gearpump community.
Regards,
Manu
On Thu, Apr 7, 2016 at 1:45 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Manu,
>
> I replied in the Jira.
>
> It sounds good to me.
>
> Just a question about the gaps:
> - it seems the Gearpump Serializer (based on Kryo) can be expressed as a
> Coder. You mean because you have only binary "input", whereas the Coder
> needs a specific type. Right ?
> - Any plan for the side inputs (I don't know if we can do it easily in
> Gearpump) ?
>
> Is it possible to see the code in progress ?
>
> Thanks !
>
> Regards
> JB
>
> On 04/07/2016 06:24 AM, Manu Zhang wrote:
>
>> Hi all,
>>
>> I've drafted a design document
>> <
>> https://drive.google.com/open?id=1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g
>> >for
>> Gearpump Runner (https://issues.apache.org/jira/browse/BEAM-79). Please
>> kindly offer your comment.
>>
>>
>> Thanks,
>> Manu
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Gearpump Runner Design Document
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Manu,
I replied in the Jira.
It sounds good to me.
Just a question about the gaps:
- it seems the Gearpump Serializer (based on Kryo) can be expressed as a
Coder. You mean because you have only binary "input", whereas the Coder
needs a specific type. Right ?
- Any plan for the side inputs (I don't know if we can do it easily in
Gearpump) ?
Is it possible to see the code in progress ?
Thanks !
Regards
JB
On 04/07/2016 06:24 AM, Manu Zhang wrote:
> Hi all,
>
> I've drafted a design document
> <https://drive.google.com/open?id=1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g>for
> Gearpump Runner (https://issues.apache.org/jira/browse/BEAM-79). Please
> kindly offer your comment.
>
>
> Thanks,
> Manu
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com