[jira] [Comment Edited] (BEAM-11) Integrate Spark runner with Beam

["Amit Sela (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183372#comment-15183372 ] 

Amit Sela edited comment on BEAM-11 at 3/7/16 6:10 PM:
-------------------------------------------------------

Yes, I'm on it :) pending the code drop. Hang tight..
I'll take a look at your work, and compare with mine. I'd merge our work, or tell you to do a pull request, but this is a weird time since there is still no code... But no worries, the code will be dropped soon and we can get things going.

There is a lot of work to do on supporting the Beam model, especially in streaming, but also in metric reporting and more, as stated in the technical vision document: https://drive.google.com/folderview?id=0B-IhJZh9Ab52OFBVZHpsNjc4eXc&usp=sharing

Currently, the runner supports batch processing, and some, limited, stream processing, which is OK - we can state the runner's current capabilities (keep in mind that not all runners will support the entire model). Having said that, we will add more support as we go.

Concerning branch organization see: https://drive.google.com/folderview?id=0B-IhJZh9Ab52OFBVZHpsNjc4eXc&usp=sharing
It makes total sense to develop large features of the runner in a feature branch, and merge once done.


was (Author: amitsela):
Yes, I'm on it :) pending the code drop. Hang tight..
I'll take a look at your work, and compare with mine. I'd merge our work, or tell you to do a pull request, but this is a weird time since there is still no code... But no worries, the code will be dropped soon and we can get things going.

There is a lot of work to do on the Beam model, especially in streaming, but also in metric reporting and more, as stated in the technical vision document: https://drive.google.com/folderview?id=0B-IhJZh9Ab52OFBVZHpsNjc4eXc&usp=sharing

Currently, the runner supports batch processing, and some, limited, stream processing, which is OK - we can state the runner's current capabilities (keep in mind that not all runners will support the entire model). Having said that, we will add more support as we go.

Concerning branch organization see: https://drive.google.com/folderview?id=0B-IhJZh9Ab52OFBVZHpsNjc4eXc&usp=sharing
It makes total sense to develop large features of the runner in a feature branch, and merge once done.

> Integrate Spark runner with Beam
> --------------------------------
>
>                 Key: BEAM-11
>                 URL: https://issues.apache.org/jira/browse/BEAM-11
>             Project: Beam
>          Issue Type: Task
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Amit Sela
>
> Refactor and integrate the Spark runner code against Google's contributed version of Dataflow - Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)