You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Aviem Zur <av...@gmail.com> on 2016/12/11 15:38:49 UTC
Scio Beam Scala API
Hi,
I've heard there has been work towards porting Scio Dataflow Scala API to
Beam.
I was wondering at what stage this is in, where is this happening (Saw no
branch in BEAM repository, and one in Scio repository that is dependent on
beam 0.2.0-INCUBATING) and if there is a way to contribute?
Re: Scio Beam Scala API
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi guys,
I will share the branch with the 0.4.0-incubating-SNAPSHOT update and
tested with Spark runner.
Regards
JB
On 12/11/2016 09:13 PM, Neville Li wrote:
> Aviem,
>
> You're right we have a scio/apache-beam
> <https://github.com/spotify/scio/tree/apache-beam> branch that works with
> Beam 0.2.0-incubating and are working on keeping it up with the latest
> releases.
>
> A few ways you can contribute:
> - It runs with the Dataflow runner but hasn't been tested with other
> runners. You're welcome to give it a try and submit issues/PRs.
> - `scio-core` is somewhat coupled with Dataflow runner and GCP IO
> dependencies right now but it'd be nice to further decouple them so users
> can swap other runner/IO packages easily.
> - We also have a master ticket #279
> <https://github.com/spotify/scio/issues/279> that keeps track of pending
> issues for Beam migration.
>
> Keep in mind that our team of 3 supports 150+ production Scio users within
> Spotify so we simply don't have the bandwidth to maintain 2 diverging repo
> (spotify/scio vs apache/beam-incubating) right now. We'll probably revisit
> this when internal users switch over to Beam sometime next year.
>
>
> On Sun, Dec 11, 2016 at 4:45 PM Jean-Baptiste Onofr� <jb...@nanthrax.net>
> wrote:
>
>> Hi
>>
>> I'm working on a feature branch with Neville and his guys. I already
>> updated to last changes. I would like to propose a feature branch later
>> this week.
>>
>> Regards
>> JB\u2063\u200b
>>
>> On Dec 11, 2016, 16:39, at 16:39, Aviem Zur <av...@gmail.com> wrote:
>>> Hi,
>>>
>>> I've heard there has been work towards porting Scio Dataflow Scala API
>>> to
>>> Beam.
>>> I was wondering at what stage this is in, where is this happening (Saw
>>> no
>>> branch in BEAM repository, and one in Scio repository that is dependent
>>> on
>>> beam 0.2.0-INCUBATING) and if there is a way to contribute?
>>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Scio Beam Scala API
Posted by Neville Li <ne...@gmail.com>.
Aviem,
You're right we have a scio/apache-beam
<https://github.com/spotify/scio/tree/apache-beam> branch that works with
Beam 0.2.0-incubating and are working on keeping it up with the latest
releases.
A few ways you can contribute:
- It runs with the Dataflow runner but hasn't been tested with other
runners. You're welcome to give it a try and submit issues/PRs.
- `scio-core` is somewhat coupled with Dataflow runner and GCP IO
dependencies right now but it'd be nice to further decouple them so users
can swap other runner/IO packages easily.
- We also have a master ticket #279
<https://github.com/spotify/scio/issues/279> that keeps track of pending
issues for Beam migration.
Keep in mind that our team of 3 supports 150+ production Scio users within
Spotify so we simply don't have the bandwidth to maintain 2 diverging repo
(spotify/scio vs apache/beam-incubating) right now. We'll probably revisit
this when internal users switch over to Beam sometime next year.
On Sun, Dec 11, 2016 at 4:45 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi
>
> I'm working on a feature branch with Neville and his guys. I already
> updated to last changes. I would like to propose a feature branch later
> this week.
>
> Regards
> JB
>
> On Dec 11, 2016, 16:39, at 16:39, Aviem Zur <av...@gmail.com> wrote:
> >Hi,
> >
> >I've heard there has been work towards porting Scio Dataflow Scala API
> >to
> >Beam.
> >I was wondering at what stage this is in, where is this happening (Saw
> >no
> >branch in BEAM repository, and one in Scio repository that is dependent
> >on
> >beam 0.2.0-INCUBATING) and if there is a way to contribute?
>
Re: Scio Beam Scala API
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
I'm working on a feature branch with Neville and his guys. I already updated to last changes. I would like to propose a feature branch later this week.
Regards
JB\u2063\u200b
On Dec 11, 2016, 16:39, at 16:39, Aviem Zur <av...@gmail.com> wrote:
>Hi,
>
>I've heard there has been work towards porting Scio Dataflow Scala API
>to
>Beam.
>I was wondering at what stage this is in, where is this happening (Saw
>no
>branch in BEAM repository, and one in Scio repository that is dependent
>on
>beam 0.2.0-INCUBATING) and if there is a way to contribute?