You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jean-Baptiste Onofré <jb...@nanthrax.net> on 2017/11/16 13:08:50 UTC
[VOTE] Choose the "new" Spark runner
Hi guys,
To illustrate the current discussion about Spark versions support, you can take
a look on:
--
Spark 1 & Spark 2 Support Branch
https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
This branch contains a Spark runner common module compatible with both Spark 1.x
and 2.x. For convenience, we introduced spark1 & spark2 modules/artifacts
containing just a pom.xml to define the dependencies set.
--
Spark 2 Only Branch
https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
As I'm ready to merge one of the other in the PR, I would like to complete the
vote/discussion pretty soon.
Correct me if I'm wrong, but it seems that the preference is to drop Spark 1.x
to focus only on Spark 2.x (for the Spark 2 Only Branch).
I would like to call a final vote to act the merge I will do:
[ ] Use Spark 1 & Spark 2 Support Branch
[ ] Use Spark 2 Only Branch
This informal vote is open for 48 hours.
Please, let me know what your preference is.
Thanks !
Regards
JB
On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
> Hi Beamers,
>
> I'm forwarding this discussion & vote from the dev mailing list to the user
> mailing list.
> The goal is to have your feedback as user.
>
> Basically, we have two options:
> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three artifacts
> (common, spark1, spark2). You, as users, pick up spark1 or spark2 in your
> dependencies set depending the Spark target version you want.
> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0. If you
> still want to use Spark 1.x, then, you will be stuck up to Beam 2.2.0.
>
> Thoughts ?
>
> Thanks !
> Regards
> JB
>
>
> -------- Forwarded Message --------
> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
> Date: Wed, 8 Nov 2017 08:27:58 +0100
> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> Reply-To: dev@beam.apache.org
> To: dev@beam.apache.org
>
> Hi all,
>
> as you might know, we are working on Spark 2.x support in the Spark runner.
>
> I'm working on a PR about that:
>
> https://github.com/apache/beam/pull/3808
>
> Today, we have something working with both Spark 1.x and 2.x from a code
> standpoint, but I have to deal with dependencies. It's the first step of the
> update as I'm still using RDD, the second step would be to support dataframe
> (but for that, I would need PCollection elements with schemas, that's another
> topic on which Eugene, Reuven and I are discussing).
>
> However, as all major distributions now ship Spark 2.x, I don't think it's
> required anymore to support Spark 1.x.
>
> If we agree, I will update and cleanup the PR to only support and focus on Spark
> 2.x.
>
> So, that's why I'm calling for a vote:
>
> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
> [ ] 0 (I don't care ;))
> [ ] -1, I would like to still support Spark 1.x, and so having support of
> both Spark 1.x and 2.x (please provide specific comment)
>
> This vote is open for 48 hours (I have the commits ready, just waiting the end
> of the vote to push on the PR).
>
> Thanks !
> Regards
> JB
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Choose the "new" Spark runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Moving my vote from previous threads:
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
Ismaël
On Thu, Nov 16, 2017 at 2:08 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi guys,
>
> To illustrate the current discussion about Spark versions support, you can
> take a look on:
>
> --
> Spark 1 & Spark 2 Support Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
>
> This branch contains a Spark runner common module compatible with both Spark
> 1.x and 2.x. For convenience, we introduced spark1 & spark2
> modules/artifacts containing just a pom.xml to define the dependencies set.
>
> --
> Spark 2 Only Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
>
> This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
>
> As I'm ready to merge one of the other in the PR, I would like to complete
> the vote/discussion pretty soon.
>
> Correct me if I'm wrong, but it seems that the preference is to drop Spark
> 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
>
> I would like to call a final vote to act the merge I will do:
>
> [ ] Use Spark 1 & Spark 2 Support Branch
> [ ] Use Spark 2 Only Branch
>
> This informal vote is open for 48 hours.
>
> Please, let me know what your preference is.
>
> Thanks !
> Regards
> JB
>
> On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
>>
>> Hi Beamers,
>>
>> I'm forwarding this discussion & vote from the dev mailing list to the
>> user mailing list.
>> The goal is to have your feedback as user.
>>
>> Basically, we have two options:
>> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
>> artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2
>> in your dependencies set depending the Spark target version you want.
>> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0. If
>> you still want to use Spark 1.x, then, you will be stuck up to Beam 2.2.0.
>>
>> Thoughts ?
>>
>> Thanks !
>> Regards
>> JB
>>
>>
>> -------- Forwarded Message --------
>> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
>> Date: Wed, 8 Nov 2017 08:27:58 +0100
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Reply-To: dev@beam.apache.org
>> To: dev@beam.apache.org
>>
>> Hi all,
>>
>> as you might know, we are working on Spark 2.x support in the Spark
>> runner.
>>
>> I'm working on a PR about that:
>>
>> https://github.com/apache/beam/pull/3808
>>
>> Today, we have something working with both Spark 1.x and 2.x from a code
>> standpoint, but I have to deal with dependencies. It's the first step of the
>> update as I'm still using RDD, the second step would be to support dataframe
>> (but for that, I would need PCollection elements with schemas, that's
>> another topic on which Eugene, Reuven and I are discussing).
>>
>> However, as all major distributions now ship Spark 2.x, I don't think it's
>> required anymore to support Spark 1.x.
>>
>> If we agree, I will update and cleanup the PR to only support and focus on
>> Spark 2.x.
>>
>> So, that's why I'm calling for a vote:
>>
>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
>> [ ] 0 (I don't care ;))
>> [ ] -1, I would like to still support Spark 1.x, and so having support
>> of both Spark 1.x and 2.x (please provide specific comment)
>>
>> This vote is open for 48 hours (I have the commits ready, just waiting the
>> end of the vote to push on the PR).
>>
>> Thanks !
>> Regards
>> JB
>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: [VOTE] Choose the "new" Spark runner
Posted by Kenneth Knowles <kl...@google.com>.
For your convenience, here's the compare view:
https://github.com/apache/beam/compare/master...jbonofre:BEAM-1920-SPARK2-MODULES
https://github.com/apache/beam/compare/master...jbonofre:BEAM-1920-SPARK2-ONLY
Kenn
On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi guys,
>
> To illustrate the current discussion about Spark versions support, you can
> take a look on:
>
> --
> Spark 1 & Spark 2 Support Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
>
> This branch contains a Spark runner common module compatible with both
> Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> modules/artifacts containing just a pom.xml to define the dependencies set.
>
> --
> Spark 2 Only Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
>
> This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
>
> As I'm ready to merge one of the other in the PR, I would like to complete
> the vote/discussion pretty soon.
>
> Correct me if I'm wrong, but it seems that the preference is to drop Spark
> 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
>
> I would like to call a final vote to act the merge I will do:
>
> [ ] Use Spark 1 & Spark 2 Support Branch
> [ ] Use Spark 2 Only Branch
>
> This informal vote is open for 48 hours.
>
> Please, let me know what your preference is.
>
> Thanks !
> Regards
> JB
>
> On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
>
>> Hi Beamers,
>>
>> I'm forwarding this discussion & vote from the dev mailing list to the
>> user mailing list.
>> The goal is to have your feedback as user.
>>
>> Basically, we have two options:
>> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
>> artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2
>> in your dependencies set depending the Spark target version you want.
>> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0.
>> If you still want to use Spark 1.x, then, you will be stuck up to Beam
>> 2.2.0.
>>
>> Thoughts ?
>>
>> Thanks !
>> Regards
>> JB
>>
>>
>> -------- Forwarded Message --------
>> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
>> Date: Wed, 8 Nov 2017 08:27:58 +0100
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Reply-To: dev@beam.apache.org
>> To: dev@beam.apache.org
>>
>> Hi all,
>>
>> as you might know, we are working on Spark 2.x support in the Spark
>> runner.
>>
>> I'm working on a PR about that:
>>
>> https://github.com/apache/beam/pull/3808
>>
>> Today, we have something working with both Spark 1.x and 2.x from a code
>> standpoint, but I have to deal with dependencies. It's the first step of
>> the update as I'm still using RDD, the second step would be to support
>> dataframe (but for that, I would need PCollection elements with schemas,
>> that's another topic on which Eugene, Reuven and I are discussing).
>>
>> However, as all major distributions now ship Spark 2.x, I don't think
>> it's required anymore to support Spark 1.x.
>>
>> If we agree, I will update and cleanup the PR to only support and focus
>> on Spark 2.x.
>>
>> So, that's why I'm calling for a vote:
>>
>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
>> [ ] 0 (I don't care ;))
>> [ ] -1, I would like to still support Spark 1.x, and so having support
>> of both Spark 1.x and 2.x (please provide specific comment)
>>
>> This vote is open for 48 hours (I have the commits ready, just waiting
>> the end of the vote to push on the PR).
>>
>> Thanks !
>> Regards
>> JB
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Holden Karau <ho...@pigscanfly.ca>.
[ ] Use Spark 1 & Spark 2 Support Branch
[ X ] Use Spark 2 Only Branch
non-binding
On Mon, Nov 20, 2017 at 1:00 AM, Etienne Chauchot <ec...@apache.org>
wrote:
> [ ] Use Spark 1 & Spark 2 Support Branch
> [X] Use Spark 2 Only Branch
>
> Best
> Etienne
>
>
>
> Le 19/11/2017 à 13:56, Tyler Akidau a écrit :
>
>> [ ] Use Spark 1 & Spark 2 Support Branch
>> [X] Use Spark 2 Only Branch
>>
>
>
--
Twitter: https://twitter.com/holdenkarau
Re: [VOTE] Choose the "new" Spark runner
Posted by Etienne Chauchot <ec...@apache.org>.
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
Best
Etienne
Le 19/11/2017 à 13:56, Tyler Akidau a écrit :
> [ ] Use Spark 1 & Spark 2 Support Branch
> [X] Use Spark 2 Only Branch
Re: [VOTE] Choose the "new" Spark runner
Posted by Tyler Akidau <ta...@apache.org>.
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
On Sun, Nov 19, 2017 at 1:46 PM Amit Sela <am...@gmail.com> wrote:
> [X] Use Spark 2 Only Branch
>
> On Sun, Nov 19, 2017, 02:46 Reuven Lax <re...@google.com.invalid> wrote:
>
> > [ ] Use Spark 1 & Spark 2 Support Branch
> > [X] Use Spark 2 Only Branch
> >
> > On Sat, Nov 18, 2017 at 1:54 AM, Ben Sidhom <si...@google.com.invalid>
> > wrote:
> >
> > > [ ] Use Spark 1 & Spark 2 Support Branch
> > > [X] Use Spark 2 Only Branch
> > >
> > > On Fri, Nov 17, 2017 at 9:46 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > [ ] Use Spark 1 & Spark 2 Support Branch
> > > > [X] Use Spark 2 Only Branch
> > > >
> > > > On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >
> > > > wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > To illustrate the current discussion about Spark versions support,
> > you
> > > > can
> > > > > take a look on:
> > > > >
> > > > > --
> > > > > Spark 1 & Spark 2 Support Branch
> > > > >
> > > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
> > > > >
> > > > > This branch contains a Spark runner common module compatible with
> > both
> > > > > Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> > > > > modules/artifacts containing just a pom.xml to define the
> > dependencies
> > > > set.
> > > > >
> > > > > --
> > > > > Spark 2 Only Branch
> > > > >
> > > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
> > > > >
> > > > > This branch is an upgrade to Spark 2.x and "drop" support of Spark
> > 1.x.
> > > > >
> > > > > As I'm ready to merge one of the other in the PR, I would like to
> > > > complete
> > > > > the vote/discussion pretty soon.
> > > > >
> > > > > Correct me if I'm wrong, but it seems that the preference is to
> drop
> > > > Spark
> > > > > 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
> > > > >
> > > > > I would like to call a final vote to act the merge I will do:
> > > > >
> > > > > [ ] Use Spark 1 & Spark 2 Support Branch
> > > > > [ ] Use Spark 2 Only Branch
> > > > >
> > > > > This informal vote is open for 48 hours.
> > > > >
> > > > > Please, let me know what your preference is.
> > > > >
> > > > > Thanks !
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
> > > > >
> > > > >> Hi Beamers,
> > > > >>
> > > > >> I'm forwarding this discussion & vote from the dev mailing list to
> > the
> > > > >> user mailing list.
> > > > >> The goal is to have your feedback as user.
> > > > >>
> > > > >> Basically, we have two options:
> > > > >> 1. Right now, in the PR, we support both Spark 1.x and 2.x using
> > three
> > > > >> artifacts (common, spark1, spark2). You, as users, pick up spark1
> or
> > > > spark2
> > > > >> in your dependencies set depending the Spark target version you
> > want.
> > > > >> 2. The other option is to upgrade and focus on Spark 2.x in Beam
> > > 2.3.0.
> > > > >> If you still want to use Spark 1.x, then, you will be stuck up to
> > Beam
> > > > >> 2.2.0.
> > > > >>
> > > > >> Thoughts ?
> > > > >>
> > > > >> Thanks !
> > > > >> Regards
> > > > >> JB
> > > > >>
> > > > >>
> > > > >> -------- Forwarded Message --------
> > > > >> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
> > > > >> Date: Wed, 8 Nov 2017 08:27:58 +0100
> > > > >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > > >> Reply-To: dev@beam.apache.org
> > > > >> To: dev@beam.apache.org
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> as you might know, we are working on Spark 2.x support in the
> Spark
> > > > >> runner.
> > > > >>
> > > > >> I'm working on a PR about that:
> > > > >>
> > > > >> https://github.com/apache/beam/pull/3808
> > > > >>
> > > > >> Today, we have something working with both Spark 1.x and 2.x from
> a
> > > code
> > > > >> standpoint, but I have to deal with dependencies. It's the first
> > step
> > > of
> > > > >> the update as I'm still using RDD, the second step would be to
> > support
> > > > >> dataframe (but for that, I would need PCollection elements with
> > > schemas,
> > > > >> that's another topic on which Eugene, Reuven and I are
> discussing).
> > > > >>
> > > > >> However, as all major distributions now ship Spark 2.x, I don't
> > think
> > > > >> it's required anymore to support Spark 1.x.
> > > > >>
> > > > >> If we agree, I will update and cleanup the PR to only support and
> > > focus
> > > > >> on Spark 2.x.
> > > > >>
> > > > >> So, that's why I'm calling for a vote:
> > > > >>
> > > > >> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
> > > > >> [ ] 0 (I don't care ;))
> > > > >> [ ] -1, I would like to still support Spark 1.x, and so having
> > > > support
> > > > >> of both Spark 1.x and 2.x (please provide specific comment)
> > > > >>
> > > > >> This vote is open for 48 hours (I have the commits ready, just
> > waiting
> > > > >> the end of the vote to push on the PR).
> > > > >>
> > > > >> Thanks !
> > > > >> Regards
> > > > >> JB
> > > > >>
> > > > >
> > > > > --
> > > > > Jean-Baptiste Onofré
> > > > > jbonofre@apache.org
> > > > > http://blog.nanthrax.net
> > > > > Talend - http://www.talend.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -Ben
> > >
> >
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Amit Sela <am...@gmail.com>.
[X] Use Spark 2 Only Branch
On Sun, Nov 19, 2017, 02:46 Reuven Lax <re...@google.com.invalid> wrote:
> [ ] Use Spark 1 & Spark 2 Support Branch
> [X] Use Spark 2 Only Branch
>
> On Sat, Nov 18, 2017 at 1:54 AM, Ben Sidhom <si...@google.com.invalid>
> wrote:
>
> > [ ] Use Spark 1 & Spark 2 Support Branch
> > [X] Use Spark 2 Only Branch
> >
> > On Fri, Nov 17, 2017 at 9:46 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > [ ] Use Spark 1 & Spark 2 Support Branch
> > > [X] Use Spark 2 Only Branch
> > >
> > > On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > To illustrate the current discussion about Spark versions support,
> you
> > > can
> > > > take a look on:
> > > >
> > > > --
> > > > Spark 1 & Spark 2 Support Branch
> > > >
> > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
> > > >
> > > > This branch contains a Spark runner common module compatible with
> both
> > > > Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> > > > modules/artifacts containing just a pom.xml to define the
> dependencies
> > > set.
> > > >
> > > > --
> > > > Spark 2 Only Branch
> > > >
> > > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
> > > >
> > > > This branch is an upgrade to Spark 2.x and "drop" support of Spark
> 1.x.
> > > >
> > > > As I'm ready to merge one of the other in the PR, I would like to
> > > complete
> > > > the vote/discussion pretty soon.
> > > >
> > > > Correct me if I'm wrong, but it seems that the preference is to drop
> > > Spark
> > > > 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
> > > >
> > > > I would like to call a final vote to act the merge I will do:
> > > >
> > > > [ ] Use Spark 1 & Spark 2 Support Branch
> > > > [ ] Use Spark 2 Only Branch
> > > >
> > > > This informal vote is open for 48 hours.
> > > >
> > > > Please, let me know what your preference is.
> > > >
> > > > Thanks !
> > > > Regards
> > > > JB
> > > >
> > > > On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
> > > >
> > > >> Hi Beamers,
> > > >>
> > > >> I'm forwarding this discussion & vote from the dev mailing list to
> the
> > > >> user mailing list.
> > > >> The goal is to have your feedback as user.
> > > >>
> > > >> Basically, we have two options:
> > > >> 1. Right now, in the PR, we support both Spark 1.x and 2.x using
> three
> > > >> artifacts (common, spark1, spark2). You, as users, pick up spark1 or
> > > spark2
> > > >> in your dependencies set depending the Spark target version you
> want.
> > > >> 2. The other option is to upgrade and focus on Spark 2.x in Beam
> > 2.3.0.
> > > >> If you still want to use Spark 1.x, then, you will be stuck up to
> Beam
> > > >> 2.2.0.
> > > >>
> > > >> Thoughts ?
> > > >>
> > > >> Thanks !
> > > >> Regards
> > > >> JB
> > > >>
> > > >>
> > > >> -------- Forwarded Message --------
> > > >> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
> > > >> Date: Wed, 8 Nov 2017 08:27:58 +0100
> > > >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > >> Reply-To: dev@beam.apache.org
> > > >> To: dev@beam.apache.org
> > > >>
> > > >> Hi all,
> > > >>
> > > >> as you might know, we are working on Spark 2.x support in the Spark
> > > >> runner.
> > > >>
> > > >> I'm working on a PR about that:
> > > >>
> > > >> https://github.com/apache/beam/pull/3808
> > > >>
> > > >> Today, we have something working with both Spark 1.x and 2.x from a
> > code
> > > >> standpoint, but I have to deal with dependencies. It's the first
> step
> > of
> > > >> the update as I'm still using RDD, the second step would be to
> support
> > > >> dataframe (but for that, I would need PCollection elements with
> > schemas,
> > > >> that's another topic on which Eugene, Reuven and I are discussing).
> > > >>
> > > >> However, as all major distributions now ship Spark 2.x, I don't
> think
> > > >> it's required anymore to support Spark 1.x.
> > > >>
> > > >> If we agree, I will update and cleanup the PR to only support and
> > focus
> > > >> on Spark 2.x.
> > > >>
> > > >> So, that's why I'm calling for a vote:
> > > >>
> > > >> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
> > > >> [ ] 0 (I don't care ;))
> > > >> [ ] -1, I would like to still support Spark 1.x, and so having
> > > support
> > > >> of both Spark 1.x and 2.x (please provide specific comment)
> > > >>
> > > >> This vote is open for 48 hours (I have the commits ready, just
> waiting
> > > >> the end of the vote to push on the PR).
> > > >>
> > > >> Thanks !
> > > >> Regards
> > > >> JB
> > > >>
> > > >
> > > > --
> > > > Jean-Baptiste Onofré
> > > > jbonofre@apache.org
> > > > http://blog.nanthrax.net
> > > > Talend - http://www.talend.com
> > > >
> > >
> >
> >
> >
> > --
> > -Ben
> >
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Reuven Lax <re...@google.com.INVALID>.
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
On Sat, Nov 18, 2017 at 1:54 AM, Ben Sidhom <si...@google.com.invalid>
wrote:
> [ ] Use Spark 1 & Spark 2 Support Branch
> [X] Use Spark 2 Only Branch
>
> On Fri, Nov 17, 2017 at 9:46 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > [ ] Use Spark 1 & Spark 2 Support Branch
> > [X] Use Spark 2 Only Branch
> >
> > On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi guys,
> > >
> > > To illustrate the current discussion about Spark versions support, you
> > can
> > > take a look on:
> > >
> > > --
> > > Spark 1 & Spark 2 Support Branch
> > >
> > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
> > >
> > > This branch contains a Spark runner common module compatible with both
> > > Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> > > modules/artifacts containing just a pom.xml to define the dependencies
> > set.
> > >
> > > --
> > > Spark 2 Only Branch
> > >
> > > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
> > >
> > > This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
> > >
> > > As I'm ready to merge one of the other in the PR, I would like to
> > complete
> > > the vote/discussion pretty soon.
> > >
> > > Correct me if I'm wrong, but it seems that the preference is to drop
> > Spark
> > > 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
> > >
> > > I would like to call a final vote to act the merge I will do:
> > >
> > > [ ] Use Spark 1 & Spark 2 Support Branch
> > > [ ] Use Spark 2 Only Branch
> > >
> > > This informal vote is open for 48 hours.
> > >
> > > Please, let me know what your preference is.
> > >
> > > Thanks !
> > > Regards
> > > JB
> > >
> > > On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
> > >
> > >> Hi Beamers,
> > >>
> > >> I'm forwarding this discussion & vote from the dev mailing list to the
> > >> user mailing list.
> > >> The goal is to have your feedback as user.
> > >>
> > >> Basically, we have two options:
> > >> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
> > >> artifacts (common, spark1, spark2). You, as users, pick up spark1 or
> > spark2
> > >> in your dependencies set depending the Spark target version you want.
> > >> 2. The other option is to upgrade and focus on Spark 2.x in Beam
> 2.3.0.
> > >> If you still want to use Spark 1.x, then, you will be stuck up to Beam
> > >> 2.2.0.
> > >>
> > >> Thoughts ?
> > >>
> > >> Thanks !
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> -------- Forwarded Message --------
> > >> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
> > >> Date: Wed, 8 Nov 2017 08:27:58 +0100
> > >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> > >> Reply-To: dev@beam.apache.org
> > >> To: dev@beam.apache.org
> > >>
> > >> Hi all,
> > >>
> > >> as you might know, we are working on Spark 2.x support in the Spark
> > >> runner.
> > >>
> > >> I'm working on a PR about that:
> > >>
> > >> https://github.com/apache/beam/pull/3808
> > >>
> > >> Today, we have something working with both Spark 1.x and 2.x from a
> code
> > >> standpoint, but I have to deal with dependencies. It's the first step
> of
> > >> the update as I'm still using RDD, the second step would be to support
> > >> dataframe (but for that, I would need PCollection elements with
> schemas,
> > >> that's another topic on which Eugene, Reuven and I are discussing).
> > >>
> > >> However, as all major distributions now ship Spark 2.x, I don't think
> > >> it's required anymore to support Spark 1.x.
> > >>
> > >> If we agree, I will update and cleanup the PR to only support and
> focus
> > >> on Spark 2.x.
> > >>
> > >> So, that's why I'm calling for a vote:
> > >>
> > >> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
> > >> [ ] 0 (I don't care ;))
> > >> [ ] -1, I would like to still support Spark 1.x, and so having
> > support
> > >> of both Spark 1.x and 2.x (please provide specific comment)
> > >>
> > >> This vote is open for 48 hours (I have the commits ready, just waiting
> > >> the end of the vote to push on the PR).
> > >>
> > >> Thanks !
> > >> Regards
> > >> JB
> > >>
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>
>
>
> --
> -Ben
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Ben Sidhom <si...@google.com.INVALID>.
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
On Fri, Nov 17, 2017 at 9:46 AM, Ted Yu <yu...@gmail.com> wrote:
> [ ] Use Spark 1 & Spark 2 Support Branch
> [X] Use Spark 2 Only Branch
>
> On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi guys,
> >
> > To illustrate the current discussion about Spark versions support, you
> can
> > take a look on:
> >
> > --
> > Spark 1 & Spark 2 Support Branch
> >
> > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
> >
> > This branch contains a Spark runner common module compatible with both
> > Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> > modules/artifacts containing just a pom.xml to define the dependencies
> set.
> >
> > --
> > Spark 2 Only Branch
> >
> > https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
> >
> > This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
> >
> > As I'm ready to merge one of the other in the PR, I would like to
> complete
> > the vote/discussion pretty soon.
> >
> > Correct me if I'm wrong, but it seems that the preference is to drop
> Spark
> > 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
> >
> > I would like to call a final vote to act the merge I will do:
> >
> > [ ] Use Spark 1 & Spark 2 Support Branch
> > [ ] Use Spark 2 Only Branch
> >
> > This informal vote is open for 48 hours.
> >
> > Please, let me know what your preference is.
> >
> > Thanks !
> > Regards
> > JB
> >
> > On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
> >
> >> Hi Beamers,
> >>
> >> I'm forwarding this discussion & vote from the dev mailing list to the
> >> user mailing list.
> >> The goal is to have your feedback as user.
> >>
> >> Basically, we have two options:
> >> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
> >> artifacts (common, spark1, spark2). You, as users, pick up spark1 or
> spark2
> >> in your dependencies set depending the Spark target version you want.
> >> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0.
> >> If you still want to use Spark 1.x, then, you will be stuck up to Beam
> >> 2.2.0.
> >>
> >> Thoughts ?
> >>
> >> Thanks !
> >> Regards
> >> JB
> >>
> >>
> >> -------- Forwarded Message --------
> >> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
> >> Date: Wed, 8 Nov 2017 08:27:58 +0100
> >> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> Reply-To: dev@beam.apache.org
> >> To: dev@beam.apache.org
> >>
> >> Hi all,
> >>
> >> as you might know, we are working on Spark 2.x support in the Spark
> >> runner.
> >>
> >> I'm working on a PR about that:
> >>
> >> https://github.com/apache/beam/pull/3808
> >>
> >> Today, we have something working with both Spark 1.x and 2.x from a code
> >> standpoint, but I have to deal with dependencies. It's the first step of
> >> the update as I'm still using RDD, the second step would be to support
> >> dataframe (but for that, I would need PCollection elements with schemas,
> >> that's another topic on which Eugene, Reuven and I are discussing).
> >>
> >> However, as all major distributions now ship Spark 2.x, I don't think
> >> it's required anymore to support Spark 1.x.
> >>
> >> If we agree, I will update and cleanup the PR to only support and focus
> >> on Spark 2.x.
> >>
> >> So, that's why I'm calling for a vote:
> >>
> >> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
> >> [ ] 0 (I don't care ;))
> >> [ ] -1, I would like to still support Spark 1.x, and so having
> support
> >> of both Spark 1.x and 2.x (please provide specific comment)
> >>
> >> This vote is open for 48 hours (I have the commits ready, just waiting
> >> the end of the vote to push on the PR).
> >>
> >> Thanks !
> >> Regards
> >> JB
> >>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
--
-Ben
Re: [VOTE] Choose the "new" Spark runner
Posted by Ted Yu <yu...@gmail.com>.
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi guys,
>
> To illustrate the current discussion about Spark versions support, you can
> take a look on:
>
> --
> Spark 1 & Spark 2 Support Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
>
> This branch contains a Spark runner common module compatible with both
> Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> modules/artifacts containing just a pom.xml to define the dependencies set.
>
> --
> Spark 2 Only Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
>
> This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
>
> As I'm ready to merge one of the other in the PR, I would like to complete
> the vote/discussion pretty soon.
>
> Correct me if I'm wrong, but it seems that the preference is to drop Spark
> 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
>
> I would like to call a final vote to act the merge I will do:
>
> [ ] Use Spark 1 & Spark 2 Support Branch
> [ ] Use Spark 2 Only Branch
>
> This informal vote is open for 48 hours.
>
> Please, let me know what your preference is.
>
> Thanks !
> Regards
> JB
>
> On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
>
>> Hi Beamers,
>>
>> I'm forwarding this discussion & vote from the dev mailing list to the
>> user mailing list.
>> The goal is to have your feedback as user.
>>
>> Basically, we have two options:
>> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
>> artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2
>> in your dependencies set depending the Spark target version you want.
>> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0.
>> If you still want to use Spark 1.x, then, you will be stuck up to Beam
>> 2.2.0.
>>
>> Thoughts ?
>>
>> Thanks !
>> Regards
>> JB
>>
>>
>> -------- Forwarded Message --------
>> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
>> Date: Wed, 8 Nov 2017 08:27:58 +0100
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Reply-To: dev@beam.apache.org
>> To: dev@beam.apache.org
>>
>> Hi all,
>>
>> as you might know, we are working on Spark 2.x support in the Spark
>> runner.
>>
>> I'm working on a PR about that:
>>
>> https://github.com/apache/beam/pull/3808
>>
>> Today, we have something working with both Spark 1.x and 2.x from a code
>> standpoint, but I have to deal with dependencies. It's the first step of
>> the update as I'm still using RDD, the second step would be to support
>> dataframe (but for that, I would need PCollection elements with schemas,
>> that's another topic on which Eugene, Reuven and I are discussing).
>>
>> However, as all major distributions now ship Spark 2.x, I don't think
>> it's required anymore to support Spark 1.x.
>>
>> If we agree, I will update and cleanup the PR to only support and focus
>> on Spark 2.x.
>>
>> So, that's why I'm calling for a vote:
>>
>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
>> [ ] 0 (I don't care ;))
>> [ ] -1, I would like to still support Spark 1.x, and so having support
>> of both Spark 1.x and 2.x (please provide specific comment)
>>
>> This vote is open for 48 hours (I have the commits ready, just waiting
>> the end of the vote to push on the PR).
>>
>> Thanks !
>> Regards
>> JB
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Kenneth Knowles <kl...@google.com>.
>
> [ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
Kenn
Re: [VOTE] Choose the "new" Spark runner
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
For your convenience, here's the compare view:
https://github.com/apache/beam/compare/master...jbonofre:BEAM-1920-SPARK2-MODULES
https://github.com/apache/beam/compare/master...jbonofre:BEAM-1920-SPARK2-ONLY
Kenn
On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi guys,
>
> To illustrate the current discussion about Spark versions support, you can
> take a look on:
>
> --
> Spark 1 & Spark 2 Support Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
>
> This branch contains a Spark runner common module compatible with both
> Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> modules/artifacts containing just a pom.xml to define the dependencies set.
>
> --
> Spark 2 Only Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
>
> This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
>
> As I'm ready to merge one of the other in the PR, I would like to complete
> the vote/discussion pretty soon.
>
> Correct me if I'm wrong, but it seems that the preference is to drop Spark
> 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
>
> I would like to call a final vote to act the merge I will do:
>
> [ ] Use Spark 1 & Spark 2 Support Branch
> [ ] Use Spark 2 Only Branch
>
> This informal vote is open for 48 hours.
>
> Please, let me know what your preference is.
>
> Thanks !
> Regards
> JB
>
> On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
>
>> Hi Beamers,
>>
>> I'm forwarding this discussion & vote from the dev mailing list to the
>> user mailing list.
>> The goal is to have your feedback as user.
>>
>> Basically, we have two options:
>> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
>> artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2
>> in your dependencies set depending the Spark target version you want.
>> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0.
>> If you still want to use Spark 1.x, then, you will be stuck up to Beam
>> 2.2.0.
>>
>> Thoughts ?
>>
>> Thanks !
>> Regards
>> JB
>>
>>
>> -------- Forwarded Message --------
>> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
>> Date: Wed, 8 Nov 2017 08:27:58 +0100
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Reply-To: dev@beam.apache.org
>> To: dev@beam.apache.org
>>
>> Hi all,
>>
>> as you might know, we are working on Spark 2.x support in the Spark
>> runner.
>>
>> I'm working on a PR about that:
>>
>> https://github.com/apache/beam/pull/3808
>>
>> Today, we have something working with both Spark 1.x and 2.x from a code
>> standpoint, but I have to deal with dependencies. It's the first step of
>> the update as I'm still using RDD, the second step would be to support
>> dataframe (but for that, I would need PCollection elements with schemas,
>> that's another topic on which Eugene, Reuven and I are discussing).
>>
>> However, as all major distributions now ship Spark 2.x, I don't think
>> it's required anymore to support Spark 1.x.
>>
>> If we agree, I will update and cleanup the PR to only support and focus
>> on Spark 2.x.
>>
>> So, that's why I'm calling for a vote:
>>
>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
>> [ ] 0 (I don't care ;))
>> [ ] -1, I would like to still support Spark 1.x, and so having support
>> of both Spark 1.x and 2.x (please provide specific comment)
>>
>> This vote is open for 48 hours (I have the commits ready, just waiting
>> the end of the vote to push on the PR).
>>
>> Thanks !
>> Regards
>> JB
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Luke cage <lu...@gmail.com>.
[ ] Use Spark 1 & Spark 2 Support Branch
[ X] Use Spark 2 Only Branch
On Thu, Nov 16, 2017 at 5:08 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi guys,
>
> To illustrate the current discussion about Spark versions support, you can
> take a look on:
>
> --
> Spark 1 & Spark 2 Support Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
>
> This branch contains a Spark runner common module compatible with both
> Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> modules/artifacts containing just a pom.xml to define the dependencies set.
>
> --
> Spark 2 Only Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
>
> This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
>
> As I'm ready to merge one of the other in the PR, I would like to complete
> the vote/discussion pretty soon.
>
> Correct me if I'm wrong, but it seems that the preference is to drop Spark
> 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
>
> I would like to call a final vote to act the merge I will do:
>
> [ ] Use Spark 1 & Spark 2 Support Branch
> [ ] Use Spark 2 Only Branch
>
> This informal vote is open for 48 hours.
>
> Please, let me know what your preference is.
>
> Thanks !
> Regards
> JB
>
> On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
>
>> Hi Beamers,
>>
>> I'm forwarding this discussion & vote from the dev mailing list to the
>> user mailing list.
>> The goal is to have your feedback as user.
>>
>> Basically, we have two options:
>> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
>> artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2
>> in your dependencies set depending the Spark target version you want.
>> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0.
>> If you still want to use Spark 1.x, then, you will be stuck up to Beam
>> 2.2.0.
>>
>> Thoughts ?
>>
>> Thanks !
>> Regards
>> JB
>>
>>
>> -------- Forwarded Message --------
>> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
>> Date: Wed, 8 Nov 2017 08:27:58 +0100
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Reply-To: dev@beam.apache.org
>> To: dev@beam.apache.org
>>
>> Hi all,
>>
>> as you might know, we are working on Spark 2.x support in the Spark
>> runner.
>>
>> I'm working on a PR about that:
>>
>> https://github.com/apache/beam/pull/3808
>>
>> Today, we have something working with both Spark 1.x and 2.x from a code
>> standpoint, but I have to deal with dependencies. It's the first step of
>> the update as I'm still using RDD, the second step would be to support
>> dataframe (but for that, I would need PCollection elements with schemas,
>> that's another topic on which Eugene, Reuven and I are discussing).
>>
>> However, as all major distributions now ship Spark 2.x, I don't think
>> it's required anymore to support Spark 1.x.
>>
>> If we agree, I will update and cleanup the PR to only support and focus
>> on Spark 2.x.
>>
>> So, that's why I'm calling for a vote:
>>
>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
>> [ ] 0 (I don't care ;))
>> [ ] -1, I would like to still support Spark 1.x, and so having support
>> of both Spark 1.x and 2.x (please provide specific comment)
>>
>> This vote is open for 48 hours (I have the commits ready, just waiting
>> the end of the vote to push on the PR).
>>
>> Thanks !
>> Regards
>> JB
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Jacob Marble <jm...@kochava.com>.
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
Spark 2 has been out for a while, so probably not going to offend many
people.
Jacob
On Thu, Nov 16, 2017 at 5:45 AM, Neville Dipale <ne...@gmail.com>
wrote:
> [ ] Use Spark 1 & Spark 2 Support Branch
> [X] Use Spark 2 Only Branch
>
> On 16 November 2017 at 15:08, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi guys,
>>
>> To illustrate the current discussion about Spark versions support, you
>> can take a look on:
>>
>> --
>> Spark 1 & Spark 2 Support Branch
>>
>> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
>>
>> This branch contains a Spark runner common module compatible with both
>> Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
>> modules/artifacts containing just a pom.xml to define the dependencies set.
>>
>> --
>> Spark 2 Only Branch
>>
>> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
>>
>> This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
>>
>> As I'm ready to merge one of the other in the PR, I would like to
>> complete the vote/discussion pretty soon.
>>
>> Correct me if I'm wrong, but it seems that the preference is to drop
>> Spark 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
>>
>> I would like to call a final vote to act the merge I will do:
>>
>> [ ] Use Spark 1 & Spark 2 Support Branch
>> [ ] Use Spark 2 Only Branch
>>
>> This informal vote is open for 48 hours.
>>
>> Please, let me know what your preference is.
>>
>> Thanks !
>> Regards
>> JB
>>
>> On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
>>
>>> Hi Beamers,
>>>
>>> I'm forwarding this discussion & vote from the dev mailing list to the
>>> user mailing list.
>>> The goal is to have your feedback as user.
>>>
>>> Basically, we have two options:
>>> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
>>> artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2
>>> in your dependencies set depending the Spark target version you want.
>>> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0.
>>> If you still want to use Spark 1.x, then, you will be stuck up to Beam
>>> 2.2.0.
>>>
>>> Thoughts ?
>>>
>>> Thanks !
>>> Regards
>>> JB
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
>>> Date: Wed, 8 Nov 2017 08:27:58 +0100
>>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> Reply-To: dev@beam.apache.org
>>> To: dev@beam.apache.org
>>>
>>> Hi all,
>>>
>>> as you might know, we are working on Spark 2.x support in the Spark
>>> runner.
>>>
>>> I'm working on a PR about that:
>>>
>>> https://github.com/apache/beam/pull/3808
>>>
>>> Today, we have something working with both Spark 1.x and 2.x from a code
>>> standpoint, but I have to deal with dependencies. It's the first step of
>>> the update as I'm still using RDD, the second step would be to support
>>> dataframe (but for that, I would need PCollection elements with schemas,
>>> that's another topic on which Eugene, Reuven and I are discussing).
>>>
>>> However, as all major distributions now ship Spark 2.x, I don't think
>>> it's required anymore to support Spark 1.x.
>>>
>>> If we agree, I will update and cleanup the PR to only support and focus
>>> on Spark 2.x.
>>>
>>> So, that's why I'm calling for a vote:
>>>
>>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
>>> [ ] 0 (I don't care ;))
>>> [ ] -1, I would like to still support Spark 1.x, and so having
>>> support of both Spark 1.x and 2.x (please provide specific comment)
>>>
>>> This vote is open for 48 hours (I have the commits ready, just waiting
>>> the end of the vote to push on the PR).
>>>
>>> Thanks !
>>> Regards
>>> JB
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Neville Dipale <ne...@gmail.com>.
[ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
On 16 November 2017 at 15:08, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi guys,
>
> To illustrate the current discussion about Spark versions support, you can
> take a look on:
>
> --
> Spark 1 & Spark 2 Support Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-MODULES
>
> This branch contains a Spark runner common module compatible with both
> Spark 1.x and 2.x. For convenience, we introduced spark1 & spark2
> modules/artifacts containing just a pom.xml to define the dependencies set.
>
> --
> Spark 2 Only Branch
>
> https://github.com/jbonofre/beam/tree/BEAM-1920-SPARK2-ONLY
>
> This branch is an upgrade to Spark 2.x and "drop" support of Spark 1.x.
>
> As I'm ready to merge one of the other in the PR, I would like to complete
> the vote/discussion pretty soon.
>
> Correct me if I'm wrong, but it seems that the preference is to drop Spark
> 1.x to focus only on Spark 2.x (for the Spark 2 Only Branch).
>
> I would like to call a final vote to act the merge I will do:
>
> [ ] Use Spark 1 & Spark 2 Support Branch
> [ ] Use Spark 2 Only Branch
>
> This informal vote is open for 48 hours.
>
> Please, let me know what your preference is.
>
> Thanks !
> Regards
> JB
>
> On 11/13/2017 09:32 AM, Jean-Baptiste Onofré wrote:
>
>> Hi Beamers,
>>
>> I'm forwarding this discussion & vote from the dev mailing list to the
>> user mailing list.
>> The goal is to have your feedback as user.
>>
>> Basically, we have two options:
>> 1. Right now, in the PR, we support both Spark 1.x and 2.x using three
>> artifacts (common, spark1, spark2). You, as users, pick up spark1 or spark2
>> in your dependencies set depending the Spark target version you want.
>> 2. The other option is to upgrade and focus on Spark 2.x in Beam 2.3.0.
>> If you still want to use Spark 1.x, then, you will be stuck up to Beam
>> 2.2.0.
>>
>> Thoughts ?
>>
>> Thanks !
>> Regards
>> JB
>>
>>
>> -------- Forwarded Message --------
>> Subject: [VOTE] Drop Spark 1.x support to focus on Spark 2.x
>> Date: Wed, 8 Nov 2017 08:27:58 +0100
>> From: Jean-Baptiste Onofré <jb...@nanthrax.net>
>> Reply-To: dev@beam.apache.org
>> To: dev@beam.apache.org
>>
>> Hi all,
>>
>> as you might know, we are working on Spark 2.x support in the Spark
>> runner.
>>
>> I'm working on a PR about that:
>>
>> https://github.com/apache/beam/pull/3808
>>
>> Today, we have something working with both Spark 1.x and 2.x from a code
>> standpoint, but I have to deal with dependencies. It's the first step of
>> the update as I'm still using RDD, the second step would be to support
>> dataframe (but for that, I would need PCollection elements with schemas,
>> that's another topic on which Eugene, Reuven and I are discussing).
>>
>> However, as all major distributions now ship Spark 2.x, I don't think
>> it's required anymore to support Spark 1.x.
>>
>> If we agree, I will update and cleanup the PR to only support and focus
>> on Spark 2.x.
>>
>> So, that's why I'm calling for a vote:
>>
>> [ ] +1 to drop Spark 1.x support and upgrade to Spark 2.x only
>> [ ] 0 (I don't care ;))
>> [ ] -1, I would like to still support Spark 1.x, and so having support
>> of both Spark 1.x and 2.x (please provide specific comment)
>>
>> This vote is open for 48 hours (I have the commits ready, just waiting
>> the end of the vote to push on the PR).
>>
>> Thanks !
>> Regards
>> JB
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Choose the "new" Spark runner
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
>
> [ ] Use Spark 1 & Spark 2 Support Branch
[X] Use Spark 2 Only Branch
Kenn