You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Vishwas Bm <bm...@gmail.com> on 2018/12/06 06:39:13 UTC
Moving to spark 2.4
Hi,
Currently I see that the spark version dependency used in Beam is "2.3.2".
As spark 2.4 is released now, is there a plan to upgrade Beam spark
dependency ?
*Thanks & Regards,*
*Vishwas *
*Mob : 9164886653*
Re: Moving to spark 2.4
Posted by Tim Robertson <ti...@gmail.com>.
To clarify Ismaël's comment
Cloudera repo indicates Cloudera 6.1 will have spark 2.4 but CDH is
currently still on 6.0.
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/2.4.0-cdh6.1.0/
With the HWX / Cloudera merger the release cycle is not announced but 6.1
will likely not be there until early 2019. Also note that many folk will
still be on CDH 5 as CDH 6 is relatively new.
On Fri, Dec 7, 2018 at 5:06 PM Ismaël Mejía <ie...@gmail.com> wrote:
> It seems that Cloudera has it now, not sure if worth to wait for the
> Hortonworks maybe worth waiting for EMR.
>
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/
>
> A pro move to Spark 2.4.0 argument is for the future oriented (non
> hadoop friends), because the support for kubernetes has improved a lot
> in this release.
>
> On Fri, Dec 7, 2018 at 4:56 PM David Morávek <da...@gmail.com>
> wrote:
> >
> > +1 for waiting for HDP and CDH adoption
> >
> > Sent from my iPhone
> >
> > On 7 Dec 2018, at 16:38, Alexey Romanenko <ar...@gmail.com>
> wrote:
> >
> > I agree with Ismael and I’d wait until the new Spark version will be
> supported by major BigData distributors.
> >
> > On 7 Dec 2018, at 14:57, Vishwas Bm <bm...@gmail.com> wrote:
> >
> > Hi Ismael,
> >
> > We have upgraded the spark to 2.4.
> > In our setup we had run few basic tests and found it to be pretty stable.
> >
> >
> > Thanks & Regards,
> > Vishwas
> >
> >
> > On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <ie...@gmail.com> wrote:
> >>
> >> Hello Vishwas,
> >>
> >> The spark dependency in the spark runner is provided so you can
> >> already pass the dependencies of spark 2.4 and it should work out of
> >> the box.
> >>
> >> JB did a PR to upgrade the version of Spark in the runner, but maybe
> >> it is worth to wait a bit before merging it, at least until some of
> >> the Big Data distributions has spark 2.4.x support available, so far
> >> nobody has upgraded it (well apart of databricks).
> >>
> >> What do others think, should we move ahead or are you aware of any
> >> issue introduced by version 2.4.0? (Notice that the PR just updates
> >> the version so code compatibility should be ok).
> >>
> >> Ismaël
> >>
> >> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >> >
> >> > Hi Vishwas
> >> >
> >> > Yes, I already started the update.
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 06/12/2018 07:39, Vishwas Bm wrote:
> >> > > Hi,
> >> > >
> >> > > Currently I see that the spark version dependency used in Beam is
> >> > > //"2.3.2".
> >> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> >> > > dependency ?
> >> > >
> >> > >
> >> > > *Thanks & Regards,*
> >> > > *Vishwas
> >> > > *
> >> > > *Mob : 9164886653*
> >> >
> >> > --
> >> > Jean-Baptiste Onofré
> >> > jbonofre@apache.org
> >> > http://blog.nanthrax.net
> >> > Talend - http://www.talend.com
> >
> >
>
Re: Moving to spark 2.4
Posted by Ismaël Mejía <ie...@gmail.com>.
It seems that Cloudera has it now, not sure if worth to wait for the
Hortonworks maybe worth waiting for EMR.
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.11/
A pro move to Spark 2.4.0 argument is for the future oriented (non
hadoop friends), because the support for kubernetes has improved a lot
in this release.
On Fri, Dec 7, 2018 at 4:56 PM David Morávek <da...@gmail.com> wrote:
>
> +1 for waiting for HDP and CDH adoption
>
> Sent from my iPhone
>
> On 7 Dec 2018, at 16:38, Alexey Romanenko <ar...@gmail.com> wrote:
>
> I agree with Ismael and I’d wait until the new Spark version will be supported by major BigData distributors.
>
> On 7 Dec 2018, at 14:57, Vishwas Bm <bm...@gmail.com> wrote:
>
> Hi Ismael,
>
> We have upgraded the spark to 2.4.
> In our setup we had run few basic tests and found it to be pretty stable.
>
>
> Thanks & Regards,
> Vishwas
>
>
> On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>> Hello Vishwas,
>>
>> The spark dependency in the spark runner is provided so you can
>> already pass the dependencies of spark 2.4 and it should work out of
>> the box.
>>
>> JB did a PR to upgrade the version of Spark in the runner, but maybe
>> it is worth to wait a bit before merging it, at least until some of
>> the Big Data distributions has spark 2.4.x support available, so far
>> nobody has upgraded it (well apart of databricks).
>>
>> What do others think, should we move ahead or are you aware of any
>> issue introduced by version 2.4.0? (Notice that the PR just updates
>> the version so code compatibility should be ok).
>>
>> Ismaël
>>
>> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>> >
>> > Hi Vishwas
>> >
>> > Yes, I already started the update.
>> >
>> > Regards
>> > JB
>> >
>> > On 06/12/2018 07:39, Vishwas Bm wrote:
>> > > Hi,
>> > >
>> > > Currently I see that the spark version dependency used in Beam is
>> > > //"2.3.2".
>> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
>> > > dependency ?
>> > >
>> > >
>> > > *Thanks & Regards,*
>> > > *Vishwas
>> > > *
>> > > *Mob : 9164886653*
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>
>
Re: Moving to spark 2.4
Posted by David Morávek <da...@gmail.com>.
+1 for waiting for HDP and CDH adoption
Sent from my iPhone
> On 7 Dec 2018, at 16:38, Alexey Romanenko <ar...@gmail.com> wrote:
>
> I agree with Ismael and I’d wait until the new Spark version will be supported by major BigData distributors.
>
>> On 7 Dec 2018, at 14:57, Vishwas Bm <bm...@gmail.com> wrote:
>>
>> Hi Ismael,
>>
>> We have upgraded the spark to 2.4.
>> In our setup we had run few basic tests and found it to be pretty stable.
>>
>>
>> Thanks & Regards,
>> Vishwas
>>
>>
>>> On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>> Hello Vishwas,
>>>
>>> The spark dependency in the spark runner is provided so you can
>>> already pass the dependencies of spark 2.4 and it should work out of
>>> the box.
>>>
>>> JB did a PR to upgrade the version of Spark in the runner, but maybe
>>> it is worth to wait a bit before merging it, at least until some of
>>> the Big Data distributions has spark 2.4.x support available, so far
>>> nobody has upgraded it (well apart of databricks).
>>>
>>> What do others think, should we move ahead or are you aware of any
>>> issue introduced by version 2.4.0? (Notice that the PR just updates
>>> the version so code compatibility should be ok).
>>>
>>> Ismaël
>>>
>>> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>> >
>>> > Hi Vishwas
>>> >
>>> > Yes, I already started the update.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 06/12/2018 07:39, Vishwas Bm wrote:
>>> > > Hi,
>>> > >
>>> > > Currently I see that the spark version dependency used in Beam is
>>> > > //"2.3.2".
>>> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
>>> > > dependency ?
>>> > >
>>> > >
>>> > > *Thanks & Regards,*
>>> > > *Vishwas
>>> > > *
>>> > > *Mob : 9164886653*
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>
Re: Moving to spark 2.4
Posted by Alexey Romanenko <ar...@gmail.com>.
I agree with Ismael and I’d wait until the new Spark version will be supported by major BigData distributors.
> On 7 Dec 2018, at 14:57, Vishwas Bm <bm...@gmail.com> wrote:
>
> Hi Ismael,
>
> We have upgraded the spark to 2.4.
> In our setup we had run few basic tests and found it to be pretty stable.
>
>
> Thanks & Regards,
> Vishwas
>
>
> On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <iemejia@gmail.com <ma...@gmail.com>> wrote:
> Hello Vishwas,
>
> The spark dependency in the spark runner is provided so you can
> already pass the dependencies of spark 2.4 and it should work out of
> the box.
>
> JB did a PR to upgrade the version of Spark in the runner, but maybe
> it is worth to wait a bit before merging it, at least until some of
> the Big Data distributions has spark 2.4.x support available, so far
> nobody has upgraded it (well apart of databricks).
>
> What do others think, should we move ahead or are you aware of any
> issue introduced by version 2.4.0? (Notice that the PR just updates
> the version so code compatibility should be ok).
>
> Ismaël
>
> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > Hi Vishwas
> >
> > Yes, I already started the update.
> >
> > Regards
> > JB
> >
> > On 06/12/2018 07:39, Vishwas Bm wrote:
> > > Hi,
> > >
> > > Currently I see that the spark version dependency used in Beam is
> > > //"2.3.2".
> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> > > dependency ?
> > >
> > >
> > > *Thanks & Regards,*
> > > *Vishwas
> > > *
> > > *Mob : 9164886653*
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net <http://blog.nanthrax.net/>
> > Talend - http://www.talend.com <http://www.talend.com/>
Re: Moving to spark 2.4
Posted by Vishwas Bm <bm...@gmail.com>.
Hi Ismael,
We have upgraded the spark to 2.4.
In our setup we had run few basic tests and found it to be pretty stable.
*Thanks & Regards,*
*Vishwas *
On Fri, Dec 7, 2018 at 2:53 PM Ismaël Mejía <ie...@gmail.com> wrote:
> Hello Vishwas,
>
> The spark dependency in the spark runner is provided so you can
> already pass the dependencies of spark 2.4 and it should work out of
> the box.
>
> JB did a PR to upgrade the version of Spark in the runner, but maybe
> it is worth to wait a bit before merging it, at least until some of
> the Big Data distributions has spark 2.4.x support available, so far
> nobody has upgraded it (well apart of databricks).
>
> What do others think, should we move ahead or are you aware of any
> issue introduced by version 2.4.0? (Notice that the PR just updates
> the version so code compatibility should be ok).
>
> Ismaël
>
> On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > Hi Vishwas
> >
> > Yes, I already started the update.
> >
> > Regards
> > JB
> >
> > On 06/12/2018 07:39, Vishwas Bm wrote:
> > > Hi,
> > >
> > > Currently I see that the spark version dependency used in Beam is
> > > //"2.3.2".
> > > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> > > dependency ?
> > >
> > >
> > > *Thanks & Regards,*
> > > *Vishwas
> > > *
> > > *Mob : 9164886653*
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
Re: Moving to spark 2.4
Posted by Ismaël Mejía <ie...@gmail.com>.
Hello Vishwas,
The spark dependency in the spark runner is provided so you can
already pass the dependencies of spark 2.4 and it should work out of
the box.
JB did a PR to upgrade the version of Spark in the runner, but maybe
it is worth to wait a bit before merging it, at least until some of
the Big Data distributions has spark 2.4.x support available, so far
nobody has upgraded it (well apart of databricks).
What do others think, should we move ahead or are you aware of any
issue introduced by version 2.4.0? (Notice that the PR just updates
the version so code compatibility should be ok).
Ismaël
On Thu, Dec 6, 2018 at 12:14 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi Vishwas
>
> Yes, I already started the update.
>
> Regards
> JB
>
> On 06/12/2018 07:39, Vishwas Bm wrote:
> > Hi,
> >
> > Currently I see that the spark version dependency used in Beam is
> > //"2.3.2".
> > As spark 2.4 is released now, is there a plan to upgrade Beam spark
> > dependency ?
> >
> >
> > *Thanks & Regards,*
> > *Vishwas
> > *
> > *Mob : 9164886653*
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: Moving to spark 2.4
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Vishwas
Yes, I already started the update.
Regards
JB
On 06/12/2018 07:39, Vishwas Bm wrote:
> Hi,
>
> Currently I see that the spark version dependency used in Beam is
> //"2.3.2".
> As spark 2.4 is released now, is there a plan to upgrade Beam spark
> dependency ?
>
>
> *Thanks & Regards,*
> *Vishwas
> *
> *Mob : 9164886653*
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com