You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Gary Dusbabek <gd...@gmail.com> on 2018/02/22 16:55:27 UTC
spark runner for Scala 2.10
Hi,
My apologies if this belongs on the dev list. If it does, let me know and
I'll shoot things over that way...
For the last day or so, I've been trying to create a Spark Runner that will
work on older deployments using Scala 2.10. I've taken a few approaches:
1. selectively changing a few dependencies in beam-runners-spark.pom (and a
few other places in the parent)
2. updating every dependency that references *_2.11 to be *_2.10
In the former case the sticking point in both cases is that there is a
library incompatibility with jackson-module-scala_2.xx. In the latter case
there is a problem with SourceRDD.SourcePartitioning not [correctly]
implementing `equals(...)` from the parent trait.
Posts on the mailing list made me think that the move to Scala 2.11 started
only last fall, so I figured it should be easy to make the switch back.
However, I have a feeling that it could be the case that I just don't
understand the Beam build system well enough to produce the right outcome
(a custom version that can be used with older Scala).
Is there a correct or better way of achieving this?
Kind Regards,
Gary Dusbabek
Re: spark runner for Scala 2.10
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Let me create a branch on my github and share it with you. Let's continue the
discussion with direct message (to avoid to flood the mailing list).
Regards
JB
On 02/22/2018 06:51 PM, Gary Dusbabek wrote:
> Yes, that would be very helpful. If possible, I'd like to understand how it is
> constructed so that I can maintain it. A link to a git repo would be great.
>
> I've spent some time trying to understand how the Beam project is built/managed.
> It looks like the poms are intended primarily for developers and packaging,
> while the gradle components are intended for CI, etc.
>
> Kind Regards,
>
> Gary.
>
> On Thu, Feb 22, 2018 at 5:45 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> OK, do you want me to provide a Scala 2.10 build for you ?
>
> Regards
> JB
>
> On 02/22/2018 06:44 PM, Gary Dusbabek wrote:
> > Jean-Baptiste,
> >
> > Thanks for responding. I agree--it would be better to use Scala 2.11. I'm in the
> > process of creating a Beam POC with an existing platform and upgrading
> > everything in that platform to Scala 2.11 as a prerequisite is out of scope.
> >
> > It would be helpful to know if Beam in it's current state is backward
> > incompatible with Scala 2.10 for reasons other than the dependencies.
> >
> > But if there is a way to make it work to enable a POC, I would appreciate some
> > pointers, as it doesn't seem to be as simple as changing the "*_2.11" references
> > in the poms.
> >
> > Cheers,
> >
> > Gary.
> >
> >
> >
> > On Thu, Feb 22, 2018 at 5:34 PM, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> >
> > Hi Gary,
> >
> > Beam 2.3.0 and the Spark runner use Scala 2.11.
> >
> > I can help you to have a smooth transition by creating a local branch
> using
> > Scala 2.10. However, I strongly advice to upgrade to 2.11 as some
> other part of
> > Beam (other runners and IOs) use 2.11 already.
> >
> > Regards
> > JB
> >
> > On 02/22/2018 05:55 PM, Gary Dusbabek wrote:
> > > Hi,
> > >
> > > My apologies if this belongs on the dev list. If it does, let me
> know and I'll
> > > shoot things over that way...
> > >
> > > For the last day or so, I've been trying to create a Spark Runner that
> > will work
> > > on older deployments using Scala 2.10. I've taken a few approaches:
> > >
> > > 1. selectively changing a few dependencies in beam-runners-spark.pom
> (and
> > a few
> > > other places in the parent)
> > > 2. updating every dependency that references *_2.11 to be *_2.10
> > >
> > > In the former case the sticking point in both cases is that there is
> a library
> > > incompatibility with jackson-module-scala_2.xx. In the latter case
> there is a
> > > problem with SourceRDD.SourcePartitioning not [correctly] implementing
> > > `equals(...)` from the parent trait.
> > >
> > > Posts on the mailing list made me think that the move to Scala 2.11
> > started only
> > > last fall, so I figured it should be easy to make the switch back.
> > >
> > > However, I have a feeling that it could be the case that I just don't
> > understand
> > > the Beam build system well enough to produce the right outcome (a custom
> > version
> > > that can be used with older Scala).
> > >
> > > Is there a correct or better way of achieving this?
> > >
> > > Kind Regards,
> > >
> > > Gary Dusbabek
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: spark runner for Scala 2.10
Posted by Gary Dusbabek <gd...@gmail.com>.
Yes, that would be very helpful. If possible, I'd like to understand how it
is constructed so that I can maintain it. A link to a git repo would be
great.
I've spent some time trying to understand how the Beam project is
built/managed. It looks like the poms are intended primarily for developers
and packaging, while the gradle components are intended for CI, etc.
Kind Regards,
Gary.
On Thu, Feb 22, 2018 at 5:45 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> OK, do you want me to provide a Scala 2.10 build for you ?
>
> Regards
> JB
>
> On 02/22/2018 06:44 PM, Gary Dusbabek wrote:
> > Jean-Baptiste,
> >
> > Thanks for responding. I agree--it would be better to use Scala 2.11.
> I'm in the
> > process of creating a Beam POC with an existing platform and upgrading
> > everything in that platform to Scala 2.11 as a prerequisite is out of
> scope.
> >
> > It would be helpful to know if Beam in it's current state is backward
> > incompatible with Scala 2.10 for reasons other than the dependencies.
> >
> > But if there is a way to make it work to enable a POC, I would
> appreciate some
> > pointers, as it doesn't seem to be as simple as changing the "*_2.11"
> references
> > in the poms.
> >
> > Cheers,
> >
> > Gary.
> >
> >
> >
> > On Thu, Feb 22, 2018 at 5:34 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> >
> > Hi Gary,
> >
> > Beam 2.3.0 and the Spark runner use Scala 2.11.
> >
> > I can help you to have a smooth transition by creating a local
> branch using
> > Scala 2.10. However, I strongly advice to upgrade to 2.11 as some
> other part of
> > Beam (other runners and IOs) use 2.11 already.
> >
> > Regards
> > JB
> >
> > On 02/22/2018 05:55 PM, Gary Dusbabek wrote:
> > > Hi,
> > >
> > > My apologies if this belongs on the dev list. If it does, let me
> know and I'll
> > > shoot things over that way...
> > >
> > > For the last day or so, I've been trying to create a Spark Runner
> that
> > will work
> > > on older deployments using Scala 2.10. I've taken a few approaches:
> > >
> > > 1. selectively changing a few dependencies in
> beam-runners-spark.pom (and
> > a few
> > > other places in the parent)
> > > 2. updating every dependency that references *_2.11 to be *_2.10
> > >
> > > In the former case the sticking point in both cases is that there
> is a library
> > > incompatibility with jackson-module-scala_2.xx. In the latter
> case there is a
> > > problem with SourceRDD.SourcePartitioning not [correctly]
> implementing
> > > `equals(...)` from the parent trait.
> > >
> > > Posts on the mailing list made me think that the move to Scala 2.11
> > started only
> > > last fall, so I figured it should be easy to make the switch back.
> > >
> > > However, I have a feeling that it could be the case that I just
> don't
> > understand
> > > the Beam build system well enough to produce the right outcome (a
> custom
> > version
> > > that can be used with older Scala).
> > >
> > > Is there a correct or better way of achieving this?
> > >
> > > Kind Regards,
> > >
> > > Gary Dusbabek
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: spark runner for Scala 2.10
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
OK, do you want me to provide a Scala 2.10 build for you ?
Regards
JB
On 02/22/2018 06:44 PM, Gary Dusbabek wrote:
> Jean-Baptiste,
>
> Thanks for responding. I agree--it would be better to use Scala 2.11. I'm in the
> process of creating a Beam POC with an existing platform and upgrading
> everything in that platform to Scala 2.11 as a prerequisite is out of scope.
>
> It would be helpful to know if Beam in it's current state is backward
> incompatible with Scala 2.10 for reasons other than the dependencies.
>
> But if there is a way to make it work to enable a POC, I would appreciate some
> pointers, as it doesn't seem to be as simple as changing the "*_2.11" references
> in the poms.
>
> Cheers,
>
> Gary.
>
>
>
> On Thu, Feb 22, 2018 at 5:34 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Hi Gary,
>
> Beam 2.3.0 and the Spark runner use Scala 2.11.
>
> I can help you to have a smooth transition by creating a local branch using
> Scala 2.10. However, I strongly advice to upgrade to 2.11 as some other part of
> Beam (other runners and IOs) use 2.11 already.
>
> Regards
> JB
>
> On 02/22/2018 05:55 PM, Gary Dusbabek wrote:
> > Hi,
> >
> > My apologies if this belongs on the dev list. If it does, let me know and I'll
> > shoot things over that way...
> >
> > For the last day or so, I've been trying to create a Spark Runner that
> will work
> > on older deployments using Scala 2.10. I've taken a few approaches:
> >
> > 1. selectively changing a few dependencies in beam-runners-spark.pom (and
> a few
> > other places in the parent)
> > 2. updating every dependency that references *_2.11 to be *_2.10
> >
> > In the former case the sticking point in both cases is that there is a library
> > incompatibility with jackson-module-scala_2.xx. In the latter case there is a
> > problem with SourceRDD.SourcePartitioning not [correctly] implementing
> > `equals(...)` from the parent trait.
> >
> > Posts on the mailing list made me think that the move to Scala 2.11
> started only
> > last fall, so I figured it should be easy to make the switch back.
> >
> > However, I have a feeling that it could be the case that I just don't
> understand
> > the Beam build system well enough to produce the right outcome (a custom
> version
> > that can be used with older Scala).
> >
> > Is there a correct or better way of achieving this?
> >
> > Kind Regards,
> >
> > Gary Dusbabek
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: spark runner for Scala 2.10
Posted by Gary Dusbabek <gd...@gmail.com>.
Jean-Baptiste,
Thanks for responding. I agree--it would be better to use Scala 2.11. I'm
in the process of creating a Beam POC with an existing platform and
upgrading everything in that platform to Scala 2.11 as a prerequisite is
out of scope.
It would be helpful to know if Beam in it's current state is backward
incompatible with Scala 2.10 for reasons other than the dependencies.
But if there is a way to make it work to enable a POC, I would appreciate
some pointers, as it doesn't seem to be as simple as changing the "*_2.11"
references in the poms.
Cheers,
Gary.
On Thu, Feb 22, 2018 at 5:34 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Gary,
>
> Beam 2.3.0 and the Spark runner use Scala 2.11.
>
> I can help you to have a smooth transition by creating a local branch using
> Scala 2.10. However, I strongly advice to upgrade to 2.11 as some other
> part of
> Beam (other runners and IOs) use 2.11 already.
>
> Regards
> JB
>
> On 02/22/2018 05:55 PM, Gary Dusbabek wrote:
> > Hi,
> >
> > My apologies if this belongs on the dev list. If it does, let me know
> and I'll
> > shoot things over that way...
> >
> > For the last day or so, I've been trying to create a Spark Runner that
> will work
> > on older deployments using Scala 2.10. I've taken a few approaches:
> >
> > 1. selectively changing a few dependencies in beam-runners-spark.pom
> (and a few
> > other places in the parent)
> > 2. updating every dependency that references *_2.11 to be *_2.10
> >
> > In the former case the sticking point in both cases is that there is a
> library
> > incompatibility with jackson-module-scala_2.xx. In the latter case
> there is a
> > problem with SourceRDD.SourcePartitioning not [correctly] implementing
> > `equals(...)` from the parent trait.
> >
> > Posts on the mailing list made me think that the move to Scala 2.11
> started only
> > last fall, so I figured it should be easy to make the switch back.
> >
> > However, I have a feeling that it could be the case that I just don't
> understand
> > the Beam build system well enough to produce the right outcome (a custom
> version
> > that can be used with older Scala).
> >
> > Is there a correct or better way of achieving this?
> >
> > Kind Regards,
> >
> > Gary Dusbabek
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: spark runner for Scala 2.10
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Gary,
Beam 2.3.0 and the Spark runner use Scala 2.11.
I can help you to have a smooth transition by creating a local branch using
Scala 2.10. However, I strongly advice to upgrade to 2.11 as some other part of
Beam (other runners and IOs) use 2.11 already.
Regards
JB
On 02/22/2018 05:55 PM, Gary Dusbabek wrote:
> Hi,
>
> My apologies if this belongs on the dev list. If it does, let me know and I'll
> shoot things over that way...
>
> For the last day or so, I've been trying to create a Spark Runner that will work
> on older deployments using Scala 2.10. I've taken a few approaches:
>
> 1. selectively changing a few dependencies in beam-runners-spark.pom (and a few
> other places in the parent)
> 2. updating every dependency that references *_2.11 to be *_2.10
>
> In the former case the sticking point in both cases is that there is a library
> incompatibility with jackson-module-scala_2.xx. In the latter case there is a
> problem with SourceRDD.SourcePartitioning not [correctly] implementing
> `equals(...)` from the parent trait.
>
> Posts on the mailing list made me think that the move to Scala 2.11 started only
> last fall, so I figured it should be easy to make the switch back.
>
> However, I have a feeling that it could be the case that I just don't understand
> the Beam build system well enough to produce the right outcome (a custom version
> that can be used with older Scala).
>
> Is there a correct or better way of achieving this?
>
> Kind Regards,
>
> Gary Dusbabek
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com