You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by Jeff Quinn <je...@nuna.com> on 2015/11/05 20:04:26 UTC
Crunch Compatibility with Spark 1.5.0?
Hello,
I am wondering if it would be possible to run a Crunch SparkPipeline on
Spark 1.5.0? I see that currently Crunch is pinned to Spark 1.3.1 libs in
master, when I try to run on a yarn-master, Spark 1.5.0, using Crunch
0.13.0, I am seeing weird exceptions such as:
java.lang.ClassNotFoundException: com.google.protobuf_spark.GeneratedMessage
Appreciate any guidance.
Thanks!
Jeff
--
*DISCLAIMER:* The contents of this email, including any attachments, may
contain information that is confidential, proprietary in nature, protected
health information (PHI), or otherwise protected by law from disclosure,
and is solely for the use of the intended recipient(s). If you are not the
intended recipient, you are hereby notified that any use, disclosure or
copying of this email, including any attachments, is unauthorized and
strictly prohibited. If you have received this email in error, please
notify the sender of this email. Please delete this and all copies of this
email from your system. Any opinions either expressed or implied in this
email and all attachments, are those of its author only, and do not
necessarily reflect those of Nuna Health, Inc.
Re: Crunch Compatibility with Spark 1.5.0?
Posted by Josh Wills <jo...@gmail.com>.
Awesome, good to know. Maybe Sean Owen is starting to introduce some sanity
into the project's dependency structure. ;-)
On Thu, Nov 5, 2015 at 11:25 AM, Jeff Quinn <je...@nuna.com> wrote:
> Thanks Josh!
>
> FWIW I threw org.spark-project.protobuf:protobuf-java:2.4.1-shaded on the
> Spark class path and the job *seems* to run fine on Spark 1.5.0 when
> built with Crunch 0.13
>
> On Thu, Nov 5, 2015 at 11:19 AM, Josh Wills <jo...@gmail.com> wrote:
>
>> Spark's dependency tree is an affront to everything good and decent in
>> the world. I'm going to run the current Crunch master against 1.5.1 locally
>> to see what all breaks.
>>
>> J
>>
>
>
> *DISCLAIMER:* The contents of this email, including any attachments, may
> contain information that is confidential, proprietary in nature, protected
> health information (PHI), or otherwise protected by law from disclosure,
> and is solely for the use of the intended recipient(s). If you are not the
> intended recipient, you are hereby notified that any use, disclosure or
> copying of this email, including any attachments, is unauthorized and
> strictly prohibited. If you have received this email in error, please
> notify the sender of this email. Please delete this and all copies of this
> email from your system. Any opinions either expressed or implied in this
> email and all attachments, are those of its author only, and do not
> necessarily reflect those of Nuna Health, Inc.
>
Re: Crunch Compatibility with Spark 1.5.0?
Posted by Jeff Quinn <je...@nuna.com>.
Thanks Josh!
FWIW I threw org.spark-project.protobuf:protobuf-java:2.4.1-shaded on the
Spark class path and the job *seems* to run fine on Spark 1.5.0 when built
with Crunch 0.13
On Thu, Nov 5, 2015 at 11:19 AM, Josh Wills <jo...@gmail.com> wrote:
> Spark's dependency tree is an affront to everything good and decent in the
> world. I'm going to run the current Crunch master against 1.5.1 locally to
> see what all breaks.
>
> J
>
--
*DISCLAIMER:* The contents of this email, including any attachments, may
contain information that is confidential, proprietary in nature, protected
health information (PHI), or otherwise protected by law from disclosure,
and is solely for the use of the intended recipient(s). If you are not the
intended recipient, you are hereby notified that any use, disclosure or
copying of this email, including any attachments, is unauthorized and
strictly prohibited. If you have received this email in error, please
notify the sender of this email. Please delete this and all copies of this
email from your system. Any opinions either expressed or implied in this
email and all attachments, are those of its author only, and do not
necessarily reflect those of Nuna Health, Inc.
Re: Crunch Compatibility with Spark 1.5.0?
Posted by Josh Wills <jo...@gmail.com>.
Spark's dependency tree is an affront to everything good and decent in the
world. I'm going to run the current Crunch master against 1.5.1 locally to
see what all breaks.
J