You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eric Friedman <er...@gmail.com> on 2014/08/10 18:36:11 UTC

CDH5, HiveContext, Parquet

I have a CDH5.0.3 cluster with Hive tables written in Parquet.

The tables have the "DeprecatedParquetInputFormat" on their metadata, and
when I try to select from one using Spark SQL, it blows up with a stack
trace like this:

java.lang.RuntimeException: java.lang.ClassNotFoundException:
parquet.hive.DeprecatedParquetInputFormat

at
org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:309)


Fair enough, DeprecatedParquetInputFormat isn't in the Spark assembly
built with Hive.


if I try to add hive-exec-0.12.0-cdh5.0.3.jar to my SPARK_CLASSPATH, in
order to get DeprecatedParquetInputFormat, I find out that there is an
incompatibility in the SerDeUtils class.  Spark's Hive snapshot expects to
find


java.lang.NoSuchMethodError:
org.apache.hadoop.hive.serde2.SerDeUtils.lookupDeserializer(Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/Deserializer;

 at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:217)


But that isn't in the Hive snapshot provided by CDH5.0.3


Both Spark and CDH label their Hive versions as 0.12.0.


According to the Apache SVN server
<http://svn.apache.org/viewvc/hive/tags/release-0.12.0/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java?revision=1532081&view=markup>,
CDH is the one that's out of step, as this method is definitely on the
0.12.0 release.  I have raised a ticket with Cloudera about this.


Has anyone found a workaround?


I did try extracting a subset of jars from hive-exec.jar, but that quickly
turned into a journey down the rabbit hole.

Re: CDH5, HiveContext, Parquet

Posted by Eric Friedman <er...@gmail.com>.
On Sun, Aug 10, 2014 at 2:43 PM, Michael Armbrust <mi...@databricks.com>
wrote:

>
>>
> if I try to add hive-exec-0.12.0-cdh5.0.3.jar to my SPARK_CLASSPATH, in
>> order to get DeprecatedParquetInputFormat, I find out that there is an
>> incompatibility in the SerDeUtils class.  Spark's Hive snapshot expects to
>> find
>
>
> Instead of including CDH's version of Hive, I'd try just including the
> Hive jars for Parquet from here:
> http://mvnrepository.com/artifact/com.twitter/parquet-hive-bundle/1.5.0
>
>
This worked for me, thank you.  In case someone else wishes to try it, I
un-jarred the spark-1.0.2 assembly and then un-jarred the
parquet-hive-bundle in the same place.  I then re-jarred the whole thing
back into an assembly and was able to run it with PySpark on YARN.  It is
really nice to be able to leverage the data partitioning through Hive.

Note that I had to use the Java6 version of jar after having found that
something in the way Java7 creates jar files makes the python code in the
assembly inaccessible.  But with Java6's jar, all is well.

Thank you, Michael, for this suggestion.

And Sean, I do value the effort Cloudera puts into making this "real."  At
this stage I am evaluating options and so it's helpful to me to be able to
kick the tires, at scale, without asking the SREs to undertake a full
upgrade effort.  Rest assured that we will do that in due course.  Thanks
again for sharing your insights.

Eric

Re: CDH5, HiveContext, Parquet

Posted by chutium <te...@gmail.com>.
hive-thriftserver does not work with parquet tables in hive metastore also,
this PR will fix it too?

do not need to change any pom.xml ?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/CDH5-HiveContext-Parquet-tp11853p11880.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: CDH5, HiveContext, Parquet

Posted by Yin Huai <hu...@gmail.com>.
If the link to PR/1819 is broken. Here is the one
https://github.com/apache/spark/pull/1819.


On Sun, Aug 10, 2014 at 5:56 PM, Eric Friedman <er...@gmail.com>
wrote:

> Thanks Michael, I can try that too.
>
> I know you guys aren't in sales/marketing (thank G-d), but given all the
> hoopla about the CDH<->DataBricks partnership, it'd be awesome if you guys
> were somewhat more aligned, by which I mean that the DataBricks releases on
> Apache that say "for CDH5" would actually work on CDH5. I know Cloudera has
> to qualify them for support and so on, but if DataBricks development
> treated mainstream CDH as the primary deployment target, well that would be
> great.  Of course I'm being selfish.  *smile*.
>
>
>
> On Aug 10, 2014, at 2:43 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
> I imagine it's not the only instance of this kind of problem people
>> will ever encounter. Can you rebuild Spark with this particular
>> release of Hive?
>
>
> Unfortunately the Hive APIs that we use change to much from release to
> release to make this possible.  There is a JIRA for compiling Spark SQL
> against Hive 13: SPARK-2706
> <https://issues.apache.org/jira/browse/SPARK-2706>.
>
> if I try to add hive-exec-0.12.0-cdh5.0.3.jar to my SPARK_CLASSPATH, in
>> order to get DeprecatedParquetInputFormat, I find out that there is an
>> incompatibility in the SerDeUtils class.  Spark's Hive snapshot expects to
>> find
>
>
> Instead of including CDH's version of Hive, I'd try just including the
> Hive jars for Parquet from here:
> http://mvnrepository.com/artifact/com.twitter/parquet-hive-bundle/1.5.0
>
> However, support for this is a work in progress.  You'll likely need to
> make sure you have a version of Spark that includes this commit (added last
> Friday)
> https://github.com/apache/spark/commit/9016af3f2729101027e33593e094332f05f48d92
>
> Another option would be to try this *experimental* patch: pr/1819.
>
>

Re: CDH5, HiveContext, Parquet

Posted by Eric Friedman <er...@gmail.com>.
Thanks Michael, I can try that too. 

I know you guys aren't in sales/marketing (thank G-d), but given all the hoopla about the CDH<->DataBricks partnership, it'd be awesome if you guys were somewhat more aligned, by which I mean that the DataBricks releases on Apache that say "for CDH5" would actually work on CDH5. I know Cloudera has to qualify them for support and so on, but if DataBricks development treated mainstream CDH as the primary deployment target, well that would be great.  Of course I'm being selfish.  *smile*. 



On Aug 10, 2014, at 2:43 PM, Michael Armbrust <mi...@databricks.com> wrote:

>> I imagine it's not the only instance of this kind of problem people
>> will ever encounter. Can you rebuild Spark with this particular
>> release of Hive?
> 
> Unfortunately the Hive APIs that we use change to much from release to release to make this possible.  There is a JIRA for compiling Spark SQL against Hive 13: SPARK-2706.
> 
>> if I try to add hive-exec-0.12.0-cdh5.0.3.jar to my SPARK_CLASSPATH, in order to get DeprecatedParquetInputFormat, I find out that there is an incompatibility in the SerDeUtils class.  Spark's Hive snapshot expects to find 
> 
> 
> Instead of including CDH's version of Hive, I'd try just including the Hive jars for Parquet from here: http://mvnrepository.com/artifact/com.twitter/parquet-hive-bundle/1.5.0
> 
> However, support for this is a work in progress.  You'll likely need to make sure you have a version of Spark that includes this commit (added last Friday) https://github.com/apache/spark/commit/9016af3f2729101027e33593e094332f05f48d92
> 
> Another option would be to try this experimental patch: pr/1819.

Re: CDH5, HiveContext, Parquet

Posted by Michael Armbrust <mi...@databricks.com>.
>
> I imagine it's not the only instance of this kind of problem people
> will ever encounter. Can you rebuild Spark with this particular
> release of Hive?


Unfortunately the Hive APIs that we use change to much from release to
release to make this possible.  There is a JIRA for compiling Spark SQL
against Hive 13: SPARK-2706
<https://issues.apache.org/jira/browse/SPARK-2706>.

if I try to add hive-exec-0.12.0-cdh5.0.3.jar to my SPARK_CLASSPATH, in
> order to get DeprecatedParquetInputFormat, I find out that there is an
> incompatibility in the SerDeUtils class.  Spark's Hive snapshot expects to
> find


Instead of including CDH's version of Hive, I'd try just including the Hive
jars for Parquet from here:
http://mvnrepository.com/artifact/com.twitter/parquet-hive-bundle/1.5.0

However, support for this is a work in progress.  You'll likely need to
make sure you have a version of Spark that includes this commit (added last
Friday)
https://github.com/apache/spark/commit/9016af3f2729101027e33593e094332f05f48d92

Another option would be to try this *experimental* patch: pr/1819.

Re: CDH5, HiveContext, Parquet

Posted by Eric Friedman <er...@gmail.com>.
Yeah, that's what I feared. Unfortunately upgrades on very large production clusters aren't a cheap way to find out what else is broken. 

Perhaps I can create an RCFile table and sidestep parquet for now. 



> On Aug 10, 2014, at 1:45 PM, Sean Owen <so...@cloudera.com> wrote:
> 
> Hm, I was thinking that the issue is that Spark has to use a forked
> hive-exec since hive-exec unfortunately includes a bunch of
> dependencies it shouldn't. It forked Hive 0.12.0:
> http://mvnrepository.com/artifact/org.spark-project.hive/hive-exec/0.12.0
> 
> ... and then I was thinking maybe CDH wasn't able to harmonize it. But
> 5.1 did harmonize this dependency, it appears.
> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-hive_2.10/1.0.0-cdh5.1.0/
> 
> Looking back at 5.0.3, which shipped 0.9.0 ... I don't think the Spark
> Hive module existed in 0.9.0?
> https://github.com/apache/spark/tree/branch-0.9/sql/hive So yeah using
> it with newer Spark 1.0 code probably means you'd have to build Spark
> to pull in the CDH version of hive-exec (or else keep it out the
> custom assembly and make sure only the CDH version is on the classpath
> at runtime) and hope you don't actually have to shade it like Spark
> does to get it to work.
> 
> I think support will suggest this 1.0 Hive code is supported and works
> in the version that ships 1.0!
> 
> (Sorry for extending a thread apparently about CDH but I think the
> issue is actually more broadly applicable.)
> 
> 
> On Sun, Aug 10, 2014 at 9:20 PM, Eric Friedman
> <er...@gmail.com> wrote:
>> Hi Sean,
>> 
>> Thanks for the reply.  I'm on CDH 5.0.3 and upgrading the whole cluster to
>> 5.1.0 will eventually happen but not immediately.
>> 
>> I've tried running the CDH spark-1.0 release and also building it from
>> source.  This, unfortunately goes into a whole other rathole of
>> dependencies.  :-(
>> 
>> Eric

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: CDH5, HiveContext, Parquet

Posted by Sean Owen <so...@cloudera.com>.
Hm, I was thinking that the issue is that Spark has to use a forked
hive-exec since hive-exec unfortunately includes a bunch of
dependencies it shouldn't. It forked Hive 0.12.0:
http://mvnrepository.com/artifact/org.spark-project.hive/hive-exec/0.12.0

... and then I was thinking maybe CDH wasn't able to harmonize it. But
5.1 did harmonize this dependency, it appears.
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-hive_2.10/1.0.0-cdh5.1.0/

Looking back at 5.0.3, which shipped 0.9.0 ... I don't think the Spark
Hive module existed in 0.9.0?
https://github.com/apache/spark/tree/branch-0.9/sql/hive So yeah using
it with newer Spark 1.0 code probably means you'd have to build Spark
to pull in the CDH version of hive-exec (or else keep it out the
custom assembly and make sure only the CDH version is on the classpath
at runtime) and hope you don't actually have to shade it like Spark
does to get it to work.

I think support will suggest this 1.0 Hive code is supported and works
in the version that ships 1.0!

(Sorry for extending a thread apparently about CDH but I think the
issue is actually more broadly applicable.)


On Sun, Aug 10, 2014 at 9:20 PM, Eric Friedman
<er...@gmail.com> wrote:
> Hi Sean,
>
> Thanks for the reply.  I'm on CDH 5.0.3 and upgrading the whole cluster to
> 5.1.0 will eventually happen but not immediately.
>
> I've tried running the CDH spark-1.0 release and also building it from
> source.  This, unfortunately goes into a whole other rathole of
> dependencies.  :-(
>
> Eric

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: CDH5, HiveContext, Parquet

Posted by Eric Friedman <er...@gmail.com>.
Hi Sean,

Thanks for the reply.  I'm on CDH 5.0.3 and upgrading the whole cluster to
5.1.0 will eventually happen but not immediately.

I've tried running the CDH spark-1.0 release and also building it from
source.  This, unfortunately goes into a whole other rathole of
dependencies.  :-(

Eric


On Sun, Aug 10, 2014 at 10:16 AM, Sean Owen <so...@cloudera.com> wrote:

> As far as I can tell, the method was removed after 0.12.0 in the fix
> for HIVE-5223 (
> https://github.com/apache/hive/commit/4059a32f34633dcef1550fdef07d9f9e044c722c#diff-948cc2a95809f584eb030e2b57be3993
> ),
> and that fix was back-ported in its entirety to 5.0.0+:
>
> http://archive.cloudera.com/cdh5/cdh/5/hive-0.12.0-cdh5.0.0.releasenotes.html
>
> The fix was evidently also important, but it's not clear the build can
> have the fix and keep this method, not without forking via a custom
> patch. Even though CDH5 never *didn't* have this version of the code,
> it creates this sort of surprising problem.
>
> I imagine it's not the only instance of this kind of problem people
> will ever encounter. Can you rebuild Spark with this particular
> release of Hive?
>
> Because that's what the Spark that was shipped with CDH would have
> done. Are you replacing / not using that?
>
>
>
>
>
> On Sun, Aug 10, 2014 at 5:36 PM, Eric Friedman
> <er...@gmail.com> wrote:
> > I have a CDH5.0.3 cluster with Hive tables written in Parquet.
> >
> > The tables have the "DeprecatedParquetInputFormat" on their metadata, and
> > when I try to select from one using Spark SQL, it blows up with a stack
> > trace like this:
> >
> > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > parquet.hive.DeprecatedParquetInputFormat
> >
> > at
> >
> org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:309)
> >
> >
> > Fair enough, DeprecatedParquetInputFormat isn't in the Spark assembly
> built
> > with Hive.
> >
> >
> > if I try to add hive-exec-0.12.0-cdh5.0.3.jar to my SPARK_CLASSPATH, in
> > order to get DeprecatedParquetInputFormat, I find out that there is an
> > incompatibility in the SerDeUtils class.  Spark's Hive snapshot expects
> to
> > find
> >
> >
> > java.lang.NoSuchMethodError:
> >
> org.apache.hadoop.hive.serde2.SerDeUtils.lookupDeserializer(Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/Deserializer;
> >
> > at
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:217)
> >
> >
> > But that isn't in the Hive snapshot provided by CDH5.0.3
> >
> >
> > Both Spark and CDH label their Hive versions as 0.12.0.
> >
> >
> > According to the Apache SVN server, CDH is the one that's out of step, as
> > this method is definitely on the 0.12.0 release.  I have raised a ticket
> > with Cloudera about this.
> >
> >
> > Has anyone found a workaround?
> >
> >
> > I did try extracting a subset of jars from hive-exec.jar, but that
> quickly
> > turned into a journey down the rabbit hole.
>

Re: CDH5, HiveContext, Parquet

Posted by Sean Owen <so...@cloudera.com>.
As far as I can tell, the method was removed after 0.12.0 in the fix
for HIVE-5223 (https://github.com/apache/hive/commit/4059a32f34633dcef1550fdef07d9f9e044c722c#diff-948cc2a95809f584eb030e2b57be3993),
and that fix was back-ported in its entirety to 5.0.0+:
http://archive.cloudera.com/cdh5/cdh/5/hive-0.12.0-cdh5.0.0.releasenotes.html

The fix was evidently also important, but it's not clear the build can
have the fix and keep this method, not without forking via a custom
patch. Even though CDH5 never *didn't* have this version of the code,
it creates this sort of surprising problem.

I imagine it's not the only instance of this kind of problem people
will ever encounter. Can you rebuild Spark with this particular
release of Hive?

Because that's what the Spark that was shipped with CDH would have
done. Are you replacing / not using that?





On Sun, Aug 10, 2014 at 5:36 PM, Eric Friedman
<er...@gmail.com> wrote:
> I have a CDH5.0.3 cluster with Hive tables written in Parquet.
>
> The tables have the "DeprecatedParquetInputFormat" on their metadata, and
> when I try to select from one using Spark SQL, it blows up with a stack
> trace like this:
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> parquet.hive.DeprecatedParquetInputFormat
>
> at
> org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:309)
>
>
> Fair enough, DeprecatedParquetInputFormat isn't in the Spark assembly built
> with Hive.
>
>
> if I try to add hive-exec-0.12.0-cdh5.0.3.jar to my SPARK_CLASSPATH, in
> order to get DeprecatedParquetInputFormat, I find out that there is an
> incompatibility in the SerDeUtils class.  Spark's Hive snapshot expects to
> find
>
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hive.serde2.SerDeUtils.lookupDeserializer(Ljava/lang/String;)Lorg/apache/hadoop/hive/serde2/Deserializer;
>
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:217)
>
>
> But that isn't in the Hive snapshot provided by CDH5.0.3
>
>
> Both Spark and CDH label their Hive versions as 0.12.0.
>
>
> According to the Apache SVN server, CDH is the one that's out of step, as
> this method is definitely on the 0.12.0 release.  I have raised a ticket
> with Cloudera about this.
>
>
> Has anyone found a workaround?
>
>
> I did try extracting a subset of jars from hive-exec.jar, but that quickly
> turned into a journey down the rabbit hole.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org