You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yana Kadiyska <ya...@gmail.com> on 2015/06/23 20:37:57 UTC
Can Spark1.4 work with CDH4.6
Hi folks, I have been using Spark against an external Metastore service
which runs Hive with Cdh 4.6
In Spark 1.2, I was able to successfully connect by building with the
following:
./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0
-Phive-thriftserver -Phive-0.12.0
I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of
spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars
When I tried to use this setup spark-shell fails for me with the following
error:
15/06/23 18:18:07 INFO hive.HiveContext: Initializing
HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c
java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError:
com/google/common/base/Preconditions when creating Hive client using
classpath: file:/hive/lib/guava-11.0.2.jar,
file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar,
file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar,
file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar,
file:/hive/lib/commons-logging-1.0.4.jar
I don't know why it's not seeing the class -- it's in the guava jar. If
anyone has had success with 0.12 version please let me know what jars need
to be on the classpath. I think my Hive version might be too outdated but I
don't control the metastore and I had success with Spark1.2 so I'm hoping...
Re: Can Spark1.4 work with CDH4.6
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can look into spark.driver.userClassPathFirst flag.
spark.driver.userClassPathFirstfalse(Experimental) Whether to give
user-added jars precedence over Spark's own jars when loading classes in
the the driver. This feature can be used to mitigate conflicts between
Spark's dependencies and user dependencies. It is currently an experimental
feature. This is used in cluster mode only.
Thanks
Best Regards
On Thu, Jun 25, 2015 at 1:08 AM, Yana Kadiyska <ya...@gmail.com>
wrote:
> Thanks, that did seem to make a difference. I am a bit scared of this
> approach as spark itself has a different guava dependency but the error
> does go away this way
>
> On Wed, Jun 24, 2015 at 10:04 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Can you try to add those jars in the SPARK_CLASSPATH and give it a try?
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska <ya...@gmail.com>
>> wrote:
>>
>>> Hi folks, I have been using Spark against an external Metastore service
>>> which runs Hive with Cdh 4.6
>>>
>>> In Spark 1.2, I was able to successfully connect by building with the
>>> following:
>>>
>>> ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0
>>> -Phive-thriftserver -Phive-0.12.0
>>>
>>> I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor
>>> of
>>> spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars
>>>
>>> When I tried to use this setup spark-shell fails for me with the
>>> following error:
>>>
>>> 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c
>>> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar
>>>
>>>
>>>
>>> I don't know why it's not seeing the class -- it's in the guava jar. If
>>> anyone has had success with 0.12 version please let me know what jars need
>>> to be on the classpath. I think my Hive version might be too outdated but I
>>> don't control the metastore and I had success with Spark1.2 so I'm hoping...
>>>
>>>
>>>
>>
>
Re: Can Spark1.4 work with CDH4.6
Posted by Yana Kadiyska <ya...@gmail.com>.
Thanks, that did seem to make a difference. I am a bit scared of this
approach as spark itself has a different guava dependency but the error
does go away this way
On Wed, Jun 24, 2015 at 10:04 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Can you try to add those jars in the SPARK_CLASSPATH and give it a try?
>
> Thanks
> Best Regards
>
> On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska <ya...@gmail.com>
> wrote:
>
>> Hi folks, I have been using Spark against an external Metastore service
>> which runs Hive with Cdh 4.6
>>
>> In Spark 1.2, I was able to successfully connect by building with the
>> following:
>>
>> ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0
>> -Phive-thriftserver -Phive-0.12.0
>>
>> I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of
>> spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars
>>
>> When I tried to use this setup spark-shell fails for me with the
>> following error:
>>
>> 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c
>> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar
>>
>>
>>
>> I don't know why it's not seeing the class -- it's in the guava jar. If
>> anyone has had success with 0.12 version please let me know what jars need
>> to be on the classpath. I think my Hive version might be too outdated but I
>> don't control the metastore and I had success with Spark1.2 so I'm hoping...
>>
>>
>>
>
Re: Can Spark1.4 work with CDH4.6
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Can you try to add those jars in the SPARK_CLASSPATH and give it a try?
Thanks
Best Regards
On Wed, Jun 24, 2015 at 12:07 AM, Yana Kadiyska <ya...@gmail.com>
wrote:
> Hi folks, I have been using Spark against an external Metastore service
> which runs Hive with Cdh 4.6
>
> In Spark 1.2, I was able to successfully connect by building with the
> following:
>
> ./make-distribution.sh --tgz -Dhadoop.version=2.0.0-mr1-cdh4.2.0
> -Phive-thriftserver -Phive-0.12.0
>
> I see that in Spark 1.4 the Hive 0.12.0 profile is deprecated in favor of
> spark.sql.hive.metastore.version/spark.sql.hive.metastore.jars
>
> When I tried to use this setup spark-shell fails for me with the following
> error:
>
> 15/06/23 18:18:07 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using [Ljava.net.URL;@7b7a9a6c
> java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: file:/hive/lib/guava-11.0.2.jar, file:/hive/lib/hive-exec-0.10.0-cdh4.6.0.jar, file:/hive/lib/hive-metastore-0.10.0-cdh4.6.0.jar, file:/hadoop/share/hadoop/mapreduce1/lib/hadoop-common-2.0.0-cdh4.6.0.jar, file:/hive/lib/commons-logging-1.0.4.jar
>
>
>
> I don't know why it's not seeing the class -- it's in the guava jar. If
> anyone has had success with 0.12 version please let me know what jars need
> to be on the classpath. I think my Hive version might be too outdated but I
> don't control the metastore and I had success with Spark1.2 so I'm hoping...
>
>
>