You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Huadong Liu <hu...@gmail.com> on 2021/06/30 17:35:15 UTC
iceberg-hive-metastore dependencies
Hi, I made a shadowJar with *org.apache.hive:hive-metastore:3.1.2 and got
errors:*
*Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapred/JobConf at
org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at
org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5109) at
org.apache.iceberg.hive.HiveClientPool.<init>(HiveClientPool.java:48) at
org.apache.iceberg.hive.HiveClientPool.<init>(HiveClientPool.java:43) at
org.apache.iceberg.hive.HiveCatalog.<init>(HiveCatalog.java:78) at
com.gradle.iceberg.HadoopToHive.main(HadoopToHive.java:52)*
The error is gone with *org.apache.hive:hive-metastore:2.3.6 in
gradle.build. *
*dependencies {*
* compile 'org.apache.iceberg:iceberg-hive-metastore:0.11.1'*
* compile 'org.apache.hive:hive-metastore:3.1.2' *
*}*
hadoop-common Any idea how to fix this? Thank you.
Re: iceberg-hive-metastore dependencies
Posted by Huadong Liu <hu...@gmail.com>.
Thanks Jack. *hadoop-common 3.1.0* is a dependency of hive-metastore.
Unfortunately it didn't make a difference by adding that. I see other
versions of hadoop-common (2.6.0 and 2.7.1) from other chained dependencies
updated to 3.1.0 from *gradle dependencies*. Maybe they get messed up?
On Wed, Jun 30, 2021 at 11:19 AM Jack Ye <ye...@gmail.com> wrote:
> Hadoop dependencies are also compileOnly in Iceberg, so you also need to
> add Hadoop packages such as hadoop-common and hadoop-client in your project
> dependency.
> -Jack Ye
>
> On Wed, Jun 30, 2021 at 10:35 AM Huadong Liu <hu...@gmail.com> wrote:
>
>> Hi, I made a shadowJar with *org.apache.hive:hive-metastore:3.1.2 and
>> got errors:*
>>
>>
>> *Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapred/JobConf at
>> org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at
>> org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5109) at
>> org.apache.iceberg.hive.HiveClientPool.<init>(HiveClientPool.java:48) at
>> org.apache.iceberg.hive.HiveClientPool.<init>(HiveClientPool.java:43) at
>> org.apache.iceberg.hive.HiveCatalog.<init>(HiveCatalog.java:78) at
>> com.gradle.iceberg.HadoopToHive.main(HadoopToHive.java:52)*
>>
>> The error is gone with *org.apache.hive:hive-metastore:2.3.6 in
>> gradle.build. *
>>
>> *dependencies {*
>> * compile 'org.apache.iceberg:iceberg-hive-metastore:0.11.1'*
>> * compile 'org.apache.hive:hive-metastore:3.1.2' *
>> *}*
>>
>> hadoop-common Any idea how to fix this? Thank you.
>>
>
Re: iceberg-hive-metastore dependencies
Posted by Jack Ye <ye...@gmail.com>.
Hadoop dependencies are also compileOnly in Iceberg, so you also need to
add Hadoop packages such as hadoop-common and hadoop-client in your project
dependency.
-Jack Ye
On Wed, Jun 30, 2021 at 10:35 AM Huadong Liu <hu...@gmail.com> wrote:
> Hi, I made a shadowJar with *org.apache.hive:hive-metastore:3.1.2 and got
> errors:*
>
>
> *Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapred/JobConf at
> org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at
> org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5109) at
> org.apache.iceberg.hive.HiveClientPool.<init>(HiveClientPool.java:48) at
> org.apache.iceberg.hive.HiveClientPool.<init>(HiveClientPool.java:43) at
> org.apache.iceberg.hive.HiveCatalog.<init>(HiveCatalog.java:78) at
> com.gradle.iceberg.HadoopToHive.main(HadoopToHive.java:52)*
>
> The error is gone with *org.apache.hive:hive-metastore:2.3.6 in
> gradle.build. *
>
> *dependencies {*
> * compile 'org.apache.iceberg:iceberg-hive-metastore:0.11.1'*
> * compile 'org.apache.hive:hive-metastore:3.1.2' *
> *}*
>
> hadoop-common Any idea how to fix this? Thank you.
>