You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by wu lihu <ro...@gmail.com> on 2017/03/17 08:38:12 UTC
kylin 无法加载hive classpath 内得lib
Hi All
我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta 报错
Caused by: MetaException(message:java.lang.ClassNotFoundException
Class org.openx.data.jsonserde.JsonSerDe not found)
但是我在 System tab 里面可以看到
/usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
在classpath 里面。 请问这个怎么解决？
Re: kylin 无法加载hive classpath 内得lib
Posted by Li Yang <li...@apache.org>.
> The Exception threw out from the kylin process, and the EMR version is
5.0.0
Then you could analyze the "bin/find-hive-dependency.sh" to see if it
catches the right hive classpath in your env.
Cheers
Yang
On Tue, Mar 21, 2017 at 11:17 AM, wu lihu <ro...@gmail.com> wrote:
> one thing needs to be reminded, if I use Hive CLI, then everything is
> great.
>
> 2017-03-21 11:13 GMT+08:00 wu lihu <ro...@gmail.com>:
> > The whole stacktrace like below :
> > Caused by: MetaException(message:java.lang.ClassNotFoundException
> > Class org.openx.data.jsonserde.JsonSerDe not found)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result$get_fields_resultStandardScheme.read(ThriftHive
> > Metastore.java:35482)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result$get_fields_resultStandardScheme.read(ThriftHive
> > Metastore.java:35449)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result.read(ThriftHiveMetastore.java:35375)
> > at org.apache.thrift.TServiceClient.receiveBase(
> TServiceClient.java:86)
> > at org.apache.hadoop.hive.metastore.api.
> ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017)
> > at org.apache.hadoop.hive.metastore.api.
> ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003)
> > at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.
> getFields(HiveMetaStoreClient.java:1497)
> > at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(
> CLIHiveClient.java:80)
> > at org.apache.kylin.source.hive.HiveSourceTableLoader.
> extractHiveTables(HiveSourceTableLoader.java:86)
> > ... 84 more
> >
> > seems like it caused by the hive thrift service read the wrong
> > metastore? or the state of the metastore has incorrect reference ????
> >
> > 2017-03-21 11:04 GMT+08:00 wu lihu <ro...@gmail.com>:
> >> Hi Li Yang & ShaoFeng
> >> The Exception threw out from the kylin process, and the EMR
> >> version is 5.0.0
> >>
> >> 2017-03-21 2:59 GMT+08:00 Li Yang <li...@apache.org>:
> >>> Also is the ClassNotFoundException from Kylin process or from MR
> process?
> >>> That makes a difference.
> >>>
> >>> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi <sh...@apache.org>
> >>> wrote:
> >>>
> >>>> Which version of EMR are you running? In my experience if using EMR,
> there
> >>>> is no additional configuration for Kylin; just download and run
> "kylin.sh
> >>>> start" will work.
> >>>>
> >>>> 2017-03-17 16:38 GMT+08:00 wu lihu <ro...@gmail.com>:
> >>>>
> >>>> > Hi All
> >>>> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
> >>>> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta 报错
> >>>> > Caused by: MetaException(message:java.lang.ClassNotFoundException
> >>>> > Class org.openx.data.jsonserde.JsonSerDe not found)
> >>>> >
> >>>> > 但是我在 System tab 里面可以看到
> >>>> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-
> dependencies.jar
> >>>> > 在classpath 里面。 请问这个怎么解决？
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>>
> >>>> Shaofeng Shi 史少锋
> >>>>
>
Re: kylin 无法加载hive classpath 内得lib
Posted by wu lihu <ro...@gmail.com>.
one thing needs to be reminded, if I use Hive CLI, then everything is great.
2017-03-21 11:13 GMT+08:00 wu lihu <ro...@gmail.com>:
> The whole stacktrace like below :
> Caused by: MetaException(message:java.lang.ClassNotFoundException
> Class org.openx.data.jsonserde.JsonSerDe not found)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
> Metastore.java:35482)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
> Metastore.java:35449)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:35375)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1497)
> at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:80)
> at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:86)
> ... 84 more
>
> seems like it caused by the hive thrift service read the wrong
> metastore? or the state of the metastore has incorrect reference ????
>
> 2017-03-21 11:04 GMT+08:00 wu lihu <ro...@gmail.com>:
>> Hi Li Yang & ShaoFeng
>> The Exception threw out from the kylin process, and the EMR
>> version is 5.0.0
>>
>> 2017-03-21 2:59 GMT+08:00 Li Yang <li...@apache.org>:
>>> Also is the ClassNotFoundException from Kylin process or from MR process?
>>> That makes a difference.
>>>
>>> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi <sh...@apache.org>
>>> wrote:
>>>
>>>> Which version of EMR are you running? In my experience if using EMR, there
>>>> is no additional configuration for Kylin; just download and run "kylin.sh
>>>> start" will work.
>>>>
>>>> 2017-03-17 16:38 GMT+08:00 wu lihu <ro...@gmail.com>:
>>>>
>>>> > Hi All
>>>> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
>>>> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta 报错
>>>> > Caused by: MetaException(message:java.lang.ClassNotFoundException
>>>> > Class org.openx.data.jsonserde.JsonSerDe not found)
>>>> >
>>>> > 但是我在 System tab 里面可以看到
>>>> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
>>>> > 在classpath 里面。 请问这个怎么解决？
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
Re: kylin 无法加载hive classpath 内得lib
Posted by wu lihu <ro...@gmail.com>.
The whole stacktrace like below :
Caused by: MetaException(message:java.lang.ClassNotFoundException
Class org.openx.data.jsonserde.JsonSerDe not found)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
Metastore.java:35482)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
Metastore.java:35449)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:35375)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1497)
at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:80)
at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:86)
... 84 more
seems like it caused by the hive thrift service read the wrong
metastore? or the state of the metastore has incorrect reference ????
2017-03-21 11:04 GMT+08:00 wu lihu <ro...@gmail.com>:
> Hi Li Yang & ShaoFeng
> The Exception threw out from the kylin process, and the EMR
> version is 5.0.0
>
> 2017-03-21 2:59 GMT+08:00 Li Yang <li...@apache.org>:
>> Also is the ClassNotFoundException from Kylin process or from MR process?
>> That makes a difference.
>>
>> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>>> Which version of EMR are you running? In my experience if using EMR, there
>>> is no additional configuration for Kylin; just download and run "kylin.sh
>>> start" will work.
>>>
>>> 2017-03-17 16:38 GMT+08:00 wu lihu <ro...@gmail.com>:
>>>
>>> > Hi All
>>> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
>>> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta 报错
>>> > Caused by: MetaException(message:java.lang.ClassNotFoundException
>>> > Class org.openx.data.jsonserde.JsonSerDe not found)
>>> >
>>> > 但是我在 System tab 里面可以看到
>>> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
>>> > 在classpath 里面。 请问这个怎么解决？
>>> >
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
Re: kylin 无法加载hive classpath 内得lib
Posted by wu lihu <ro...@gmail.com>.
Hi Li Yang & ShaoFeng
The Exception threw out from the kylin process, and the EMR
version is 5.0.0
2017-03-21 2:59 GMT+08:00 Li Yang <li...@apache.org>:
> Also is the ClassNotFoundException from Kylin process or from MR process?
> That makes a difference.
>
> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> Which version of EMR are you running? In my experience if using EMR, there
>> is no additional configuration for Kylin; just download and run "kylin.sh
>> start" will work.
>>
>> 2017-03-17 16:38 GMT+08:00 wu lihu <ro...@gmail.com>:
>>
>> > Hi All
>> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
>> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta 报错
>> > Caused by: MetaException(message:java.lang.ClassNotFoundException
>> > Class org.openx.data.jsonserde.JsonSerDe not found)
>> >
>> > 但是我在 System tab 里面可以看到
>> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
>> > 在classpath 里面。 请问这个怎么解决？
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
Re: kylin 无法加载hive classpath 内得lib
Posted by Li Yang <li...@apache.org>.
Also is the ClassNotFoundException from Kylin process or from MR process?
That makes a difference.
On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi <sh...@apache.org>
wrote:
> Which version of EMR are you running? In my experience if using EMR, there
> is no additional configuration for Kylin; just download and run "kylin.sh
> start" will work.
>
> 2017-03-17 16:38 GMT+08:00 wu lihu <ro...@gmail.com>:
>
> > Hi All
> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta 报错
> > Caused by: MetaException(message:java.lang.ClassNotFoundException
> > Class org.openx.data.jsonserde.JsonSerDe not found)
> >
> > 但是我在 System tab 里面可以看到
> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
> > 在classpath 里面。 请问这个怎么解决？
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Re: kylin 无法加载hive classpath 内得lib
Posted by ShaoFeng Shi <sh...@apache.org>.
Which version of EMR are you running? In my experience if using EMR, there
is no additional configuration for Kylin; just download and run "kylin.sh
start" will work.
2017-03-17 16:38 GMT+08:00 wu lihu <ro...@gmail.com>:
> Hi All
> 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
> find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta 报错
> Caused by: MetaException(message:java.lang.ClassNotFoundException
> Class org.openx.data.jsonserde.JsonSerDe not found)
>
> 但是我在 System tab 里面可以看到
> /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
> 在classpath 里面。 请问这个怎么解决？
>
--
Best regards,
Shaofeng Shi 史少锋