You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@phoenix.apache.org by Stepan Migunov <st...@firstlinesoftware.com> on 2018/03/27 08:46:57 UTC

Storage Handler for Apache Hive

Hi,

Phoenix 4.12.0-HBase-1.1, hadoop 2.6.4, hive 2.1.1

I have setup Hive for using external Phoenix tables. But after phoenix-hive.jar was included into the hive-site.xml, the hive console give exception on some operations (e.g. show databases or query with order by clause):

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/StopWatch
        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:314)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:372)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:304)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:459)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2098)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:252)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.StopWatch
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 19 more

Any suggestions are welcome.

Re: Storage Handler for Apache Hive

Posted by Jaanai <cl...@gmail.com>.

Hi ,  Stepan Migunov!
The Phoenix 4.12.0-HBase-1.1 depends on hadoop 2.7.1 version,  you shoud run it on hadoop of higher version.

Re: Storage Handler for Apache Hive

Posted by Artem Ervits <ar...@gmail.com>.

Oh, I didn't realize you specified Phoenix 4.12 in your original question,
then answer is as yes, 2.7 is minimum as Jaanai pointed out. The other
option is to choose another phonenix version with Hadoop 2.6 and your
version of HBase. Though Hadoop 2.6 been out for years so it makes sense to
upgrade, it's time :).

On Tue, Mar 27, 2018, 7:12 AM Artem Ervits <ar...@gmail.com> wrote:

> I can't say for sure but in my experience, Hadoop was at the minimum on
> 2.7. I can't tell which minimum release of Phoenix 4.x this plugin was
> merged to but Jira says it affects v. 4.5 and for that release Hadoop 2.5
> was minimum. I also see Hive 2.1 supports Hadoop 2.6 so I have no other
> leads except for classpath issues, check if hadoop-common jar is available?
>
> On Tue, Mar 27, 2018, 5:45 AM Stepan Migunov <
> Stepan.Migunov@firstlinesoftware.com> wrote:
>
>> Thank you, Artem!
>>
>> Does it mean that Phoenix-Hive integration works with Hadoop >= 2.7 only?
>>
>>
>>
>>
>>
>> *From:* Artem Ervits [mailto:artemervits@gmail.com]
>> *Sent:* Tuesday, March 27, 2018 12:10 PM
>> *To:* user@phoenix.apache.org
>> *Subject:* Re: Storage Handler for Apache Hive
>>
>>
>>
>> Stepan, you're using version of Hadoop where StopWatch class is not
>> defined
>>
>>
>>
>>
>> https://github.com/apache/hadoop/tree/release-2.6.4-RC0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java
>>
>>
>>
>> If you at least go to Hadoop 2.7, this error will disappear
>>
>>
>>
>>
>> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java
>>
>>
>>
>>
>>
>> On Tue, Mar 27, 2018, 4:47 AM Stepan Migunov <
>> stepan.migunov@firstlinesoftware.com> wrote:
>>
>> Hi,
>>
>> Phoenix 4.12.0-HBase-1.1, hadoop 2.6.4, hive 2.1.1
>>
>> I have setup Hive for using external Phoenix tables. But after
>> phoenix-hive.jar was included into the hive-site.xml, the hive console give
>> exception on some operations (e.g. show databases or query with order by
>> clause):
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/util/StopWatch
>>         at
>> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:314)
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:372)
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:304)
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:459)
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
>>         at
>> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
>>         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2098)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:252)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
>>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
>>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.util.StopWatch
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>         ... 19 more
>>
>> Any suggestions are welcome.
>>
>>

Re: Storage Handler for Apache Hive

Posted by Artem Ervits <ar...@gmail.com>.

I can't say for sure but in my experience, Hadoop was at the minimum on
2.7. I can't tell which minimum release of Phoenix 4.x this plugin was
merged to but Jira says it affects v. 4.5 and for that release Hadoop 2.5
was minimum. I also see Hive 2.1 supports Hadoop 2.6 so I have no other
leads except for classpath issues, check if hadoop-common jar is available?

On Tue, Mar 27, 2018, 5:45 AM Stepan Migunov <
Stepan.Migunov@firstlinesoftware.com> wrote:

> Thank you, Artem!
>
> Does it mean that Phoenix-Hive integration works with Hadoop >= 2.7 only?
>
>
>
>
>
> *From:* Artem Ervits [mailto:artemervits@gmail.com]
> *Sent:* Tuesday, March 27, 2018 12:10 PM
> *To:* user@phoenix.apache.org
> *Subject:* Re: Storage Handler for Apache Hive
>
>
>
> Stepan, you're using version of Hadoop where StopWatch class is not defined
>
>
>
>
> https://github.com/apache/hadoop/tree/release-2.6.4-RC0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java
>
>
>
> If you at least go to Hadoop 2.7, this error will disappear
>
>
>
>
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java
>
>
>
>
>
> On Tue, Mar 27, 2018, 4:47 AM Stepan Migunov <
> stepan.migunov@firstlinesoftware.com> wrote:
>
> Hi,
>
> Phoenix 4.12.0-HBase-1.1, hadoop 2.6.4, hive 2.1.1
>
> I have setup Hive for using external Phoenix tables. But after
> phoenix-hive.jar was included into the hive-site.xml, the hive console give
> exception on some operations (e.g. show databases or query with order by
> clause):
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/util/StopWatch
>         at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:314)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:372)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:304)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:459)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
>         at
> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
>         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2098)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:252)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
>         at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.util.StopWatch
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 19 more
>
> Any suggestions are welcome.
>
>

RE: Storage Handler for Apache Hive

Posted by Stepan Migunov <St...@firstlinesoftware.com>.

Thank you, Artem!

Does it mean that Phoenix-Hive integration works with Hadoop >= 2.7 only?





*From:* Artem Ervits [mailto:artemervits@gmail.com]
*Sent:* Tuesday, March 27, 2018 12:10 PM
*To:* user@phoenix.apache.org
*Subject:* Re: Storage Handler for Apache Hive



Stepan, you're using version of Hadoop where StopWatch class is not defined



https://github.com/apache/hadoop/tree/release-2.6.4-RC0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java



If you at least go to Hadoop 2.7, this error will disappear



https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java





On Tue, Mar 27, 2018, 4:47 AM Stepan Migunov <
stepan.migunov@firstlinesoftware.com> wrote:

Hi,

Phoenix 4.12.0-HBase-1.1, hadoop 2.6.4, hive 2.1.1

I have setup Hive for using external Phoenix tables. But after
phoenix-hive.jar was included into the hive-site.xml, the hive console give
exception on some operations (e.g. show databases or query with order by
clause):

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/util/StopWatch
        at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:314)
        at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:372)
        at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:304)
        at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:459)
        at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
        at
org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2098)
        at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:252)
        at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
        at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.util.StopWatch
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 19 more

Any suggestions are welcome.

Re: Storage Handler for Apache Hive

Posted by Artem Ervits <ar...@gmail.com>.

Stepan, you're using version of Hadoop where StopWatch class is not defined

https://github.com/apache/hadoop/tree/release-2.6.4-RC0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java

If you at least go to Hadoop 2.7, this error will disappear

https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StopWatch.java


On Tue, Mar 27, 2018, 4:47 AM Stepan Migunov <
stepan.migunov@firstlinesoftware.com> wrote:

> Hi,
>
> Phoenix 4.12.0-HBase-1.1, hadoop 2.6.4, hive 2.1.1
>
> I have setup Hive for using external Phoenix tables. But after
> phoenix-hive.jar was included into the hive-site.xml, the hive console give
> exception on some operations (e.g. show databases or query with order by
> clause):
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/util/StopWatch
>         at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:314)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:372)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:304)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:459)
>         at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
>         at
> org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
>         at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2098)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:252)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
>         at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.util.StopWatch
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 19 more
>
> Any suggestions are welcome.
>
>
>