You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by vijay shinde <vi...@gmail.com> on 2012/07/25 10:45:41 UTC
HBASE and HIVE Integration
I am facing issue while executing HIVE queries on HBASE-HIVE integration.
I followed the wiki hbase-hive integration
https://cwiki.apache.org/Hive/hbaseintegration.html
I have already passed all the required jars for auxpath in hive-site.xml
file.
I am using Cloudera CDH demo VM.. Any help would be highly appreciated
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
Re: HBASE and HIVE Integration
Posted by "kulkarni.swarnim@gmail.com" <ku...@gmail.com>.
Can you also post logs from "/tmp/<USER>/hive.log". That might contain some
info on your job failure.
On Wed, Jul 25, 2012 at 8:28 AM, vijay shinde <vi...@gmail.com>wrote:
> Hi Bejoy,
>
> Thanks for quick reply. Here are some additional details
>
> Cloudera Version - CDH3U4
>
> *hive-site.xml*
> **
> *<property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
> </property>
> *
> *Execution Log*
>
>
> 1. *start zookeeper*
>
> [root@localhost zookeeper]# ./bin/zkServer.sh start
>
>
>
> 2. *start hbase*
>
>
>
> 3. *start hive. I am setting hive jars in hive-site.xml*
>
>
>
> ./bin/hive -hiveconf hbase.master=127.0.1.1:60010
>
>
> 4. *Create new HBase table which is to be managed by Hive*
> **
>
> CREATE TABLE hive_hbasetable_k(key int, value string)
>
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
>
> TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
>
>
>
> 5. *Create a logical table pokes in Hive*
>
> CREATE TABLE pokes (foo INT, bar STRING);
>
> 6. *HIve error while inserting the data from Hive Poke table to HBASE
> table*
>
> *hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;*
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
>
> *FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> *
> Let me know if you need any additonal information.
>
> Thanks,
> Vijay
>
> On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
>
>> **
>> Hi Vijay
>>
>> Can you share more details like
>>
>> The CDH Version/Hive version you are using
>>
>> Steps you followed for hive hbase integration with the values you set
>>
>> The DDL used for hive hbase integration
>>
>> The actual error from failed map reduce task
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> ------------------------------
>> *From: *vijay shinde <vi...@gmail.com>
>> *Date: *Wed, 25 Jul 2012 04:45:41 -0400
>> *To: *<us...@hive.apache.org>
>> *ReplyTo: *user@hive.apache.org
>> *Subject: *HBASE and HIVE Integration
>>
>> I am facing issue while executing HIVE queries on HBASE-HIVE integration.
>> I followed the wiki hbase-hive integration
>> https://cwiki.apache.org/Hive/hbaseintegration.html
>>
>> I have already passed all the required jars for auxpath in hive-site.xml
>> file.
>> I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>>
>> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
>> foo=98;
>> Total MapReduce jobs = 1
>> Launching Job 1 out of 1
>> Number of reduce tasks is set to 0 since there's no reduce operator
>> Starting Job = job_201207250246_0005, Tracking URL =
>> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
>> 0.0.0.0:8021 -kill job_201207250246_0005
>> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
>> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
>> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
>> Ended Job = job_201207250246_0005 with errors
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.MapRedTask
>> hive>
>>
>>
>
--
Swarnim
Re: HBASE and HIVE Integration
Posted by Vijay <vi...@gmail.com>.
Sent from my iPhone
On Jul 25, 2012, at 8:58 PM, vijay shinde <vi...@gmail.com> wrote:
> Hi Bejoy,
>
> I made some changes as per your suggetion.
>
> Here is the error from the http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
>
> Error: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$
>
> I went ahead and updated the hadoop-env.sh file and set the class path for hbase and zookeeper as follows:
> # Extra Java CLASSPATH elements. Optional.
> export HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
>
> Here is snippet of hive-site.xml file
>
> <name>hive.aux.jars.path</name>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
> </property>
> <name>hbase.zookeeper.quorum</name>
> <value>localhost</value>
> </property>
>
> Error message while executing hive query
>
> [root@localhost hive]# ./bin/hive
> Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207251858_0004, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207251858_0004
> 2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207251858_0004 with errors
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
> I am lost need help badly!!!!!!!!!!!!!
>
> Vijay
>
>
>
> On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <be...@yahoo.com> wrote:
> Hi Vijay
>
> You have provided the hbase master directly. (It is fine for single node hbase installation). But still can you try providing the zookeeper quorum instead.
>
> If that doesn't work as well , please post the error log from the mapreduce tasks?
>
> Just go the jobtracker page and drill down on the corresponding job to get the failed tasks. From each failed tasks you can get the error logs.
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>
>
> Regards
> Bejoy KS
>
> From: vijay shinde <vi...@gmail.com>
> To: user@hive.apache.org; bejoy_ks@yahoo.com
> Sent: Wednesday, July 25, 2012 6:58 PM
> Subject: Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> Thanks for quick reply. Here are some additional details
>
> Cloudera Version - CDH3U4
>
> hive-site.xml
> <property>
> <name>hive.aux.jars.path</name>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
> </property>
>
> Execution Log
>
> 1. start zookeeper
> [root@localhost zookeeper]# ./bin/zkServer.sh start
>
> 2. start hbase
>
> 3. start hive. I am setting hive jars in hive-site.xml
>
> ./bin/hive -hiveconf hbase.master=127.0.1.1:60010
>
> 4. Create new HBase table which is to be managed by Hive
>
> CREATE TABLE hive_hbasetable_k(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
>
> 5. Create a logical table pokes in Hive
> CREATE TABLE pokes (foo INT, bar STRING);
> 6. HIve error while inserting the data from Hive Poke table to HBASE table
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
>
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
> Let me know if you need any additonal information.
>
> Thanks,
> Vijay
>
> On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
> Hi Vijay
>
> Can you share more details like
>
> The CDH Version/Hive version you are using
>
> Steps you followed for hive hbase integration with the values you set
>
> The DDL used for hive hbase integration
>
> The actual error from failed map reduce task
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> From: vijay shinde <vi...@gmail.com>
> Date: Wed, 25 Jul 2012 04:45:41 -0400
> To: <us...@hive.apache.org>
> ReplyTo: user@hive.apache.org
> Subject: HBASE and HIVE Integration
>
> I am facing issue while executing HIVE queries on HBASE-HIVE integration.
> I followed the wiki hbase-hive integration https://cwiki.apache.org/Hive/hbaseintegration.html
>
> I have already passed all the required jars for auxpath in hive-site.xml file.
> I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
>
>
>
>
Re: HBASE and HIVE Integration
Posted by vijay shinde <vi...@gmail.com>.
Hi Bejoy,
My normal hive query works but I am receiving an error while inserting data
into hbase table..
see attached log..
hive> select * from pokes where foo=9;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207260911_0005, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207260911_0005
2012-07-26 15:07:29,827 Stage-1 map = 0%, reduce = 0%
2012-07-26 15:07:34,942 Stage-1 map = 100%, reduce = 0%
2012-07-26 15:07:39,292 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201207260911_0005
OK
9 val_9
Time taken: 19.359 seconds
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
foo=9;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207260911_0006, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0006
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207260911_0006
2012-07-26 15:08:14,784 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:09:14,977 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:10:15,947 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:11:16,814 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:12:17,692 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:13:18,615 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:14:19,448 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:15:20,353 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:16:21,197 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:17:21,972 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:18:22,148 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:19:23,061 Stage-0 map = 0%, reduce = 0%
2012-07-26 15:19:24,079 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207260911_0006 with errors
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
On Thu, Jul 26, 2012 at 3:39 PM, Bejoy Ks <be...@yahoo.com> wrote:
> Hi Vijay
>
> Your current error looks like some issue with the Select query. Is the
> select query working as desired?
> hive> SELECT * FROM pokes where foo=98;
>
> Regards
> Bejoy KS
>
> ------------------------------
> *From:* vijay shinde <vi...@gmail.com>
> *To:* user@hive.apache.org; Bejoy Ks <be...@yahoo.com>
> *Sent:* Friday, July 27, 2012 12:24 AM
>
> *Subject:* Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> Thanks for your reply. My Hbase is working fine independently.
> I was able reolve the zookeeper error. I had issues with hive-site.xml
> configuration..
>
> Now my job run little bit longer and this is the lastest error
>
> I am on Cloudera CDH3U4 demo VM
>
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207260911_0003, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0003
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207260911_0003
> 2012-07-26 13:34:55,900 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:35:56,207 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:36:56,364 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:37:57,342 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:38:57,398 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:39:57,467 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:40:57,852 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:41:58,724 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:42:59,700 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:44:00,582 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:45:00,977 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:46:01,110 Stage-0 map = 0%, reduce = 0%
> 2012-07-26 13:46:05,443 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207260911_0003 with errors
>
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
> foo=98;
>
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"foo":98,"bar":"val_98"}
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"foo":98,"bar":"val_98"}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
> ... 8 more
> Caused by: or
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
>
>
> On Thu, Jul 26, 2012 at 9:55 AM, Bejoy Ks <be...@yahoo.com> wrote:
>
> Hi Vijay
>
> Is your hbase working independently without any issues. I mean, are you
> able to insert data into hbase tables without using hive integration?
>
> Was the same error message thrown when you directly provided hbase.master
> instead of zookeeper quorum?
>
> Regards
> Bejoy KS
>
>
> ________________________________
> From: Hezhiqiang (Ransom) <ra...@huawei.com>
> To: "user@hive.apache.org" <us...@hive.apache.org>
> Cc: Zhaojun (Terry) <te...@huawei.com>
> Sent: Thursday, July 26, 2012 9:06 AM
> Subject: RE: HBASE and HIVE Integration
>
>
>
> Hi，Vijay
> You need to add zookeeper.jar to the hive-site.xml
> <name>hive.aux.jars.path</name>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar,
> file:///usr/lib/hive/lib/ zookeeper-3.4.3.jar</value>
> </property>
> <name>hbase.zookeeper.quorum</name>
> <value>localhost</value>
> </property>
>
>
> Regards
> Ransom
>
> From:vijay shinde [mailto:vijaysanjyot@gmail.com]
> Sent: Thursday, July 26, 2012 8:59 AM
> To: user@hive.apache.org; Bejoy Ks
> Subject: Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> I made some changes as per your suggetion.
>
> Here is the error from the
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
>
> Error: java.lang.ClassNotFoundException:
> org.apache.zookeeper.KeeperException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$
>
> I went ahead and updated the hadoop-env.sh file and set the class path for
> hbase and zookeeper as follows:
> # Extra Java CLASSPATH elements. Optional.
> export
> HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
>
> Here is snippet of hive-site.xml file
>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
> </property>
> <name>hbase.zookeeper.quorum</name>
> <value>localhost</value>
> </property>
>
> Error message while executing hive query
>
> [root@localhost hive]# ./bin/hive
> Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207251858_0004, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207251858_0004
> 2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207251858_0004 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
> I am lost need help badly!!!!!!!!!!!!!
>
> Vijay
>
>
>
> On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <be...@yahoo.com> wrote:
> Hi Vijay
>
> You have provided the hbase master directly. (It is fine for single node
> hbase installation). But still can you try providing the zookeeper quorum
> instead.
>
> If that doesn't work as well , please post the error log from the
> mapreduce tasks?
>
> Just go the jobtracker page and drill down on the corresponding job to get
> the failed tasks. From each failed tasks you can get the error logs.
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>
>
> Regards
> Bejoy KS
>
>
> ________________________________
>
> From:vijay shinde <vi...@gmail.com>
> To:user@hive.apache.org; bejoy_ks@yahoo.com
> Sent: Wednesday, July 25, 2012 6:58 PM
> Subject: Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> Thanks for quick reply. Here are some additional details
>
> Cloudera Version - CDH3U4
>
> hive-site.xml
>
>
> <property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
> </property>
> Execution Log
>
> 1. start zookeeper
> [root@localhost zookeeper]# ./bin/zkServer.sh start
>
> 2. start hbase
>
> 3. start hive. I am setting hive jars in hive-site.xml
>
> ./bin/hive -hiveconf hbase.master=127.0.1.1:60010
>
> 4. Create new HBase table which is to be managed by Hive
>
> CREATE TABLE hive_hbasetable_k(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
>
> 5. Create a logical table pokes in Hive
> CREATE TABLE pokes (foo INT, bar STRING);
> 6. HIve error while inserting the data from Hive Poke table to HBASE table
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> Let me know if you need any additonal information.
>
> Thanks,
> Vijay
> On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
> Hi Vijay
>
> Can you share more details like
>
> The CDH Version/Hive version you are using
>
> Steps you followed for hive hbase integration with the values you set
>
> The DDL used for hive hbase integration
>
> The actual error from failed map reduce task
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
>
> ________________________________
>
> From: vijay shinde <vi...@gmail.com>
> Date: Wed, 25 Jul 2012 04:45:41 -0400
> To: <us...@hive.apache.org>
> ReplyTo: user@hive.apache.org
> Subject: HBASE and HIVE Integration
>
> I am facing issue while executing HIVE queries on HBASE-HIVE integration.
> I followed the wiki hbase-hive integration
> https://cwiki.apache.org/Hive/hbaseintegration.html
>
> I have already passed all the required jars for auxpath in hive-site.xml
> file.
> I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
>
>
>
>
Re: HBASE and HIVE Integration
Posted by Bejoy Ks <be...@yahoo.com>.
Hi Vijay
Your current error looks like some issue with the Select query. Is the select query working as desired?
hive> SELECT * FROM pokes where foo=98;
Regards
Bejoy KS
________________________________
From: vijay shinde <vi...@gmail.com>
To: user@hive.apache.org; Bejoy Ks <be...@yahoo.com>
Sent: Friday, July 27, 2012 12:24 AM
Subject: Re: HBASE and HIVE Integration
Hi Bejoy,
Thanks for your reply. My Hbase is working fine independently.
I was able reolve the zookeeper error. I had issues with hive-site.xml configuration..
Now my job run little bit longer and this is the lastest error
I am on Cloudera CDH3U4 demo VM
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207260911_0003, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0003
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207260911_0003
2012-07-26 13:34:55,900 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:35:56,207 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:36:56,364 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:37:57,342 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:38:57,398 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:39:57,467 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:40:57,852 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:41:58,724 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:42:59,700 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:44:00,582 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:45:00,977 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:46:01,110 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:46:05,443 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207260911_0003 with errors
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where foo=98;
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"foo":98,"bar":"val_98"}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"foo":98,"bar":"val_98"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: or
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
On Thu, Jul 26, 2012 at 9:55 AM, Bejoy Ks <be...@yahoo.com> wrote:
Hi Vijay
>
>Is your hbase working independently without any issues. I mean, are you able to insert data into hbase tables without using hive integration?
>
>Was the same error message thrown when you directly provided hbase.master instead of zookeeper quorum?
>
>Regards
>Bejoy KS
>
>
>________________________________
>From: Hezhiqiang (Ransom) <ra...@huawei.com>
>To: "user@hive.apache.org" <us...@hive.apache.org>
>Cc: Zhaojun (Terry) <te...@huawei.com>
>Sent: Thursday, July 26, 2012 9:06 AM
>Subject: RE: HBASE and HIVE Integration
>
>
>
>
>Hi，Vijay
> You need to add zookeeper.jar to the hive-site.xml
><name>hive.aux.jars.path</name>
><value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar, file:///usr/lib/hive/lib/ zookeeper-3.4.3.jar</value>
></property>
><name>hbase.zookeeper.quorum</name>
><value>localhost</value>
></property>
>
>
>Regards
>Ransom
>
>From:vijay shinde [mailto:vijaysanjyot@gmail.com]
>Sent: Thursday, July 26, 2012 8:59 AM
>To: user@hive.apache.org; Bejoy Ks
>Subject: Re: HBASE and HIVE Integration
>
>Hi Bejoy,
>
>I made some changes as per your suggetion.
>
>Here is the error from the http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
>
>Error: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$
>
>I went ahead and updated the hadoop-env.sh file and set the class path for hbase and zookeeper as follows:
># Extra Java CLASSPATH elements. Optional.
>export HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
>
>Here is snippet of hive-site.xml file
>
><name>hive.aux.jars.path</name>
><value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
></property>
><name>hbase.zookeeper.quorum</name>
><value>localhost</value>
></property>
>
>Error message while executing hive query
>
>[root@localhost hive]# ./bin/hive
>Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
>hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where foo=98;
>Total MapReduce jobs = 1
>Launching Job 1 out of 1
>Number of reduce tasks is set to 0 since there's no reduce operator
>Starting Job = job_201207251858_0004, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
>Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207251858_0004
>2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
>2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
>Ended Job = job_201207251858_0004 with errors
>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>hive>
>
>I am lost need help badly!!!!!!!!!!!!!
>
>Vijay
>
>
>
>On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <be...@yahoo.com> wrote:
>Hi Vijay
>
>You have provided the hbase master directly. (It is fine for single node hbase installation). But still can you try providing the zookeeper quorum instead.
>
>If that doesn't work as well , please post the error log from the mapreduce tasks?
>
>Just go the jobtracker page and drill down on the corresponding job to get the failed tasks. From each failed tasks you can get the error logs.
>http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>
>
>Regards
>Bejoy KS
>
>
>________________________________
>
>From:vijay shinde <vi...@gmail.com>
>To:user@hive.apache.org; bejoy_ks@yahoo.com
>Sent: Wednesday, July 25, 2012 6:58 PM
>Subject: Re: HBASE and HIVE Integration
>
>Hi Bejoy,
>
>Thanks for quick reply. Here are some additional details
>
>Cloudera Version - CDH3U4
>
>hive-site.xml
>
>
><property>
><name>hive.aux.jars.path</name>
><value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
></property>
>Execution Log
>
>1. start zookeeper
>[root@localhost zookeeper]# ./bin/zkServer.sh start
>
>2. start hbase
>
>3. start hive. I am setting hive jars in hive-site.xml
>
>./bin/hive -hiveconf hbase.master=127.0.1.1:60010
>
>4. Create new HBase table which is to be managed by Hive
>
>CREATE TABLE hive_hbasetable_k(key int, value string)
>STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
>TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
>
>5. Create a logical table pokes in Hive
>CREATE TABLE pokes (foo INT, bar STRING);
>6. HIve error while inserting the data from Hive Poke table to HBASE table
>hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
>Total MapReduce jobs = 1
>Launching Job 1 out of 1
>Number of reduce tasks is set to 0 since there's no reduce operator
>Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
>2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
>2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
>2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
>Ended Job = job_201207250246_0005 with errors
>
>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>Let me know if you need any additonal information.
>
>Thanks,
>Vijay
>On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
>Hi Vijay
>
>Can you share more details like
>
>The CDH Version/Hive version you are using
>
>Steps you followed for hive hbase integration with the values you set
>
>The DDL used for hive hbase integration
>
>The actual error from failed map reduce task
>Regards
>Bejoy KS
>
>Sent from handheld, please excuse typos.
>
>________________________________
>
>From: vijay shinde <vi...@gmail.com>
>Date: Wed, 25 Jul 2012 04:45:41 -0400
>To: <us...@hive.apache.org>
>ReplyTo: user@hive.apache.org
>Subject: HBASE and HIVE Integration
>
>I am facing issue while executing HIVE queries on HBASE-HIVE integration.
>I followed the wiki hbase-hive integration https://cwiki.apache.org/Hive/hbaseintegration.html
>
>I have already passed all the required jars for auxpath in hive-site.xml file.
>I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
>hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
>Total MapReduce jobs = 1
>Launching Job 1 out of 1
>Number of reduce tasks is set to 0 since there's no reduce operator
>Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
>2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
>2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
>2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
>Ended Job = job_201207250246_0005 with errors
>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>hive>
>
Re: HBASE and HIVE Integration
Posted by vijay shinde <vi...@gmail.com>.
Hi Bejoy,
Thanks for your reply. My Hbase is working fine independently.
I was able reolve the zookeeper error. I had issues with hive-site.xml
configuration..
Now my job run little bit longer and this is the lastest error
I am on Cloudera CDH3U4 demo VM
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207260911_0003, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207260911_0003
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207260911_0003
2012-07-26 13:34:55,900 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:35:56,207 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:36:56,364 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:37:57,342 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:38:57,398 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:39:57,467 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:40:57,852 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:41:58,724 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:42:59,700 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:44:00,582 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:45:00,977 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:46:01,110 Stage-0 map = 0%, reduce = 0%
2012-07-26 13:46:05,443 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207260911_0003 with errors
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
foo=98;
java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"foo":98,"bar":"val_98"}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row {"foo":98,"bar":"val_98"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: or
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
On Thu, Jul 26, 2012 at 9:55 AM, Bejoy Ks <be...@yahoo.com> wrote:
> Hi Vijay
>
> Is your hbase working independently without any issues. I mean, are you
> able to insert data into hbase tables without using hive integration?
>
> Was the same error message thrown when you directly provided hbase.master
> instead of zookeeper quorum?
>
> Regards
> Bejoy KS
>
>
> ________________________________
> From: Hezhiqiang (Ransom) <ra...@huawei.com>
> To: "user@hive.apache.org" <us...@hive.apache.org>
> Cc: Zhaojun (Terry) <te...@huawei.com>
> Sent: Thursday, July 26, 2012 9:06 AM
> Subject: RE: HBASE and HIVE Integration
>
>
>
> Hi，Vijay
> You need to add zookeeper.jar to the hive-site.xml
> <name>hive.aux.jars.path</name>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar,
> file:///usr/lib/hive/lib/ zookeeper-3.4.3.jar</value>
> </property>
> <name>hbase.zookeeper.quorum</name>
> <value>localhost</value>
> </property>
>
>
> Regards
> Ransom
>
> From:vijay shinde [mailto:vijaysanjyot@gmail.com]
> Sent: Thursday, July 26, 2012 8:59 AM
> To: user@hive.apache.org; Bejoy Ks
> Subject: Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> I made some changes as per your suggetion.
>
> Here is the error from the
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
>
> Error: java.lang.ClassNotFoundException:
> org.apache.zookeeper.KeeperException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$
>
> I went ahead and updated the hadoop-env.sh file and set the class path for
> hbase and zookeeper as follows:
> # Extra Java CLASSPATH elements. Optional.
> export
> HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
>
> Here is snippet of hive-site.xml file
>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
> </property>
> <name>hbase.zookeeper.quorum</name>
> <value>localhost</value>
> </property>
>
> Error message while executing hive query
>
> [root@localhost hive]# ./bin/hive
> Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207251858_0004, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207251858_0004
> 2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207251858_0004 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
> I am lost need help badly!!!!!!!!!!!!!
>
> Vijay
>
>
>
> On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <be...@yahoo.com> wrote:
> Hi Vijay
>
> You have provided the hbase master directly. (It is fine for single node
> hbase installation). But still can you try providing the zookeeper quorum
> instead.
>
> If that doesn't work as well , please post the error log from the
> mapreduce tasks?
>
> Just go the jobtracker page and drill down on the corresponding job to get
> the failed tasks. From each failed tasks you can get the error logs.
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>
>
> Regards
> Bejoy KS
>
>
> ________________________________
>
> From:vijay shinde <vi...@gmail.com>
> To:user@hive.apache.org; bejoy_ks@yahoo.com
> Sent: Wednesday, July 25, 2012 6:58 PM
> Subject: Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> Thanks for quick reply. Here are some additional details
>
> Cloudera Version - CDH3U4
>
> hive-site.xml
>
>
> <property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
> </property>
> Execution Log
>
> 1. start zookeeper
> [root@localhost zookeeper]# ./bin/zkServer.sh start
>
> 2. start hbase
>
> 3. start hive. I am setting hive jars in hive-site.xml
>
> ./bin/hive -hiveconf hbase.master=127.0.1.1:60010
>
> 4. Create new HBase table which is to be managed by Hive
>
> CREATE TABLE hive_hbasetable_k(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
>
> 5. Create a logical table pokes in Hive
> CREATE TABLE pokes (foo INT, bar STRING);
> 6. HIve error while inserting the data from Hive Poke table to HBASE table
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> Let me know if you need any additonal information.
>
> Thanks,
> Vijay
> On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
> Hi Vijay
>
> Can you share more details like
>
> The CDH Version/Hive version you are using
>
> Steps you followed for hive hbase integration with the values you set
>
> The DDL used for hive hbase integration
>
> The actual error from failed map reduce task
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
>
> ________________________________
>
> From: vijay shinde <vi...@gmail.com>
> Date: Wed, 25 Jul 2012 04:45:41 -0400
> To: <us...@hive.apache.org>
> ReplyTo: user@hive.apache.org
> Subject: HBASE and HIVE Integration
>
> I am facing issue while executing HIVE queries on HBASE-HIVE integration.
> I followed the wiki hbase-hive integration
> https://cwiki.apache.org/Hive/hbaseintegration.html
>
> I have already passed all the required jars for auxpath in hive-site.xml
> file.
> I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
Re: HBASE and HIVE Integration
Posted by Bejoy Ks <be...@yahoo.com>.
Hi Vijay
Is your hbase working independently without any issues. I mean, are you able to insert data into hbase tables without using hive integration?
Was the same error message thrown when you directly provided hbase.master instead of zookeeper quorum?
Regards
Bejoy KS
________________________________
From: Hezhiqiang (Ransom) <ra...@huawei.com>
To: "user@hive.apache.org" <us...@hive.apache.org>
Cc: Zhaojun (Terry) <te...@huawei.com>
Sent: Thursday, July 26, 2012 9:06 AM
Subject: RE: HBASE and HIVE Integration
Hi，Vijay
You need to add zookeeper.jar to the hive-site.xml
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar, file:///usr/lib/hive/lib/ zookeeper-3.4.3.jar</value>
</property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
Regards
Ransom
From:vijay shinde [mailto:vijaysanjyot@gmail.com]
Sent: Thursday, July 26, 2012 8:59 AM
To: user@hive.apache.org; Bejoy Ks
Subject: Re: HBASE and HIVE Integration
Hi Bejoy,
I made some changes as per your suggetion.
Here is the error from the http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
Error: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$
I went ahead and updated the hadoop-env.sh file and set the class path for hbase and zookeeper as follows:
# Extra Java CLASSPATH elements. Optional.
export HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
Here is snippet of hive-site.xml file
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
</property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
Error message while executing hive query
[root@localhost hive]# ./bin/hive
Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207251858_0004, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207251858_0004
2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207251858_0004 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
I am lost need help badly!!!!!!!!!!!!!
Vijay
On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <be...@yahoo.com> wrote:
Hi Vijay
You have provided the hbase master directly. (It is fine for single node hbase installation). But still can you try providing the zookeeper quorum instead.
If that doesn't work as well , please post the error log from the mapreduce tasks?
Just go the jobtracker page and drill down on the corresponding job to get the failed tasks. From each failed tasks you can get the error logs.
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Regards
Bejoy KS
________________________________
From:vijay shinde <vi...@gmail.com>
To:user@hive.apache.org; bejoy_ks@yahoo.com
Sent: Wednesday, July 25, 2012 6:58 PM
Subject: Re: HBASE and HIVE Integration
Hi Bejoy,
Thanks for quick reply. Here are some additional details
Cloudera Version - CDH3U4
hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
</property>
Execution Log
1. start zookeeper
[root@localhost zookeeper]# ./bin/zkServer.sh start
2. start hbase
3. start hive. I am setting hive jars in hive-site.xml
./bin/hive -hiveconf hbase.master=127.0.1.1:60010
4. Create new HBase table which is to be managed by Hive
CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
5. Create a logical table pokes in Hive
CREATE TABLE pokes (foo INT, bar STRING);
6. HIve error while inserting the data from Hive Poke table to HBASE table
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
Let me know if you need any additonal information.
Thanks,
Vijay
On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
Hi Vijay
Can you share more details like
The CDH Version/Hive version you are using
Steps you followed for hive hbase integration with the values you set
The DDL used for hive hbase integration
The actual error from failed map reduce task
Regards
Bejoy KS
Sent from handheld, please excuse typos.
________________________________
From: vijay shinde <vi...@gmail.com>
Date: Wed, 25 Jul 2012 04:45:41 -0400
To: <us...@hive.apache.org>
ReplyTo: user@hive.apache.org
Subject: HBASE and HIVE Integration
I am facing issue while executing HIVE queries on HBASE-HIVE integration.
I followed the wiki hbase-hive integration https://cwiki.apache.org/Hive/hbaseintegration.html
I have already passed all the required jars for auxpath in hive-site.xml file.
I am using Cloudera CDH demo VM.. Any help would be highly appreciated
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
RE: HBASE and HIVE Integration
Posted by "Hezhiqiang (Ransom)" <ra...@huawei.com>.
Hi，Vijay
You need to add zookeeper.jar to the hive-site.xml
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar, file:///usr/lib/hive/lib/ zookeeper-3.4.3.jar </value>
</property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
Regards
Ransom
From: vijay shinde [mailto:vijaysanjyot@gmail.com]
Sent: Thursday, July 26, 2012 8:59 AM
To: user@hive.apache.org; Bejoy Ks
Subject: Re: HBASE and HIVE Integration
Hi Bejoy,
I made some changes as per your suggetion.
Here is the error from the http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
Error: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$
I went ahead and updated the hadoop-env.sh file and set the class path for hbase and zookeeper as follows:
# Extra Java CLASSPATH elements. Optional.
export HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
Here is snippet of hive-site.xml file
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
</property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
Error message while executing hive query
[root@localhost hive]# ./bin/hive
Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207251858_0004, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021<http://0.0.0.0:8021> -kill job_201207251858_0004
2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207251858_0004 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
I am lost need help badly!!!!!!!!!!!!!
Vijay
On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <be...@yahoo.com>> wrote:
Hi Vijay
You have provided the hbase master directly. (It is fine for single node hbase installation). But still can you try providing the zookeeper quorum instead.
If that doesn't work as well , please post the error log from the mapreduce tasks?
Just go the jobtracker page and drill down on the corresponding job to get the failed tasks. From each failed tasks you can get the error logs.
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Regards
Bejoy KS
________________________________
From: vijay shinde <vi...@gmail.com>>
To: user@hive.apache.org<ma...@hive.apache.org>; bejoy_ks@yahoo.com<ma...@yahoo.com>
Sent: Wednesday, July 25, 2012 6:58 PM
Subject: Re: HBASE and HIVE Integration
Hi Bejoy,
Thanks for quick reply. Here are some additional details
Cloudera Version - CDH3U4
hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
</property>
Execution Log
1. start zookeeper
[root@localhost zookeeper]# ./bin/zkServer.sh start
2. start hbase
3. start hive. I am setting hive jars in hive-site.xml
./bin/hive -hiveconf hbase.master=127.0.1.1:60010<http://127.0.1.1:60010/>
4. Create new HBase table which is to be managed by Hive
CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name<http://hbase.table.name/>" = "hivehbasek");
5. Create a logical table pokes in Hive
CREATE TABLE pokes (foo INT, bar STRING);
6. HIve error while inserting the data from Hive Poke table to HBASE table
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021<http://0.0.0.0:8021/> -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
Let me know if you need any additonal information.
Thanks,
Vijay
On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com>> wrote:
Hi Vijay
Can you share more details like
The CDH Version/Hive version you are using
Steps you followed for hive hbase integration with the values you set
The DDL used for hive hbase integration
The actual error from failed map reduce task
Regards
Bejoy KS
Sent from handheld, please excuse typos.
________________________________
From: vijay shinde <vi...@gmail.com>>
Date: Wed, 25 Jul 2012 04:45:41 -0400
To: <us...@hive.apache.org>>
ReplyTo: user@hive.apache.org<ma...@hive.apache.org>
Subject: HBASE and HIVE Integration
I am facing issue while executing HIVE queries on HBASE-HIVE integration.
I followed the wiki hbase-hive integration https://cwiki.apache.org/Hive/hbaseintegration.html
I have already passed all the required jars for auxpath in hive-site.xml file.
I am using Cloudera CDH demo VM.. Any help would be highly appreciated
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021<http://0.0.0.0:8021/> -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
Re: HBASE and HIVE Integration
Posted by vijay shinde <vi...@gmail.com>.
Hi Bejoy,
I made some changes as per your suggetion.
Here is the error from the
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004 Job:
Error: java.lang.ClassNotFoundException:
org.apache.zookeeper.KeeperException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$
I went ahead and updated the hadoop-env.sh file and set the class path for
hbase and zookeeper as follows:
# Extra Java CLASSPATH elements. Optional.
export
HADOOP_CLASSPATH="/usr/lib/hive/lib/hbase-0.92.jar:/usr/lib/hive/lib/zookeeper-3.4.3.jar:$HADOOP_CLASSPATH"
Here is snippet of hive-site.xml file
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u4.jar,file:///usr/lib/hive/lib/hbase-0.92.jar,file:///usr/lib/hive/lib/zookeeper-3.4.3.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u4.jar</value>
</property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
Error message while executing hive query
[root@localhost hive]# ./bin/hive
Hive history file=/tmp/root/hive_job_log_root_201207252044_1993919630.txt
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes where
foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207251858_0004, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207251858_0004
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207251858_0004
2012-07-25 20:46:38,207 Stage-0 map = 0%, reduce = 0%
2012-07-25 20:47:35,920 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207251858_0004 with errors
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
hive>
I am lost need help badly!!!!!!!!!!!!!
Vijay
On Wed, Jul 25, 2012 at 9:47 AM, Bejoy Ks <be...@yahoo.com> wrote:
> Hi Vijay
>
> You have provided the hbase master directly. (It is fine for single node
> hbase installation). But still can you try providing the zookeeper quorum
> instead.
>
> If that doesn't work as well , please post the error log from the
> mapreduce tasks?
>
> Just go the jobtracker page and drill down on the corresponding job to get
> the failed tasks. From each failed tasks you can get the error logs.
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>
>
> Regards
> Bejoy KS
>
> ------------------------------
> *From:* vijay shinde <vi...@gmail.com>
> *To:* user@hive.apache.org; bejoy_ks@yahoo.com
> *Sent:* Wednesday, July 25, 2012 6:58 PM
> *Subject:* Re: HBASE and HIVE Integration
>
> Hi Bejoy,
>
> Thanks for quick reply. Here are some additional details
>
> Cloudera Version - CDH3U4
>
> *hive-site.xml*
> **
> *<property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
> </property>
> *
> *Execution Log*
>
> 1. *start zookeeper*
> [root@localhost zookeeper]# ./bin/zkServer.sh start
>
> 2. *start hbase*
>
> 3. *start hive. I am setting hive jars in hive-site.xml*
>
> ./bin/hive -hiveconf hbase.master=127.0.1.1:60010
>
> 4. *Create new HBase table which is to be managed by Hive*
> **
> CREATE TABLE hive_hbasetable_k(key int, value string)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
> TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
>
> 5. *Create a logical table pokes in Hive*
> CREATE TABLE pokes (foo INT, bar STRING);
> 6. *HIve error while inserting the data from Hive Poke table to HBASE
> table*
> *hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;*
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
>
> *FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> *
> Let me know if you need any additonal information.
>
> Thanks,
> Vijay
>
> On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
>
> **
> Hi Vijay
>
> Can you share more details like
>
> The CDH Version/Hive version you are using
>
> Steps you followed for hive hbase integration with the values you set
>
> The DDL used for hive hbase integration
>
> The actual error from failed map reduce task
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> ------------------------------
> *From: *vijay shinde <vi...@gmail.com>
> *Date: *Wed, 25 Jul 2012 04:45:41 -0400
> *To: *<us...@hive.apache.org>
> *ReplyTo: *user@hive.apache.org
> *Subject: *HBASE and HIVE Integration
>
> I am facing issue while executing HIVE queries on HBASE-HIVE integration.
> I followed the wiki hbase-hive integration
> https://cwiki.apache.org/Hive/hbaseintegration.html
>
> I have already passed all the required jars for auxpath in hive-site.xml
> file.
> I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
>
>
>
>
Re: HBASE and HIVE Integration
Posted by Bejoy Ks <be...@yahoo.com>.
Hi Vijay
You have provided the hbase master directly. (It is fine for single node hbase installation). But still can you try providing the zookeeper quorum instead.
If that doesn't work as well , please post the error log from the mapreduce tasks?
Just go the jobtracker page and drill down on the corresponding job to get the failed tasks. From each failed tasks you can get the error logs.
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Regards
Bejoy KS
________________________________
From: vijay shinde <vi...@gmail.com>
To: user@hive.apache.org; bejoy_ks@yahoo.com
Sent: Wednesday, July 25, 2012 6:58 PM
Subject: Re: HBASE and HIVE Integration
Hi Bejoy,
Thanks for quick reply. Here are some additional details
Cloudera Version - CDH3U4
hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
</property>
Execution Log
1. start zookeeper
[root@localhost zookeeper]# ./bin/zkServer.sh start
2. start hbase
3. start hive. I am setting hive jars in hive-site.xml
./bin/hive -hiveconf hbase.master=127.0.1.1:60010
4. Create new HBase table which is to be managed by Hive
CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
5. Create a logical table pokes in Hive
CREATE TABLE pokes (foo INT, bar STRING);
6. HIve error while inserting the data from Hive Poke table to HBASE table
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
Let me know if you need any additonal information.
Thanks,
Vijay
On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
>Hi Vijay
>
>Can you share more details like
>
>The CDH Version/Hive version you are using
>
>Steps you followed for hive hbase integration with the values you set
>
>The DDL used for hive hbase integration
>
>The actual error from failed map reduce task
>
>
>Regards
>Bejoy KS
>
>Sent from handheld, please excuse typos.
>________________________________
>
>From: vijay shinde <vi...@gmail.com>
>Date: Wed, 25 Jul 2012 04:45:41 -0400
>To: <us...@hive.apache.org>
>ReplyTo: user@hive.apache.org
>Subject: HBASE and HIVE Integration
>
>I am facing issue while executing HIVE queries on HBASE-HIVE integration.
>I followed the wiki hbase-hive integration https://cwiki.apache.org/Hive/hbaseintegration.html
>
>I have already passed all the required jars for auxpath in hive-site.xml file.
>I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
>hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE foo=98;
>Total MapReduce jobs = 1
>Launching Job 1 out of 1
>Number of reduce tasks is set to 0 since there's no reduce operator
>Starting Job = job_201207250246_0005, Tracking URL = http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
>Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=0.0.0.0:8021 -kill job_201207250246_0005
>2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
>2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
>2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
>Ended Job = job_201207250246_0005 with errors
>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>hive>
>
>
Re: HBASE and HIVE Integration
Posted by vijay shinde <vi...@gmail.com>.
Hi Bejoy,
Thanks for quick reply. Here are some additional details
Cloudera Version - CDH3U4
*hive-site.xml*
**
*<property>
<name>hive.aux.jars.path</name>
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u2.jar,file:///usr/lib/hive/lib/hbase-0.90.4-cdh3u2.jar,file:///usr/lib/hive/lib/zookeeper-3.3.1.jar,file:///usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u2.jar</value>
</property>
*
*Execution Log*
1. *start zookeeper*
[root@localhost zookeeper]# ./bin/zkServer.sh start
2. *start hbase*
3. *start hive. I am setting hive jars in hive-site.xml*
./bin/hive -hiveconf hbase.master=127.0.1.1:60010
4. *Create new HBase table which is to be managed by Hive*
**
CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");
5. *Create a logical table pokes in Hive*
CREATE TABLE pokes (foo INT, bar STRING);
6. *HIve error while inserting the data from Hive Poke table to HBASE table*
*hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
foo=98;*
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
*FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
*
Let me know if you need any additonal information.
Thanks,
Vijay
On Wed, Jul 25, 2012 at 5:30 AM, Bejoy KS <be...@yahoo.com> wrote:
> **
> Hi Vijay
>
> Can you share more details like
>
> The CDH Version/Hive version you are using
>
> Steps you followed for hive hbase integration with the values you set
>
> The DDL used for hive hbase integration
>
> The actual error from failed map reduce task
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> ------------------------------
> *From: *vijay shinde <vi...@gmail.com>
> *Date: *Wed, 25 Jul 2012 04:45:41 -0400
> *To: *<us...@hive.apache.org>
> *ReplyTo: *user@hive.apache.org
> *Subject: *HBASE and HIVE Integration
>
> I am facing issue while executing HIVE queries on HBASE-HIVE integration.
> I followed the wiki hbase-hive integration
> https://cwiki.apache.org/Hive/hbaseintegration.html
>
> I have already passed all the required jars for auxpath in hive-site.xml
> file.
> I am using Cloudera CDH demo VM.. Any help would be highly appreciated
>
> hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
> foo=98;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201207250246_0005, Tracking URL =
> http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
> Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
> 0.0.0.0:8021 -kill job_201207250246_0005
> 2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
> 2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
> Ended Job = job_201207250246_0005 with errors
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> hive>
>
>
Re: HBASE and HIVE Integration
Posted by Bejoy KS <be...@yahoo.com>.
Hi Vijay
Can you share more details like
The CDH Version/Hive version you are using
Steps you followed for hive hbase integration with the values you set
The DDL used for hive hbase integration
The actual error from failed map reduce task
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-----Original Message-----
From: vijay shinde <vi...@gmail.com>
Date: Wed, 25 Jul 2012 04:45:41
To: <us...@hive.apache.org>
Reply-To: user@hive.apache.org
Subject: HBASE and HIVE Integration
I am facing issue while executing HIVE queries on HBASE-HIVE integration.
I followed the wiki hbase-hive integration
https://cwiki.apache.org/Hive/hbaseintegration.html
I have already passed all the required jars for auxpath in hive-site.xml
file.
I am using Cloudera CDH demo VM.. Any help would be highly appreciated
hive> INSERT OVERWRITE TABLE hive_hbasetable_k SELECT * FROM pokes WHERE
foo=98;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201207250246_0005, Tracking URL =
http://0.0.0.0:50030/jobdetails.jsp?jobid=job_201207250246_0005
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=
0.0.0.0:8021 -kill job_201207250246_0005
2012-07-25 04:26:00,198 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:00,767 Stage-0 map = 0%, reduce = 0%
2012-07-25 04:27:08,844 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201207250246_0005 with errors
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
hive>