You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Botelho, Andrew" <An...@emc.com> on 2013/08/23 22:41:40 UTC
Getting HBaseStorage() to work in Pig
I am trying to use the function HBaseStorage() in my Pig code in order to load an HBase table into Pig.
When I run my code, I get this error:
ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/WritableByteArrayComparable
I believe the PIG_CLASSPATH needs to be extended to include the classpath for loading HBase, but I am not sure how to do this. I've tried several export commands in the unix shell to change the PIG_CLASSPATH, but nothing seems to be working.
Any advice would be much appreciated.
Thanks,
Andrew Botelho
Re: Getting HBaseStorage() to work in Pig
Posted by Shahab Yunus <sh...@gmail.com>.
You are here running multiple UNIX commands and the end result or the end
command is to run hbase-<YOUR VERSION>.jar using hadoop's *jar* command. So
basically you add HBase jars to the classpath of your Hadoop environment
and then execute hbase tools using hadoop. If you get the message as
specified in the doc then it means that you have successfully added Hbase
libs to your Hadoop setup.
First you are setting your HADOOP_CLASSPATH by assigning it the classpath
of your HBase libs by executing the following command.
*`${HBASE_HOME}/bin/hbase classpath` *
Any command within back ticks in unix shell, is executed and its output is
being assigned to your HADOOP_CLASSPATH. Note that the 'classpath' argument
or command to the /bin/hbase executable returns the classpath of your HBase
setup.
Then
*${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar*
you are running the hadoop executable with the 'jar' command (
http://hadoop.apache.org/docs/r1.0.4/commands_manual.html#jar) The 'jar'
command takes a jar as argument and here you are passing it your Hbase main
lib.jar to run. Note that there are 2 parts here. The main command is:
*${HADOOP_HOME}/bin/hadoop jar *
*
*
and the argument, the jar file is:
*${HBASE_HOME}/hbase-VERSION.jar*
${STRING} is the convention to refer to properties' holders/variables.
If you get the message from the docs, it means you have set your Hbase jars
to your Hadoop classpath correcty.
Regards,
Shahab
On Fri, Aug 23, 2013 at 4:57 PM, Botelho, Andrew <An...@emc.com>wrote:
> Could you explain what is going on here:****
>
> HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`
> ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar****
>
> ** **
>
> I’m not a Unix expert by any means.****
>
> How can I use this to enable HBaseStorage() in Pig?****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Andrew****
>
> ** **
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, August 23, 2013 4:50 PM
> *To:* common-user@hadoop.apache.org
> *Subject:* Re: Getting HBaseStorage() to work in Pig****
>
> ** **
>
> Please look at the example in 15.1.1 under
> http://hbase.apache.org/book.html#tools****
>
> ** **
>
> On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>
> wrote:****
>
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ****
>
> ** **
>
Re: Getting HBaseStorage() to work in Pig
Posted by Shahab Yunus <sh...@gmail.com>.
You are here running multiple UNIX commands and the end result or the end
command is to run hbase-<YOUR VERSION>.jar using hadoop's *jar* command. So
basically you add HBase jars to the classpath of your Hadoop environment
and then execute hbase tools using hadoop. If you get the message as
specified in the doc then it means that you have successfully added Hbase
libs to your Hadoop setup.
First you are setting your HADOOP_CLASSPATH by assigning it the classpath
of your HBase libs by executing the following command.
*`${HBASE_HOME}/bin/hbase classpath` *
Any command within back ticks in unix shell, is executed and its output is
being assigned to your HADOOP_CLASSPATH. Note that the 'classpath' argument
or command to the /bin/hbase executable returns the classpath of your HBase
setup.
Then
*${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar*
you are running the hadoop executable with the 'jar' command (
http://hadoop.apache.org/docs/r1.0.4/commands_manual.html#jar) The 'jar'
command takes a jar as argument and here you are passing it your Hbase main
lib.jar to run. Note that there are 2 parts here. The main command is:
*${HADOOP_HOME}/bin/hadoop jar *
*
*
and the argument, the jar file is:
*${HBASE_HOME}/hbase-VERSION.jar*
${STRING} is the convention to refer to properties' holders/variables.
If you get the message from the docs, it means you have set your Hbase jars
to your Hadoop classpath correcty.
Regards,
Shahab
On Fri, Aug 23, 2013 at 4:57 PM, Botelho, Andrew <An...@emc.com>wrote:
> Could you explain what is going on here:****
>
> HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`
> ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar****
>
> ** **
>
> I’m not a Unix expert by any means.****
>
> How can I use this to enable HBaseStorage() in Pig?****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Andrew****
>
> ** **
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, August 23, 2013 4:50 PM
> *To:* common-user@hadoop.apache.org
> *Subject:* Re: Getting HBaseStorage() to work in Pig****
>
> ** **
>
> Please look at the example in 15.1.1 under
> http://hbase.apache.org/book.html#tools****
>
> ** **
>
> On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>
> wrote:****
>
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ****
>
> ** **
>
Re: Getting HBaseStorage() to work in Pig
Posted by Shahab Yunus <sh...@gmail.com>.
You are here running multiple UNIX commands and the end result or the end
command is to run hbase-<YOUR VERSION>.jar using hadoop's *jar* command. So
basically you add HBase jars to the classpath of your Hadoop environment
and then execute hbase tools using hadoop. If you get the message as
specified in the doc then it means that you have successfully added Hbase
libs to your Hadoop setup.
First you are setting your HADOOP_CLASSPATH by assigning it the classpath
of your HBase libs by executing the following command.
*`${HBASE_HOME}/bin/hbase classpath` *
Any command within back ticks in unix shell, is executed and its output is
being assigned to your HADOOP_CLASSPATH. Note that the 'classpath' argument
or command to the /bin/hbase executable returns the classpath of your HBase
setup.
Then
*${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar*
you are running the hadoop executable with the 'jar' command (
http://hadoop.apache.org/docs/r1.0.4/commands_manual.html#jar) The 'jar'
command takes a jar as argument and here you are passing it your Hbase main
lib.jar to run. Note that there are 2 parts here. The main command is:
*${HADOOP_HOME}/bin/hadoop jar *
*
*
and the argument, the jar file is:
*${HBASE_HOME}/hbase-VERSION.jar*
${STRING} is the convention to refer to properties' holders/variables.
If you get the message from the docs, it means you have set your Hbase jars
to your Hadoop classpath correcty.
Regards,
Shahab
On Fri, Aug 23, 2013 at 4:57 PM, Botelho, Andrew <An...@emc.com>wrote:
> Could you explain what is going on here:****
>
> HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`
> ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar****
>
> ** **
>
> I’m not a Unix expert by any means.****
>
> How can I use this to enable HBaseStorage() in Pig?****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Andrew****
>
> ** **
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, August 23, 2013 4:50 PM
> *To:* common-user@hadoop.apache.org
> *Subject:* Re: Getting HBaseStorage() to work in Pig****
>
> ** **
>
> Please look at the example in 15.1.1 under
> http://hbase.apache.org/book.html#tools****
>
> ** **
>
> On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>
> wrote:****
>
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ****
>
> ** **
>
Re: Getting HBaseStorage() to work in Pig
Posted by Shahab Yunus <sh...@gmail.com>.
You are here running multiple UNIX commands and the end result or the end
command is to run hbase-<YOUR VERSION>.jar using hadoop's *jar* command. So
basically you add HBase jars to the classpath of your Hadoop environment
and then execute hbase tools using hadoop. If you get the message as
specified in the doc then it means that you have successfully added Hbase
libs to your Hadoop setup.
First you are setting your HADOOP_CLASSPATH by assigning it the classpath
of your HBase libs by executing the following command.
*`${HBASE_HOME}/bin/hbase classpath` *
Any command within back ticks in unix shell, is executed and its output is
being assigned to your HADOOP_CLASSPATH. Note that the 'classpath' argument
or command to the /bin/hbase executable returns the classpath of your HBase
setup.
Then
*${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar*
you are running the hadoop executable with the 'jar' command (
http://hadoop.apache.org/docs/r1.0.4/commands_manual.html#jar) The 'jar'
command takes a jar as argument and here you are passing it your Hbase main
lib.jar to run. Note that there are 2 parts here. The main command is:
*${HADOOP_HOME}/bin/hadoop jar *
*
*
and the argument, the jar file is:
*${HBASE_HOME}/hbase-VERSION.jar*
${STRING} is the convention to refer to properties' holders/variables.
If you get the message from the docs, it means you have set your Hbase jars
to your Hadoop classpath correcty.
Regards,
Shahab
On Fri, Aug 23, 2013 at 4:57 PM, Botelho, Andrew <An...@emc.com>wrote:
> Could you explain what is going on here:****
>
> HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`
> ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar****
>
> ** **
>
> I’m not a Unix expert by any means.****
>
> How can I use this to enable HBaseStorage() in Pig?****
>
> ** **
>
> Thanks,****
>
> ** **
>
> Andrew****
>
> ** **
>
> *From:* Ted Yu [mailto:yuzhihong@gmail.com]
> *Sent:* Friday, August 23, 2013 4:50 PM
> *To:* common-user@hadoop.apache.org
> *Subject:* Re: Getting HBaseStorage() to work in Pig****
>
> ** **
>
> Please look at the example in 15.1.1 under
> http://hbase.apache.org/book.html#tools****
>
> ** **
>
> On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>
> wrote:****
>
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ****
>
> ** **
>
RE: Getting HBaseStorage() to work in Pig
Posted by "Botelho, Andrew" <An...@emc.com>.
Could you explain what is going on here:
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar
I'm not a Unix expert by any means.
How can I use this to enable HBaseStorage() in Pig?
Thanks,
Andrew
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, August 23, 2013 4:50 PM
To: common-user@hadoop.apache.org
Subject: Re: Getting HBaseStorage() to work in Pig
Please look at the example in 15.1.1 under http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>> wrote:
I am trying to use the function HBaseStorage() in my Pig code in order to load an HBase table into Pig.
When I run my code, I get this error:
ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/WritableByteArrayComparable
I believe the PIG_CLASSPATH needs to be extended to include the classpath for loading HBase, but I am not sure how to do this. I've tried several export commands in the unix shell to change the PIG_CLASSPATH, but nothing seems to be working.
Any advice would be much appreciated.
Thanks,
Andrew Botelho
RE: Getting HBaseStorage() to work in Pig
Posted by "Botelho, Andrew" <An...@emc.com>.
Could you explain what is going on here:
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar
I'm not a Unix expert by any means.
How can I use this to enable HBaseStorage() in Pig?
Thanks,
Andrew
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, August 23, 2013 4:50 PM
To: common-user@hadoop.apache.org
Subject: Re: Getting HBaseStorage() to work in Pig
Please look at the example in 15.1.1 under http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>> wrote:
I am trying to use the function HBaseStorage() in my Pig code in order to load an HBase table into Pig.
When I run my code, I get this error:
ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/WritableByteArrayComparable
I believe the PIG_CLASSPATH needs to be extended to include the classpath for loading HBase, but I am not sure how to do this. I've tried several export commands in the unix shell to change the PIG_CLASSPATH, but nothing seems to be working.
Any advice would be much appreciated.
Thanks,
Andrew Botelho
RE: Getting HBaseStorage() to work in Pig
Posted by "Botelho, Andrew" <An...@emc.com>.
Could you explain what is going on here:
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar
I'm not a Unix expert by any means.
How can I use this to enable HBaseStorage() in Pig?
Thanks,
Andrew
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, August 23, 2013 4:50 PM
To: common-user@hadoop.apache.org
Subject: Re: Getting HBaseStorage() to work in Pig
Please look at the example in 15.1.1 under http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>> wrote:
I am trying to use the function HBaseStorage() in my Pig code in order to load an HBase table into Pig.
When I run my code, I get this error:
ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/WritableByteArrayComparable
I believe the PIG_CLASSPATH needs to be extended to include the classpath for loading HBase, but I am not sure how to do this. I've tried several export commands in the unix shell to change the PIG_CLASSPATH, but nothing seems to be working.
Any advice would be much appreciated.
Thanks,
Andrew Botelho
RE: Getting HBaseStorage() to work in Pig
Posted by "Botelho, Andrew" <An...@emc.com>.
Could you explain what is going on here:
HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-VERSION.jar
I'm not a Unix expert by any means.
How can I use this to enable HBaseStorage() in Pig?
Thanks,
Andrew
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, August 23, 2013 4:50 PM
To: common-user@hadoop.apache.org
Subject: Re: Getting HBaseStorage() to work in Pig
Please look at the example in 15.1.1 under http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>> wrote:
I am trying to use the function HBaseStorage() in my Pig code in order to load an HBase table into Pig.
When I run my code, I get this error:
ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/WritableByteArrayComparable
I believe the PIG_CLASSPATH needs to be extended to include the classpath for loading HBase, but I am not sure how to do this. I've tried several export commands in the unix shell to change the PIG_CLASSPATH, but nothing seems to be working.
Any advice would be much appreciated.
Thanks,
Andrew Botelho
Re: Getting HBaseStorage() to work in Pig
Posted by Ted Yu <yu...@gmail.com>.
Please look at the example in 15.1.1 under
http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>wrote:
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ** **
>
Re: Getting HBaseStorage() to work in Pig
Posted by Ted Yu <yu...@gmail.com>.
Please look at the example in 15.1.1 under
http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>wrote:
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ** **
>
Re: Getting HBaseStorage() to work in Pig
Posted by Ted Yu <yu...@gmail.com>.
Please look at the example in 15.1.1 under
http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>wrote:
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ** **
>
Re: Getting HBaseStorage() to work in Pig
Posted by Ted Yu <yu...@gmail.com>.
Please look at the example in 15.1.1 under
http://hbase.apache.org/book.html#tools
On Fri, Aug 23, 2013 at 1:41 PM, Botelho, Andrew <An...@emc.com>wrote:
> I am trying to use the function HBaseStorage() in my Pig code in order to
> load an HBase table into Pig.****
>
> ****
>
> When I run my code, I get this error:****
>
> ****
>
> ERROR 2998: Unhandled internal error.
> org/apache/hadoop/hbase/filter/WritableByteArrayComparable****
>
> ****
>
> ****
>
> I believe the PIG_CLASSPATH needs to be extended to include the classpath
> for loading HBase, but I am not sure how to do this. I've tried several
> export commands in the unix shell to change the PIG_CLASSPATH, but nothing
> seems to be working.****
>
> ****
>
> Any advice would be much appreciated.****
>
> ****
>
> Thanks,****
>
> ****
>
> Andrew Botelho****
>
> ** **
>