You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Jilani Shaik <ji...@gmail.com> on 2016/06/11 07:57:03 UTC
hbase bulk load with map reduce error
Hi,
I am trying to do hbase table bulk load from data file using map reduce. in
the process I am getting the Null pointer exception during the shuffle and
sort. I have both mapper and reducer in the flow. where as mapper is ending
with success and reducer not being invoked even setup.
It is showing the below error in terminal where the job executed.
Error: java.lang.NullPointerException
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
at com.google.common.base.Splitter.split(Splitter.java:375)
at
org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser.<init>(ImportTsv.java:151)
at
org.apache.hadoop.hbase.mapreduce.TextSortReducer.setup(TextSortReducer.java:105)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Please let me know what could be the issue.
Thanks,
Jilani
Re: hbase bulk load with map reduce error
Posted by Jilani Shaik <ji...@gmail.com>.
Please let me know what am I missing here.
I am using MapR hadoop. Please find the classpath which is showing all the
jars with versions, mean while I will get the code snippet also. I am using
bulk write approach with mapper and reducer. HBase table created with bulk
load enabled as true.
+ HADOOP_CLASSPATH='/opt/mapr/hbase/hbase-0.98.9/bin/../
conf:/opt/mapr/java/jdk1.7.0_25/lib/tools.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/..:/opt/mapr/lib/zookeeper-3.4.5-mapr-1406.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/activation-1.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/aopalliance-1.0.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/api-asn1-api-1.0.0-M20.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/api-util-1.0.0-M20.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/asm-3.1.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-beanutils-1.7.0.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-beanutils-core-1.8.0.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-cli-1.2.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-codec-1.7.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/commons-collections-3.2.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-compress-1.4.1.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-configuration-1.6.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-daemon-1.0.13.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-digester-1.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-el-1.0.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-httpclient-3.1.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-io-2.4.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-lang-2.6.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/commons-logging-1.1.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/commons-math-2.1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/commons-math3-3.1.1.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/commons-net-3.1.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/findbugs-annotations-1.3.9-1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/guava-12.0.1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/guice-3.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
guice-servlet-3.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/hamcrest-core-1.3.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase
-annotations-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-checkstyle-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-client-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-common-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-common-0.98.9-mapr-1503-tests.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/hbase-examples-0.98.9-mapr-1503.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/hbase-hadoop2-compat-0.98.9-mapr-1503.jar:/opt/mapr/
hbase/hbase-0.98.9/bin/../lib/hbase-hadoop-compat-0.98.9-mapr-1503.jar:/
opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase-it-0.98.9-
mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase-
it-0.98.9-mapr-1503-tests.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/hbase
-prefix-tree-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-protocol-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-rest-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-server-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
hbase-server-0.98.9-mapr-1503-tests.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-shell-0.98.9-mapr-1503.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/hbase-testing-util-0.98.9-mapr-1503.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/hbase-thrift-0.98.9-mapr-1503.jar:/opt/mapr/hbase/
hbase-0.98.9/bin/../lib/high-scale-lib-1.1.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/htrace-core-2.04.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/httpclient-4.2.5.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/httpcore-4.1.3.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jackson-core-asl-1.8.8.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/jackson-jaxrs-1.8.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jackson-mapper-asl-1.8.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jackson-xc-1.8.8.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/jamon-runtime-2.3.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jasper-compiler-5.5.23.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jasper-runtime-5.5.23.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/javax.inject-1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/java-xmlbuilder-0.4.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jaxb-api-2.2.2.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jaxb-impl-2.2.3-1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/jcodings-1.0.8.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/jersey-client-1.9.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jersey-core-1.8.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/jersey-guice-1.9.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/jersey-json-1.8.jar:/opt/mapr/hbase/hbase-
0.98.9/bin/../lib/jersey-server-1.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jets3t-0.9.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jettison-1.3.1.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jetty-6.1.26.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jetty-sslengine-6.1.26.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/jetty-util-6.1.26.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/joni-2.1.2.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/jruby-complete-1.6.8.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/jsch-0.1.42.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jsp-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
jsp-api-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/jsr305-1.3.9.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/junit-4.11.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
leveldbjni-all-1.8.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/libthrift-0.9.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/log4j-1.2.17.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/metrics-core-2.2.0.jar:/opt/mapr/hbase/hbase-0.98.9/
bin/../lib/netty-3.6.6.Final.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/paranamer-2.3.jar:/opt/mapr/hbase/hbase-0.
98.9/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/servlet-api-2.5.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/snappy-java-1.0.4.1.jar:/opt/mapr/hbase/hbase
-0.98.9/bin/../lib/xmlenc-0.52.jar:/opt/mapr/hbase/hbase-0.98.9/bin/..
/lib/xz-1.0.jar:/opt/mapr/hbase/hbase-0.98.9/bin/../lib/
zookeeper-3.4.5-mapr-1406.jar:/opt/mapr/lib/libprotodefs-4.
0.2-mapr.jar:/opt/mapr/lib/baseutils-4.0.2-mapr.jar:/opt/
mapr/lib/JPam-1.1.jar:/opt/mapr/hadoop/hadoop-2.5.1/etc/
hadoop:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/
common/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/
common/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/
hdfs:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/hdfs/lib/*:
/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/hdfs/*:/opt/mapr/
hadoop/hadoop-2.5.1/share/hadoop/yarn/lib/*:/opt/mapr/
hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/
hadoop/mapreduce/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/
share/hadoop/mapreduce/*
Thanks,
Jilani
On Sat, Jun 11, 2016 at 6:39 AM, Ted Yu <yu...@gmail.com> wrote:
> Which version of hbase / Hadoop are you using ?
> (So that line number matching can be more accurate)
>
> It would be good if you can show your code snippet.
>
> Thanks
>
> On Jun 11, 2016, at 12:57 AM, Jilani Shaik <ji...@gmail.com> wrote:
>
> Hi,
>
> I am trying to do hbase table bulk load from data file using map reduce.
> in the process I am getting the Null pointer exception during the shuffle
> and sort. I have both mapper and reducer in the flow. where as mapper is
> ending with success and reducer not being invoked even setup.
>
> It is showing the below error in terminal where the job executed.
>
> Error: java.lang.NullPointerException
>
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
>
> at com.google.common.base.Splitter.split(Splitter.java:375)
>
> at
> org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser.<init>(ImportTsv.java:151)
>
> at
> org.apache.hadoop.hbase.mapreduce.TextSortReducer.setup(TextSortReducer.java:105)
>
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168)
>
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
>
>
> Please let me know what could be the issue.
>
>
> Thanks,
> Jilani
>
>
Re: hbase bulk load with map reduce error
Posted by Ted Yu <yu...@gmail.com>.
Which version of hbase / Hadoop are you using ?
(So that line number matching can be more accurate)
It would be good if you can show your code snippet.
Thanks
> On Jun 11, 2016, at 12:57 AM, Jilani Shaik <ji...@gmail.com> wrote:
>
> Hi,
>
> I am trying to do hbase table bulk load from data file using map reduce. in the process I am getting the Null pointer exception during the shuffle and sort. I have both mapper and reducer in the flow. where as mapper is ending with success and reducer not being invoked even setup.
>
> It is showing the below error in terminal where the job executed.
>
> Error: java.lang.NullPointerException
> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
> at com.google.common.base.Splitter.split(Splitter.java:375)
> at org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser.<init>(ImportTsv.java:151)
> at org.apache.hadoop.hbase.mapreduce.TextSortReducer.setup(TextSortReducer.java:105)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
>
> Please let me know what could be the issue.
>
>
> Thanks,
> Jilani