You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Hao Ji (JIRA)" <ji...@apache.org> on 2015/08/11 08:14:45 UTC
[jira] [Created] (HBASE-14204) HBase Client API not working with
pig 0.14 and 0.15
Hao Ji created HBASE-14204:
------------------------------
Summary: HBase Client API not working with pig 0.14 and 0.15
Key: HBASE-14204
URL: https://issues.apache.org/jira/browse/HBASE-14204
Project: HBase
Issue Type: Bug
Components: API
Affects Versions: 1.0.1.1
Environment: CentOS 6
Hadoop 2.4.1
HBase 1.0.1.1
Pig 0.14.0 or 0.15.0
Reporter: Hao Ji
After upgrade hbase-0.98.3-hadoop2 to hbase-1.0.11, everything works fine, HMaster and RegionServers all started OK, hbase shell works OK, table scan works OK. Except pig script failed to store data to Hbase using org.apache.pig.backend.hadoop.hbase.HBaseStorage.
Detailed exception from pig.
{quote}
Pig Stack Trace
---------------
ERROR 1200: Pig script failed to parse:
<line 13, column 0> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[cf:*]'
Failed to parse: Pig script failed to parse:
<line 13, column 0> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[cf:*]'
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1707)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1680)
at org.apache.pig.PigServer.registerQuery(PigServer.java:623)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1082)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:505)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:565)
at org.apache.pig.Main.main(Main.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
<line 13, column 0> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[cf:*]'
at org.apache.pig.parser.LogicalPlanBuilder.buildStoreOp(LogicalPlanBuilder.java:1009)
at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7806)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
... 15 more
Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[cf:*]'
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772)
at org.apache.pig.parser.LogicalPlanBuilder.buildStoreOp(LogicalPlanBuilder.java:988)
... 21 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740)
... 22 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan(HBaseStorage.java:427)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:368)
at org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:239)
... 27 more
================================================================================
{quote}
Here is the classpath for running pig, as you can see, I am using the hbase-client.1.0.1.1 version.
{quote}
[hadoop@hadoop-master-1 ~]$ pig -useHCatalog
ls: cannot access /opt/apache-hive-0.14.0-bin/lib/slf4j-api-*.jar: No such file or directory
ls: cannot access /opt/apache-hive-0.14.0-bin/hcatalog/lib/*hbase-storage-handler-*.jar: No such file or directory
Find hadoop at /opt/hadoop-2.4.1/bin/hadoop
dry run:
HADOOP_CLASSPATH: /opt/hbase-1.0.1.1/conf:/opt/pig-0.15.0/conf:/usr/java/latest/lib/tools.jar:/opt/apache-hive-0.14.0-bin/lib/hive-metastore-0.14.0.jar:/opt/apache-hive-0.14.0-bin/lib/libthrift-0.9.0.jar:/opt/apache-hive-0.14.0-bin/lib/hive-exec-0.14.0.jar:/opt/apache-hive-0.14.0-bin/lib/libfb303-0.9.0.jar:/opt/apache-hive-0.14.0-bin/lib/jdo-api-3.0.1.jar::/opt/apache-hive-0.14.0-bin/lib/hive-hbase-handler-0.14.0.jar:/opt/apache-hive-0.14.0-bin/hcatalog/share/hcatalog/hive-hcatalog-core-0.14.0.jar::/opt/apache-hive-0.14.0-bin/hcatalog/share/hcatalog/hive-hcatalog-pig-adapter-0.14.0.jar:/opt/apache-hive-0.14.0-bin/conf:/opt/hadoop-2.4.1/etc/hadoop/:/opt/pig-0.15.0/lib/accumulo-core-1.5.0.jar:/opt/pig-0.15.0/lib/accumulo-fate-1.5.0.jar:/opt/pig-0.15.0/lib/accumulo-server-1.5.0.jar:/opt/pig-0.15.0/lib/accumulo-start-1.5.0.jar:/opt/pig-0.15.0/lib/accumulo-trace-1.5.0.jar:/opt/pig-0.15.0/lib/antlr-runtime-3.4.jar:/opt/pig-0.15.0/lib/asm-3.3.1.jar:/opt/pig-0.15.0/lib/automaton-1.11-8.jar:/opt/pig-0.15.0/lib/avro-1.7.5.jar:/opt/pig-0.15.0/lib/avro-tools-1.7.5-nodeps.jar:/opt/pig-0.15.0/lib/groovy-all-1.8.6.jar:/opt/pig-0.15.0/lib/guava-11.0.jar:/opt/pig-0.15.0/lib/hive-common-0.14.0.jar:/opt/pig-0.15.0/lib/hive-exec-0.14.0-core.jar:/opt/pig-0.15.0/lib/hive-serde-0.14.0.jar:/opt/pig-0.15.0/lib/hive-shims-common-0.14.0.jar:/opt/pig-0.15.0/lib/hive-shims-common-secure-0.14.0.jar:/opt/pig-0.15.0/lib/jackson-core-asl-1.8.8.jar:/opt/pig-0.15.0/lib/jackson-mapper-asl-1.8.8.jar:/opt/pig-0.15.0/lib/jansi-1.9.jar:/opt/pig-0.15.0/lib/jline-1.0.jar:/opt/pig-0.15.0/lib/joda-time-2.5.jar:/opt/pig-0.15.0/lib/jruby-complete-1.6.7.jar:/opt/pig-0.15.0/lib/js-1.7R2.jar:/opt/pig-0.15.0/lib/json-simple-1.1.jar:/opt/pig-0.15.0/lib/jython-standalone-2.5.3.jar:/opt/pig-0.15.0/lib/kryo-2.22.jar:/opt/pig-0.15.0/lib/piggybank.jar:/opt/pig-0.15.0/lib/protobuf-java-2.5.0.jar:/opt/pig-0.15.0/lib/snappy-java-1.1.0.1.jar:/opt/pig-0.15.0/lib/ST4-4.0.4.jar:/opt/pig-0.15.0/lib/trevni-avro-1.7.5.jar:/opt/pig-0.15.0/lib/trevni-core-1.7.5.jar:/opt/pig-0.15.0/lib/zookeeper-3.4.5.jar:/opt/hbase-1.0.1.1/lib/activation-1.1.jar:/opt/hbase-1.0.1.1/lib/aopalliance-1.0.jar:/opt/hbase-1.0.1.1/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase-1.0.1.1/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase-1.0.1.1/lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase-1.0.1.1/lib/api-util-1.0.0-M20.jar:/opt/hbase-1.0.1.1/lib/asm-3.1.jar:/opt/hbase-1.0.1.1/lib/avro-1.7.4.jar:/opt/hbase-1.0.1.1/lib/commons-beanutils-1.7.0.jar:/opt/hbase-1.0.1.1/lib/commons-beanutils-core-1.8.0.jar:/opt/hbase-1.0.1.1/lib/commons-cli-1.2.jar:/opt/hbase-1.0.1.1/lib/commons-codec-1.9.jar:/opt/hbase-1.0.1.1/lib/commons-collections-3.2.1.jar:/opt/hbase-1.0.1.1/lib/commons-compress-1.4.1.jar:/opt/hbase-1.0.1.1/lib/commons-configuration-1.6.jar:/opt/hbase-1.0.1.1/lib/commons-daemon-1.0.13.jar:/opt/hbase-1.0.1.1/lib/commons-digester-1.8.jar:/opt/hbase-1.0.1.1/lib/commons-el-1.0.jar:/opt/hbase-1.0.1.1/lib/commons-httpclient-3.1.jar:/opt/hbase-1.0.1.1/lib/commons-io-2.4.jar:/opt/hbase-1.0.1.1/lib/commons-lang-2.6.jar:/opt/hbase-1.0.1.1/lib/commons-logging-1.2.jar:/opt/hbase-1.0.1.1/lib/commons-math-2.2.jar:/opt/hbase-1.0.1.1/lib/commons-math3-3.1.1.jar:/opt/hbase-1.0.1.1/lib/commons-net-3.1.jar:/opt/hbase-1.0.1.1/lib/disruptor-3.3.0.jar:/opt/hbase-1.0.1.1/lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase-1.0.1.1/lib/guava-12.0.1.jar:/opt/hbase-1.0.1.1/lib/guice-3.0.jar:/opt/hbase-1.0.1.1/lib/guice-servlet-3.0.jar:/opt/hbase-1.0.1.1/lib/hadoop-annotations-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-auth-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-client-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-common-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-hdfs-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-mapreduce-client-app-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-mapreduce-client-common-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-mapreduce-client-core-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-yarn-api-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-yarn-client-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-yarn-common-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-yarn-server-common-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hadoop-yarn-server-nodemanager-2.5.1.jar:/opt/hbase-1.0.1.1/lib/hbase-annotations-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-annotations-1.0.1.1-tests.jar:/opt/hbase-1.0.1.1/lib/hbase-checkstyle-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-client-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-common-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-common-1.0.1.1-tests.jar:/opt/hbase-1.0.1.1/lib/hbase-examples-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-hadoop2-compat-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-hadoop-compat-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-it-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-it-1.0.1.1-tests.jar:/opt/hbase-1.0.1.1/lib/hbase-prefix-tree-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-protocol-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-rest-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-server-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-server-1.0.1.1-tests.jar:/opt/hbase-1.0.1.1/lib/hbase-shell-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-testing-util-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/hbase-thrift-1.0.1.1.jar:/opt/hbase-1.0.1.1/lib/htrace-core-3.1.0-incubating.jar:/opt/hbase-1.0.1.1/lib/httpclient-4.2.5.jar:/opt/hbase-1.0.1.1/lib/httpcore-4.1.3.jar:/opt/hbase-1.0.1.1/lib/jackson-core-asl-1.8.8.jar:/opt/hbase-1.0.1.1/lib/jackson-jaxrs-1.8.8.jar:/opt/hbase-1.0.1.1/lib/jackson-mapper-asl-1.8.8.jar:/opt/hbase-1.0.1.1/lib/jackson-xc-1.8.8.jar:/opt/hbase-1.0.1.1/lib/jamon-runtime-2.3.1.jar:/opt/hbase-1.0.1.1/lib/jasper-compiler-5.5.23.jar:/opt/hbase-1.0.1.1/lib/jasper-runtime-5.5.23.jar:/opt/hbase-1.0.1.1/lib/javax.inject-1.jar:/opt/hbase-1.0.1.1/lib/java-xmlbuilder-0.4.jar:/opt/hbase-1.0.1.1/lib/jaxb-api-2.2.2.jar:/opt/hbase-1.0.1.1/lib/jaxb-impl-2.2.3-1.jar:/opt/hbase-1.0.1.1/lib/jcodings-1.0.8.jar:/opt/hbase-1.0.1.1/lib/jersey-client-1.9.jar:/opt/hbase-1.0.1.1/lib/jersey-core-1.9.jar:/opt/hbase-1.0.1.1/lib/jersey-guice-1.9.jar:/opt/hbase-1.0.1.1/lib/jersey-json-1.9.jar:/opt/hbase-1.0.1.1/lib/jersey-server-1.9.jar:/opt/hbase-1.0.1.1/lib/jets3t-0.9.0.jar:/opt/hbase-1.0.1.1/lib/jettison-1.3.3.jar:/opt/hbase-1.0.1.1/lib/jetty-6.1.26.jar:/opt/hbase-1.0.1.1/lib/jetty-sslengine-6.1.26.jar:/opt/hbase-1.0.1.1/lib/jetty-util-6.1.26.jar:/opt/hbase-1.0.1.1/lib/joni-2.1.2.jar:/opt/hbase-1.0.1.1/lib/jruby-complete-1.6.8.jar:/opt/hbase-1.0.1.1/lib/jsch-0.1.42.jar:/opt/hbase-1.0.1.1/lib/jsp-2.1-6.1.14.jar:/opt/hbase-1.0.1.1/lib/jsp-api-2.1-6.1.14.jar:/opt/hbase-1.0.1.1/lib/jsr305-1.3.9.jar:/opt/hbase-1.0.1.1/lib/junit-4.11.jar:/opt/hbase-1.0.1.1/lib/leveldbjni-all-1.8.jar:/opt/hbase-1.0.1.1/lib/libthrift-0.9.0.jar:/opt/hbase-1.0.1.1/lib/log4j-1.2.17.jar:/opt/hbase-1.0.1.1/lib/metrics-core-2.2.0.jar:/opt/hbase-1.0.1.1/lib/netty-3.2.4.Final.jar:/opt/hbase-1.0.1.1/lib/netty-all-4.0.23.Final.jar:/opt/hbase-1.0.1.1/lib/paranamer-2.3.jar:/opt/hbase-1.0.1.1/lib/protobuf-java-2.5.0.jar:/opt/hbase-1.0.1.1/lib/servlet-api-2.5-6.1.14.jar:/opt/hbase-1.0.1.1/lib/servlet-api-2.5.jar:/opt/hbase-1.0.1.1/lib/slf4j-api-1.7.7.jar:/opt/hbase-1.0.1.1/lib/slf4j-log4j12-1.7.7.jar:/opt/hbase-1.0.1.1/lib/snappy-java-1.0.4.1.jar:/opt/hbase-1.0.1.1/lib/xmlenc-0.52.jar:/opt/hbase-1.0.1.1/lib/xz-1.0.jar:/opt/hbase-1.0.1.1/lib/zookeeper-3.4.6.jar:/opt/zookeeper-3.4.6/zookeeper-3.4.6.jar:/opt/pig-0.15.0/pig-0.15.0-core-h2.jar:/opt/pig-0.15.0/lib/h2/avro-mapred-1.7.5-hadoop2.jar:/opt/pig-0.15.0/lib/h2/commons-collections4-4.0.jar:/opt/pig-0.15.0/lib/h2/hbase-client-0.98.12-hadoop2.jar:/opt/pig-0.15.0/lib/h2/hbase-common-0.98.12-hadoop2.jar:/opt/pig-0.15.0/lib/h2/hbase-hadoop2-compat-0.98.12-hadoop2.jar:/opt/pig-0.15.0/lib/h2/hbase-hadoop-compat-0.98.12-hadoop2.jar:/opt/pig-0.15.0/lib/h2/hbase-protocol-0.98.12-hadoop2.jar:/opt/pig-0.15.0/lib/h2/hbase-server-0.98.12-hadoop2.jar:/opt/pig-0.15.0/lib/h2/hive-shims-0.23-0.14.0.jar:/opt/pig-0.15.0/lib/h2/tez-api-0.7.0.jar:/opt/pig-0.15.0/lib/h2/tez-common-0.7.0.jar:/opt/pig-0.15.0/lib/h2/tez-dag-0.7.0.jar:/opt/pig-0.15.0/lib/h2/tez-mapreduce-0.7.0.jar:/opt/pig-0.15.0/lib/h2/tez-runtime-internals-0.7.0.jar:/opt/pig-0.15.0/lib/h2/tez-runtime-library-0.7.0.jar:/opt/pig-0.15.0/lib/h2/tez-yarn-timeline-history-with-acls-0.7.0.jar:
HADOOP_OPTS:
HADOOP_CLIENT_OPTS: -Xmx1000m -Dpig.log.dir=/opt/pig-0.15.0/logs -Dpig.log.file=pig.log -Dpig.home.dir=/opt/pig-0.15.0 -Dpig.additional.jars.uris=file:///opt/apache-hive-0.14.0-bin/lib/hive-metastore-0.14.0.jar,file:///opt/apache-hive-0.14.0-bin/lib/libthrift-0.9.0.jar,file:///opt/apache-hive-0.14.0-bin/lib/hive-exec-0.14.0.jar,file:///opt/apache-hive-0.14.0-bin/lib/libfb303-0.9.0.jar,file:///opt/apache-hive-0.14.0-bin/lib/jdo-api-3.0.1.jar,file://,file:///opt/apache-hive-0.14.0-bin/lib/hive-hbase-handler-0.14.0.jar,file:///opt/apache-hive-0.14.0-bin/hcatalog/share/hcatalog/hive-hcatalog-core-0.14.0.jar,file://,file:///opt/apache-hive-0.14.0-bin/hcatalog/share/hcatalog/hive-hcatalog-pig-adapter-0.14.0.jar
/opt/hadoop-2.4.1/bin/hadoop jar /opt/pig-0.15.0/pig-0.15.0-core-h2.jar
{quote}
Also I try to find this class org.apache.hadoop.hbase.client.Scan, there is no other jar contains it, So I am not sure why there is such incompatible. From the source code, setCacheBlocks, this method should exist.
{quote}
[hadoop@hadoop-master-1 opt]$ find . -name "*.jar" -exec grep -l org.apache.hadoop.hbase.client.Scan {} \;
./hbase-1.0.1.1/lib/hbase-client-1.0.1.1.jar
./pig-0.15.0/lib/h2/hbase-client-0.98.12-hadoop2.jar
./pig-0.15.0/lib/h1/hbase-client-0.98.12-hadoop1.jar
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)