Unable to run pig script with hcatalog

["Khera, Deepika" <dk...@paypal.com>]

Hi,

I am trying to run a simple script with hcatalog and am running into issues.

Here is the pig script


test.pig

deepika_data = load 'hivetest' using org.apache.hcatalog.pig.HCatLoader();

dump deepika_data;



I run it using this on command line -

pig -Dpig.additional.jars=$PIG_CLASSPATH test.pig


I get the following exception when I run the command:



HBASE_HOME isn't set: $HBASE_HOME defaulting to /usr/lib/hbase

2013-05-22 13:57:33,528 [main] INFO  org.apache.pig.Main - Apache Pig version 0.10.0-cdh3u4a (rexported) compiled Sep 04 2012, 14:03:46

2013-05-22 13:57:33,529 [main] INFO  org.apache.pig.Main - Logging error messages to: /x/home/dkhera/scripts/pig/pig_1369256253526.log

2013-05-22 13:57:33,866 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://my-namenode:8020/

2013-05-22 13:57:33,956 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: my-jobtracker:8021

2013-05-22 13:57:34,874 [main] ERROR org.apache.hadoop.hive.conf.HiveConf - Found both hive.metastore.uris and javax.jdo.option.ConnectionURL Recommended to have exactly one of those config keyin configuration

2013-05-22 13:57:34,979 [main] INFO  hive.metastore - Trying to connect to metastore with URI thrift://my-hive-metastore:3116

2013-05-22 13:57:34,987 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. java.lang.IncompatibleClassChangeError: class com.facebook.fb303.FacebookService$Client has interface org.apache.thrift.TServiceClient as super class



PIG_CLASSPATH looks like:


HCAT_HOME/share/hcatalog/hcatalog-pig-adapter-0.5.0-incubating.jar:\

$HCAT_HOME/share/hcatalog/storage-handlers/hbase/lib/hbase-storage-handler-0.1.0.jar:\

$HIVE_HOME/lib/hive-metastore-0.9.0-cdh3u4b-SNAPSHOT.jar:\

$HIVE_HOME/lib/hive-exec-0.9.0-cdh3u4b-SNAPSHOT.jar:\

$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:\

$HIVE_HOME/conf:$HADOOP_HOME/conf:\

$HIVE_HOME/lib/libfb303-0.7.0.jar:\

$HIVE_HOME/lib/slf4j-api-1.6.1.jar


export PIG_OPTS=-Dhive.metastore.uris=thrift://my-hive-metastore:3116


This is as described here

http://hive.apache.org/docs/hcat_r0.5.0/loadstore.html


When I use the debug flag on pig, I get


HADOOP_CLASSPATH: /usr/lib/hbase/conf:/usr/lib/pig/bin/../conf:/usr/java/jdk1.6.0_27/lib/tools.jar:/usr/lib/hbase/hbase.jar:/usr/lib/hbase/lib/*:/usr/lib/hadoop/conf:/usr/lib/hbase/conf:/usr/lib/pig/bin/../conf:/x/home/dkhera/hcat-client/hcatalog-incubating/share/hcatalog/hcatalog-core-0.5.0-incubating.jar:/x/home/dkhera/hcat-client/hcatalog-incubating/share/hcatalog/hcatalog-pig-adapter-0.5.0-incubating.jar:/x/home/dkhera/hcat-client/hcatalog-incubating/share/hcatalog/storage-handlers/hbase/lib/hbase-storage-handler-0.1.0.jar:/usr/lib/hive/lib/hive-metastore-0.9.0-cdh3u4b-SNAPSHOT.jar:/usr/lib/hive/lib/hive-exec-0.9.0-cdh3u4b-SNAPSHOT.jar:/usr/lib/hive/lib/jdo2-api-2.3-ec.jar:/usr/lib/hive/conf:/usr/lib/hadoop/conf:/usr/lib/hive/lib/libfb303-0.7.0.jar:/usr/lib/hive/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop/conf:/usr/lib/pig/bin/../lib/automaton.jar:/usr/lib/pig/bin/../lib/avro-1.5.4.jar:/usr/lib/pig/bin/../lib/jackson-core-asl-1.7.3.jar:/usr/lib/pig/bin/../lib/jackson-mapper-asl-1.7.3.jar:/usr/lib/pig/bin/../lib/json-simple-1.1.jar:/usr/lib/pig/bin/../lib/jython-2.5.0.jar:/usr/lib/pig/bin/../lib/snappy-java-1.0.3.2.jar:/usr/lib/pig/bin/../pig-0.10.0-cdh3u4a-withouthadoop.jar:

HADOOP_OPTS: -Xmx1000m -Dhive.metastore.uris=thrift://horton-mysql.vip.lvs.paypal.com:3116 -Djava.library.path=/usr/lib/hadoop/lib/native/Linux-amd64-64 -Djava.io.tmpdir=/x/cdh/pig/tmp -Dpig.log.dir=/usr/lib/pig/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/usr/lib/pig/bin/..

/usr/bin/hadoop jar /usr/lib/pig/bin/../pig-0.10.0-cdh3u4a-withouthadoop.jar test.pig


Note that libthrift-0.7.0.jar is in hbase lib (already in class path), so I did not explicitly add it in pig class path.



Does anyone know what is causing this. It certainly seems like a version conflict of thrift jars, but I am unable to figure it out yet.


Thanks,

Deepika