You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Roman Shaposhnik (JIRA)" <ji...@apache.org> on 2011/08/16 01:55:27 UTC
[jira] [Created] (PIG-2220) Pig/HBase integration could be improved
Pig/HBase integration could be improved
---------------------------------------
Key: PIG-2220
URL: https://issues.apache.org/jira/browse/PIG-2220
Project: Pig
Issue Type: Improvement
Components: grunt
Affects Versions: 0.9.0, 0.10
Reporter: Roman Shaposhnik
Priority: Minor
For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
register /usr/lib/hbase/lib/guava-r06.jar;
the longer term solution, perhaps, should be to have the code at submit time call hbase's
TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085969#comment-13085969 ]
Roman Shaposhnik commented on PIG-2220:
---------------------------------------
First of all, thanks for pointing out that Pig (0.8.1+) already tries to do that.
However, it doesn't quite seem to work for me. Even the simplest query:
{noformat}
store data into 'rvs-tbl' using org.apache.pig.backend.hadoop.hbase.HBaseStorage('family1:cnt1 family1:cnt2 family2:name', '-caster HBaseBinaryConverter');
{noformat}
generates class not found:
{noformat}
Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.filter.WritableByteArrayComparable
{noformat}
Registering hbase JAR manually fixes the problem. So perhaps, there's a bug in there somewhere. Or may be I'm not
fully following your comment. Please elaborate.
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Dmitriy V. Ryaboy (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy resolved PIG-2220.
------------------------------------
Resolution: Fixed
I just checked the current code and we definitely register the extra jars both for loads and stores.
Resolving as fixed.
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086105#comment-13086105 ]
Roman Shaposhnik commented on PIG-2220:
---------------------------------------
Hm. That's not what I'm seeing with a fresh download of Pig 0.9. Same issue -- unless I register the JAR I get Class not found exception. Any clues as to what could be going wrong?
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086504#comment-13086504 ]
Roman Shaposhnik commented on PIG-2220:
---------------------------------------
You're absolutely correct! Loads work perfectly on both Pig 0.9 and Pig 0.8.1 (provided, of course, that I have required JAR files on the classpath). Stores, however, don't. I can't yet check the trunk. But I can if you would like me to.
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086006#comment-13086006 ]
Dmitriy V. Ryaboy commented on PIG-2220:
----------------------------------------
Aah. Pig 8.1
This was fixed in 0.9: https://issues.apache.org/jira/browse/PIG-1886
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085988#comment-13085988 ]
Roman Shaposhnik commented on PIG-2220:
---------------------------------------
Hi Dmitry,
I thought of that and did modify the class-path during the Pig execution time. Somehow it didn't make any difference. Here's an exact command of how I ran it (notice the last 2 elements of the CP that I added -- the path names there are verbatim from what I was registering):
{noformat}
/usr/java/64/jdk1.6.0_22/bin/java -Xmx1000m -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Dpig.log.dir=/usr/lib/pig/bin/../logs -Dpig.log.file=pig.log -Dpig.home.dir=/usr/lib/pig/bin/.. -Dpig.root.logger=INFO,console,DRFA -classpath /etc/hbase/conf:/usr/lib/pig/bin/../conf:/usr/java/64/jdk1.6.0_22/lib/tools.jar:/usr/lib/pig/bin/../pig-0.8.1-cdh3u1-core.jar:/usr/lib/pig/bin/../build/pig-*-SNAPSHOT.jar:/usr/lib/pig/bin/../lib/ant-contrib-1.0b3.jar:/usr/lib/pig/bin/../lib/automaton.jar:/usr/lib/pig/bin/../build/ivy/lib/Pig/*.jar:/usr/lib/hadoop/hadoop-core-0.20.2-cdh3u1.jar:/usr/lib/hadoop/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop/lib/commons-net-1.4.1.jar:/usr/lib/hadoop/lib/core-3.1.1.jar:/usr/lib/hadoop/lib/hadoop-fairscheduler-0.20.2-cdh3u1.jar:/usr/lib/hadoop/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop/lib/jdiff:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/jetty-servlet-tester-6.1.26.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/jsp-2.1:/usr/lib/hadoop/lib/junit-4.5.jar:/usr/lib/hadoop/lib/kfs-0.2.2.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/lib/hadoop/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop/lib/native:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/conf:/usr/lib/zookeeper/zookeeper-3.3.3-cdh3u1.jar:/usr/lib/hbase/hbase-0.90.3-cdh3u1.jar org.apache.pig.Main
{noformat}
Thanks,
Roman.
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086147#comment-13086147 ]
Dmitriy V. Ryaboy commented on PIG-2220:
----------------------------------------
Something just occurred to me.. can you try *loading*, not storing?
If that works, then the problem is that the bit of code I pasted above only gets called during the loader constructor, but not the storer. That may be fixed in trunk, but if not, I can patch that up, it's pretty straightforward.
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086511#comment-13086511 ]
Dmitriy V. Ryaboy commented on PIG-2220:
----------------------------------------
Whew, mystery solved.
I think we either fixed this already, or a fix for this is in one of the tickets I have in my review queue. Sorry this bit you!
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085983#comment-13085983 ]
Dmitriy V. Ryaboy commented on PIG-2220:
----------------------------------------
Hi Roman,
"Register" accomplishes two things -- gets pig to put the jars on its classpath, and tells it to ship the registered jars. From the HBaseStorage code, we can take care of the second part of this -- automatically tell pig to always ship the jars if the HBaseStorage is used. Naturally, the local java process still needs to be able to find the jars in the first place.
I believe what's happening here is that you need to have the guava/hbase/zk jars on your *classpath* when you start pig. If it can find the jars, everything will work. If it can't, it'll throw the ClassNotFoundException.
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2220) Pig/HBase integration could be
improved
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085459#comment-13085459 ]
Dmitriy V. Ryaboy commented on PIG-2220:
----------------------------------------
Huh?
{code}
// Make sure the HBase, ZooKeeper, and Guava jars get shipped.
TableMapReduceUtil.addDependencyJars(job.getConfiguration(),
org.apache.hadoop.hbase.client.HTable.class,
com.google.common.collect.Lists.class,
org.apache.zookeeper.ZooKeeper.class);
{code}
> Pig/HBase integration could be improved
> ---------------------------------------
>
> Key: PIG-2220
> URL: https://issues.apache.org/jira/browse/PIG-2220
> Project: Pig
> Issue Type: Improvement
> Components: grunt
> Affects Versions: 0.9.0, 0.10
> Reporter: Roman Shaposhnik
> Priority: Minor
>
> For now any Pig/HBase queries would require the following jars to be explicitly added via Pig's register command:
> register /usr/lib/hbase/lib/zookeeper-3.3.3-cdh3u1.jar;
> register /usr/lib/hbase/hbase-0.90.3-cdh3u1.jar;
> register /usr/lib/hbase/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time call hbase's
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it in distributedcache.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira