You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2012/08/22 01:40:38 UTC
[jira] [Created] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Cheolsoo Park created PIG-2885:
----------------------------------
Summary: TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
Key: PIG-2885
URL: https://issues.apache.org/jira/browse/PIG-2885
Project: Pig
Issue Type: Bug
Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
Reporter: Cheolsoo Park
Priority: Minor
Attachments: PIG-2885.patch
I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
- Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
- The HTable constructor in HBase 0.94 changed:
{code}
- HTable table = new HTable(TESTTABLE_2);
+ HTable table = new HTable(conf, TESTTABLE_2);
{code}
- The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
{code}
@@ -541,7 +543,7 @@ public class TestJobSubmission {
// use the estimation
Configuration conf = cluster.getConfiguration();
HBaseTestingUtility util = new HBaseTestingUtility(conf);
- util.startMiniZKCluster();
+ int clientPort = util.startMiniZKCluster().getClientPort();
util.startMiniHBaseCluster(1, 1);
String query = "a = load '/passwd';" +
@@ -553,6 +555,7 @@ public class TestJobSubmission {
pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
pc.getConf().setProperty("pig.exec.reducers.max", "10");
+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
ConfigurationValidator.validatePigProperties(pc.getProperties());
conf = ConfigurationUtil.toConfiguration(pc.getProperties());
JobControlCompiler jcc = new JobControlCompiler(pc, conf);
{code}
With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442215#comment-13442215 ]
Dmitriy V. Ryaboy commented on PIG-2885:
----------------------------------------
Great, I think we are getting close.
I still don't like manually adding hbase's dependencies. Pretty sure we can add configurations to our ivy and make a similar distinction to HBase's, and depend on hbase compile->runtime in test, but on compile->master for the normal build.
Also, we should update all the docs that mention compatibility and mention somewhere that one needs to set HBASE_CLASSPATH.
I noticed that hadoop version bump. Is that required for hbase, or just something you took care of while you were in there? I'm not sure what we want to do about minor hadoop version compatibility, but just thought I'd bring that up...
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2885:
-------------------------------
Attachment: PIG-2885-2.patch
Updating the patch:
1. Stop bundling hbase.jar and zookeoper.jar with pig.jar. So there should be no longer incompatibility issues when using pig.jar with different versions of hbase.jar. But to use HBaseStorage, HBASE_HOME and ZOOKEEPER_HOME must be set by the user.
Note that I am adding protobuf-java.jar to pig.jar because otherwise it has to be explicitly added to PIG_CLASSPATH to use HBaseStorage, which is not very intuitive.
2. Bump hbase and zk to 0.94.1 and 3.4.3 respectively. Since we no longer bundle them in pig.jar, which versions we use doesn't matter. These jar files will be used for unit test only.
3. Make the unit test cases work with newer versions of hbase and zk.
4. Add hbase runtime dependencies to ivy.xml. I tried to find an alternative way of specifying distinct dependencies in ivy.xml without adding them explicitly but no luck.
Basically, what I want to do is as follows:
- pulls down hbase.jar only at compile time.
- pulls down runtime dependencies as well at test time.
What I tried was "conf=compile->master;test->default", but apparently, it doesn't work. Instead, runtime depedencies are always pulled down. Please let me know if anyone has a better idea about how to achieve this.
Thanks!
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465003#comment-13465003 ]
Dmitriy V. Ryaboy commented on PIG-2885:
----------------------------------------
HBASE-5680 is marked as resolved -- does that unblock us, or are we depending on them to make a release?
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Ionut Ignatescu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466139#comment-13466139 ]
Ionut Ignatescu commented on PIG-2885:
--------------------------------------
Is this patch compatible with older HBase&Hadoop versions or it just fixes problem with Hbase 0.94 and Hadoop 1.0.x?
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santhosh Srinivasan updated PIG-2885:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.12
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Fix For: 0.12
>
> Attachments: PIG-2885-2.patch, PIG-2885-3.patch, PIG-2885-4.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2885:
-------------------------------
Attachment: PIG-2885.patch
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466257#comment-13466257 ]
Cheolsoo Park commented on PIG-2885:
------------------------------------
Hi Ionut,
If you're asking about compatibility in terms of using pig with hbase, please see my [comment|https://issues.apache.org/jira/browse/PIG-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466255#comment-13466255].
This patch is not compatible with older versions of hbase, but that's fine because this is only for testing. Currently, pig already works fine with hbase-0.90/92/94, so nothing has to be fixed in that regard.
Thanks!
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465135#comment-13465135 ]
Dmitriy V. Ryaboy commented on PIG-2885:
----------------------------------------
Yeah let's disable them for now and move on.
I am sure this will bite us at some point.. is there an HBase ticket to publish jars compatible with hadoop-23?
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2885:
-------------------------------
Release Note: This makes hbase.jar and zookeeper.jar no longer be embedded in pig.jar, and thus, to use HBaseStorage with pig.jar, hbase.jar and zookeeper.jar must be present in classpath. To add them to classpath, the user can set either PIG_CLASSPATH=<path_to_hbase.jar:path_to_zookeeper.jar>, or HBASE_HOME=<hbase_home_dir> and ZOOKEEPER_HOME=<zookeeper_home_dir>.
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885-3.patch, PIG-2885-4.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442194#comment-13442194 ]
Cheolsoo Park commented on PIG-2885:
------------------------------------
Hi Dmitriy, they're defined in hbase's pom, but they are not pulled down by pig's ivy because the dependency is defined as follows:
{code}
<dependency org="org.apache.hbase" name="hbase" rev="${hbase.version}" conf="compile->master">
{code}
As can be seen, the "compile" configuration depends on "master" not "runtime", so runtime dependencies such as protobuf and high-scale-lib are not pulled down at compile time. I could change "master" to "runtime" instead of adding separate dependency tags, but then we would pull down unnecessary dependencies at compile time even though they're only needed at runtime.
Thanks!
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Santhosh Srinivasan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482930#comment-13482930 ]
Santhosh Srinivasan commented on PIG-2885:
------------------------------------------
Patch has been committed. Thanks Cheolsoo!
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885-3.patch, PIG-2885-4.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465118#comment-13465118 ]
Cheolsoo Park commented on PIG-2885:
------------------------------------
Hi Dmitriy,
What HBASE-5680 did is improving the error message (as shown below) when there is incompatibility between hbase and hadoop:
{code}
You may need to recompile HBase against Hadoop version 0.23.1 or change your hadoop jars to start properly
{code}
So the hbase jar in maven repository is still not compatible with hadoop 0.23.x and 2.0.x.
What I am thinking is to disable the hbase tests when hadoopversion=23. Or we have to download the source tarball, compile, and run the test. I think that this is a bit too much for a unit test.
If this sounds fine to everyone, I will resume working on this.
Thanks!
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2885:
-------------------------------
Status: Patch Available (was: Open)
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2885:
-------------------------------
Attachment: PIG-2885-4.patch
Bumping ZK to 3.4.4 since ZK 3.4.3 has known issues. (Please see Santhosh's comment for details.)
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885-3.patch, PIG-2885-4.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442203#comment-13442203 ]
Dmitriy V. Ryaboy commented on PIG-2885:
----------------------------------------
I think the right thing to do here would be to just stop bundling HBase altogether, and still depend on compile->master.
I'm not sure what that would look like (meaning, how users would provide path to their runtime hbase jar). Probably we just have to document that HBASE_CLASSPATH has to be set?
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442212#comment-13442212 ]
Cheolsoo Park commented on PIG-2885:
------------------------------------
I agree that we should stop bundling hbase jar and not change "compile->master". It also makes sense to ask the user to set HBASE_CLASSPATH.
But we still need to pull down hbase jar and its dependencies for *test*, right? For that, I added two dependency tags for protobuf and high-scale-lib with "test->default".
Thanks!
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442691#comment-13442691 ]
Cheolsoo Park commented on PIG-2885:
------------------------------------
Actually, we have a bigger problem. The hbase jar in the maven repository is not binary-compatible with hadoop 0.23 (HBASE-5680). To get hbase working with hadoop 0.23, we have to recompile the source code against hadoop 0.23.
Using the hbase jar from the maven repository makes "TestHBaseStorage -Dhadoopversion=23" fail with the following error:
{code}
2012-08-25 12:55:47,100 FATAL [Master:0;localhost.localdomain,49603,1345924546912] master.HMaster (HMaster.java:abort(1388)) - HBase is having a problem with its Hadoop jars. You may need to recompile HBase against Hadoop version 0.23.1 or change your hadoop jars to start properly
java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/FSConstants$SafeModeAction
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:524)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:324)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:112)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:480)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:343)
at java.lang.Thread.run(Thread.java:662)
{code}
I am going to table this for now.
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Attachments: PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2885) TestJobSumission and TestHBaseStorage
don't work with HBase 0.94 and ZK 3.4.3
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2885:
-------------------------------
Attachment: PIG-2885-3.patch
Removed tabs and tailing whitespaces. It may be easier to see code changes in the RB by hiding whitespace changes: https://reviews.apache.org/r/7676/
Tests done:
- ant test-commit
- ant test -Dtestcase=TestHBaseStorage -Dhadoopversion=20
- ant test -Dtestcase=TestJobSubmission -Dhadoopversion=20
- Manual testing HBaseStorage using pig.jar with hbase-0.90 and 0.94
Note that TestHBaseStorage is excluded in hadoopversion=23.
> TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3
> -----------------------------------------------------------------------------
>
> Key: PIG-2885
> URL: https://issues.apache.org/jira/browse/PIG-2885
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 1.0.3, CentOS 6.3 64 bit
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Priority: Minor
> Labels: hbase
> Attachments: PIG-2885-2.patch, PIG-2885-3.patch, PIG-2885.patch
>
>
> I ran into two unit test failures (TestJobSubmission and TestHBaseStorage) by bumping the version of HBase and ZK to 0.94 and 3.4.3 respectively in hadoop 1.0.3. I am opening a jira to capture what I found for future reference.
> - Two dependency libraries of HBase 0.94 are missing in ivy.xml - high-scale-lib and protobuf-java.
> - The HTable constructor in HBase 0.94 changed:
> {code}
> - HTable table = new HTable(TESTTABLE_2);
> + HTable table = new HTable(conf, TESTTABLE_2);
> {code}
> - The default client port of MiniZooKeeperCluster in HBase 0.94 is no longer 21818. Since it is chosen randomly at runtime, it has to be set in PigContext.
> {code}
> @@ -541,7 +543,7 @@ public class TestJobSubmission {
> // use the estimation
> Configuration conf = cluster.getConfiguration();
> HBaseTestingUtility util = new HBaseTestingUtility(conf);
> - util.startMiniZKCluster();
> + int clientPort = util.startMiniZKCluster().getClientPort();
> util.startMiniHBaseCluster(1, 1);
>
> String query = "a = load '/passwd';" +
> @@ -553,6 +555,7 @@ public class TestJobSubmission {
>
> pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", "100");
> pc.getConf().setProperty("pig.exec.reducers.max", "10");
> + pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, Integer.toString(clientPort));
> ConfigurationValidator.validatePigProperties(pc.getProperties());
> conf = ConfigurationUtil.toConfiguration(pc.getProperties());
> JobControlCompiler jcc = new JobControlCompiler(pc, conf);
> {code}
> With the attached patch, both tests pass with hadoop 1.0.3. Please note that TestHBaseStorage fails in hadoop 0.23, and I haven't investigated that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira