You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2010/10/14 07:58:32 UTC
[jira] Created: (HBASE-3111) TestTableMapReduce broken up on hudson
TestTableMapReduce broken up on hudson
--------------------------------------
Key: HBASE-3111
URL: https://issues.apache.org/jira/browse/HBASE-3111
Project: HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.90.0
So, test has failed for various reasons since fixed over last week or so. It is currently failing when the reducer starts up. Its failing because its not picking up the zk servers location; its using stale config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3111) TestTableMapReduce broken up on
hudson
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921217#action_12921217 ]
stack commented on HBASE-3111:
------------------------------
So, the above 'fix' didn't work (looking closer, of course it didn't work). We're still picking up wrong ensemble out in TableOutputFormat. I can replicate the issue now by changing the hardcoded MiniZooKeeperCluster clientport underneath the deprecated HBaseClusterTestCase that is used by TestTableMapReduce. Digging.
> TestTableMapReduce broken up on hudson
> --------------------------------------
>
> Key: HBASE-3111
> URL: https://issues.apache.org/jira/browse/HBASE-3111
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> So, test has failed for various reasons since fixed over last week or so. It is currently failing when the reducer starts up. Its failing because its not picking up the zk servers location; its using stale config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3111) TestTableMapReduce broken up on hudson
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3111:
-------------------------
Attachment: 3111-2nd-attempt.txt
Problem looks to have been a little more insidious. TableInputFormat implements Configurable. TableOutputFormat does not and hbase configs were hoisted up into TOF inside in the RecordWriter constructor by calling HBaseConfiguration.addHbaseResources(context); This doesn't looks like it would ever pick up config. set by the job. Let me try this patch against hudson. Committing it now. Here's a commit message.
{code}
M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
Changed mentions of 'quorum' to 'ensemble'.
M src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableReducer.java
M src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java
Minor formatting.
M src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
Removed unused imports and minor formatting.
M src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
Documented what the quorumAddress parameter is for.
Removed an unnecessary looking HBaseConfiguration.addHbaseResources(conf);
(and adjusted documentation of job to say no hbase config set by the
reduce setup method).
M src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
Made this class implment Configurable. Moved creation of table from
RecordWrite constructor based off passed TaskAttemptContext instead
into the new setConf method. Added table and conf data members.
{code}
> TestTableMapReduce broken up on hudson
> --------------------------------------
>
> Key: HBASE-3111
> URL: https://issues.apache.org/jira/browse/HBASE-3111
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
> Attachments: 3111-2nd-attempt.txt
>
>
> So, test has failed for various reasons since fixed over last week or so. It is currently failing when the reducer starts up. Its failing because its not picking up the zk servers location; its using stale config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3111) TestTableMapReduce broken up on hudson
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3111:
-------------------------
Assignee: stack
Status: Patch Available (was: Open)
> TestTableMapReduce broken up on hudson
> --------------------------------------
>
> Key: HBASE-3111
> URL: https://issues.apache.org/jira/browse/HBASE-3111
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.90.0
>
> Attachments: 3111-2nd-attempt.txt
>
>
> So, test has failed for various reasons since fixed over last week or so. It is currently failing when the reducer starts up. Its failing because its not picking up the zk servers location; its using stale config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3111) TestTableMapReduce broken up on
hudson
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920856#action_12920856 ]
stack commented on HBASE-3111:
------------------------------
I'm going to try this change over night:
{code}
Index: src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
===================================================================
--- src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java (revision 1022380)
+++ src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java (working copy)
@@ -180,7 +180,6 @@
job.setOutputKeyClass(ImmutableBytesWritable.class);
job.setOutputValueClass(Writable.class);
if (partitioner == HRegionPartitioner.class) {
- HBaseConfiguration.addHbaseResources(conf);
job.setPartitionerClass(HRegionPartitioner.class);
HTable outputTable = new HTable(conf, table);
int regions = outputTable.getRegionsInfo().size();
{code}
My supposition is that its overwriting the config. we've passed in which has in it the amended zk location.
This is the last failing test on hudson. If this passes, I'll make up a better patch... will need documentation.
> TestTableMapReduce broken up on hudson
> --------------------------------------
>
> Key: HBASE-3111
> URL: https://issues.apache.org/jira/browse/HBASE-3111
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> So, test has failed for various reasons since fixed over last week or so. It is currently failing when the reducer starts up. Its failing because its not picking up the zk servers location; its using stale config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3111) TestTableMapReduce broken up on
hudson
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921017#action_12921017 ]
stack commented on HBASE-3111:
------------------------------
This fix looks to have taken care of the failing. Lets run hudson a few more times before closing.
> TestTableMapReduce broken up on hudson
> --------------------------------------
>
> Key: HBASE-3111
> URL: https://issues.apache.org/jira/browse/HBASE-3111
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> So, test has failed for various reasons since fixed over last week or so. It is currently failing when the reducer starts up. Its failing because its not picking up the zk servers location; its using stale config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3111) TestTableMapReduce broken up on hudson
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3111:
-------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
We don't see this failure in the last few hudson runs. Resolving.
> TestTableMapReduce broken up on hudson
> --------------------------------------
>
> Key: HBASE-3111
> URL: https://issues.apache.org/jira/browse/HBASE-3111
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.90.0
>
> Attachments: 3111-2nd-attempt.txt
>
>
> So, test has failed for various reasons since fixed over last week or so. It is currently failing when the reducer starts up. Its failing because its not picking up the zk servers location; its using stale config.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.