You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org> on 2010/07/27 21:28:16 UTC
[jira] Created: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
-------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-2884
URL: https://issues.apache.org/jira/browse/HBASE-2884
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
{code}
Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-2884:
-------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.90.0
Resolution: Fixed
> TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2884
> URL: https://issues.apache.org/jira/browse/HBASE-2884
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Assignee: Todd Lipcon
> Fix For: 0.90.0
>
> Attachments: hbase-2884.txt, TestHFileOutputFormat-output.txt, TestHFileOutputFormat.txt
>
>
> {code}
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
> testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
> java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-2884:
-------------------------------
Status: Patch Available (was: Open)
> TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2884
> URL: https://issues.apache.org/jira/browse/HBASE-2884
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Assignee: Todd Lipcon
> Attachments: hbase-2884.txt, TestHFileOutputFormat-output.txt, TestHFileOutputFormat.txt
>
>
> {code}
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
> testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
> java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893273#action_12893273 ]
Todd Lipcon commented on HBASE-2884:
------------------------------------
Looked at this a bit -- the thing is that it's expecting 2048 entries, which would indicate mapred.map.tasks was set to 2. Each mapper is expected to generate 1024 rows. But, we seed Random() in RandomKVGeneratingMapper with System.currentTimeMillis - perhaps in this flaky case both mappers started in the same millisecond and thus generated identical data. When we go and scan, we only get 1024 rows because we just double-inserted everything?
> TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2884
> URL: https://issues.apache.org/jira/browse/HBASE-2884
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Attachments: TestHFileOutputFormat-output.txt, TestHFileOutputFormat.txt
>
>
> {code}
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
> testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
> java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kannan Muthukkaruppan updated HBASE-2884:
-----------------------------------------
Attachment: TestHFileOutputFormat.txt
TestHFileOutputFormat-output.txt
Uploaded test output files.
> TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2884
> URL: https://issues.apache.org/jira/browse/HBASE-2884
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Attachments: TestHFileOutputFormat-output.txt, TestHFileOutputFormat.txt
>
>
> {code}
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
> testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
> java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned HBASE-2884:
----------------------------------
Assignee: Todd Lipcon
> TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2884
> URL: https://issues.apache.org/jira/browse/HBASE-2884
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Assignee: Todd Lipcon
> Attachments: hbase-2884.txt, TestHFileOutputFormat-output.txt, TestHFileOutputFormat.txt
>
>
> {code}
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
> testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
> java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893479#action_12893479 ]
Kannan Muthukkaruppan commented on HBASE-2884:
----------------------------------------------
+1. Patch looks good.
> TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2884
> URL: https://issues.apache.org/jira/browse/HBASE-2884
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Assignee: Todd Lipcon
> Attachments: hbase-2884.txt, TestHFileOutputFormat-output.txt, TestHFileOutputFormat.txt
>
>
> {code}
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
> testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
> java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2884) TestHFileOutputFormat intermittent
diff: LoadIncrementalHFiles should put expected data in table
expected:<2048> but was:<1024>
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HBASE-2884:
-------------------------------
Attachment: hbase-2884.txt
Attached patch ensures that different mappers generate different data, even if they have the same random number generator. Tested this by changing it the old code to 'new Random(5)' and seeing that it failed with the same message Kannan saw. Then added the workaround to set the last byte of the key to the task ID, and verified that the test passed.
> TestHFileOutputFormat intermittent diff: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-2884
> URL: https://issues.apache.org/jira/browse/HBASE-2884
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Attachments: hbase-2884.txt, TestHFileOutputFormat-output.txt, TestHFileOutputFormat.txt
>
>
> {code}
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 160.582 sec <<< FAILURE!
> testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 85.676 sec <<< FAILURE!
> java.lang.AssertionError: LoadIncrementalHFiles should put expected data in table expected:<2048> but was:<1024>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:300)
> at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoad(TestHFileOutputFormat.java:248)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Will upload full test output shortly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.