You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jingkei Ly (JIRA)" <ji...@apache.org> on 2009/03/25 13:56:52 UTC
[jira] Created: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
TupleWritable can return incorrect results if it contains more than 32 values
-----------------------------------------------------------------------------
Key: HADOOP-5571
URL: https://issues.apache.org/jira/browse/HADOOP-5571
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.1
Reporter: Jingkei Ly
When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
The following Junit test demonstrates the problem:
{code}
public void testWideTuple() throws Exception {
Text emptyText = new Text("Should be empty");
Writable[] values = new Writable[64];
Arrays.fill(values,emptyText);
values[42] = new Text("Number 42");
TupleWritable tuple = new TupleWritable(values);
tuple.setWritten(42);
for (int pos=0; pos<tuple.size();pos++) {
boolean has = tuple.has(pos);
if (pos == 42) {
assertTrue(has);
}
else {
assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
}
}
}
{code}
Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by jason hadoop <ja...@gmail.com>.
We have seen this behavior too
On Wed, Mar 25, 2009 at 5:56 AM, Jingkei Ly (JIRA) <ji...@apache.org> wrote:
> TupleWritable can return incorrect results if it contains more than 32
> values
>
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
>
>
> When attempting to do an outer join on 45 files with the
> CompositeInputFormat, I've been encountering unexpected results in the
> TupleWritable returned by the record reader. On closer inspection, it seems
> to be because TupleWritable.setWritten(int) is incorrectly setting some
> tuple positions as written, i.e when you set setWritten(42), it also sets
> position 10.
>
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos,
> has);
> }
> }
> }
> {code}
>
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to
> incorrectly return true.
>
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
[jira] Commented: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689653#action_12689653 ]
Chris Douglas commented on HADOOP-5571:
---------------------------------------
Sorry, I don't know what I was thinking about the unit test. +1 on the patch
bq. I was also thinking of raising a separate JIRA on replacing the written field in TupleWritable with a java.util.BitSet so that you can do joins over 64 datasets - do you have an opinion on this?
It might motivate some long-deferred work on memory consumption as well, but I think that's a good idea.
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5571:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this. Thanks Jingkei
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Fix For: 0.20.0
>
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689451#action_12689451 ]
Jingkei Ly commented on HADOOP-5571:
------------------------------------
There should be a unit test added to TestTupleWritable as part of the original patch.
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingkei Ly updated HADOOP-5571:
-------------------------------
Attachment: HADOOP-5571-1.patch
I think the problem is that some of the bit-shift operations in TupleWritable with the Long field, written, are done with Integers. I've attached a patch which I think fixes this problem and unit tests to test it.
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689453#action_12689453 ]
Jingkei Ly commented on HADOOP-5571:
------------------------------------
I was also thinking of raising a separate JIRA on replacing the written field in TupleWritable with a java.util.BitSet so that you can do joins over 64 datasets - do you have an opinion on this?
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689666#action_12689666 ]
Hadoop QA commented on HADOOP-5571:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12403606/HADOOP-5571-1.patch
against trunk revision 758593.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/142/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/142/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/142/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/142/console
This message is automatically generated.
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695442#action_12695442 ]
Hudson commented on HADOOP-5571:
--------------------------------
Integrated in Hadoop-trunk #796 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/796/])
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Fix For: 0.20.0
>
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5571:
----------------------------------
Assignee: Jingkei Ly
Status: Patch Available (was: Open)
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5571) TupleWritable can return incorrect
results if it contains more than 32 values
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689448#action_12689448 ]
Chris Douglas commented on HADOOP-5571:
---------------------------------------
Would it be possible to add a unit test for this?
> TupleWritable can return incorrect results if it contains more than 32 values
> -----------------------------------------------------------------------------
>
> Key: HADOOP-5571
> URL: https://issues.apache.org/jira/browse/HADOOP-5571
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: HADOOP-5571-1.patch
>
>
> When attempting to do an outer join on 45 files with the CompositeInputFormat, I've been encountering unexpected results in the TupleWritable returned by the record reader. On closer inspection, it seems to be because TupleWritable.setWritten(int) is incorrectly setting some tuple positions as written, i.e when you set setWritten(42), it also sets position 10.
> The following Junit test demonstrates the problem:
> {code}
> public void testWideTuple() throws Exception {
> Text emptyText = new Text("Should be empty");
> Writable[] values = new Writable[64];
> Arrays.fill(values,emptyText);
> values[42] = new Text("Number 42");
>
> TupleWritable tuple = new TupleWritable(values);
> tuple.setWritten(42);
>
> for (int pos=0; pos<tuple.size();pos++) {
> boolean has = tuple.has(pos);
> if (pos == 42) {
> assertTrue(has);
> }
> else {
> assertFalse("Tuple position is incorrectly labelled as set: " + pos, has);
> }
> }
> }
> {code}
> Similarly, TupleWritable.setWritten(9) also causes TupleWritable.has(41) to incorrectly return true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.