You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2008/03/18 22:49:24 UTC
[jira] Created: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
------------------------------------------------------------------------------------------------------------------------------
Key: HADOOP-3046
URL: https://issues.apache.org/jira/browse/HADOOP-3046
Project: Hadoop Core
Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 0.17.0
It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580202#action_12580202 ]
Chris Douglas commented on HADOOP-3046:
---------------------------------------
BytesWritable doesn't use vints, so the offsets are fixed; Text does, but it already uses WritableUtils::getVIntSize which should be all that's required.
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.17.0
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581022#action_12581022 ]
Hadoop QA commented on HADOOP-3046:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378349/3046.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2019/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2019/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2019/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2019/console
This message is automatically generated.
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.17.0
>
> Attachments: 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580127#action_12580127 ]
Doug Cutting commented on HADOOP-3046:
--------------------------------------
The length passed includes the bytes that encode the prefixed length, so we'll need to account for that. a length of 128 means a 1 byte prefix + 127 byte payload, 130 = 2 prefix + 128 payload, 129 should never happen, etc. Right?
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.17.0
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3046:
----------------------------------
Attachment: 3046.patch
This patch adds a decodeVIntSize that given the first byte of a vint decodes the total number of bytes. It uses this method to determine the number of bytes to skip over and subtract from the length.
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.17.0
>
> Attachments: 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3046:
----------------------------------
Status: Patch Available (was: Open)
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3046-2.patch, 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3046:
----------------------------------
Attachment: 3046-2.patch
This patch fixes a wrong value bug and includes a test case. It also substantially simplifies getVIntSize.
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3046-2.patch, 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581755#action_12581755 ]
Hadoop QA commented on HADOOP-3046:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12378526/3046-2.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2041/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2041/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2041/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2041/console
This message is automatically generated.
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3046-2.patch, 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582149#action_12582149 ]
Chris Douglas commented on HADOOP-3046:
---------------------------------------
+1
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3046-2.patch, 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582611#action_12582611 ]
Hudson commented on HADOOP-3046:
--------------------------------
Integrated in Hadoop-trunk #443 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/443/])
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3046-2.patch, 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580487#action_12580487 ]
Doug Cutting commented on HADOOP-3046:
--------------------------------------
> BytesWritable doesn't use vints, so the offsets are fixed
Good point. So the fix is as simple as:
{noformat}
public int compare(byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2) {
- int size1 = readInt(b1, s1);
- int size2 = readInt(b2, s2);
- return compareBytes(b1, s1+4, size1, b2, s2+4, size2);
+ return compareBytes(b1, s1+4, l1-4, b2, s2+4, l2-4);
}
}
{noformat}
> Text does, but it already uses WritableUtils::getVIntSize which should be all that's required.
I think this case is a bit more complicated, as I mentioned above. To calculate the length without parsing it from the buffer requires some VInt logic that's not in getVIntSize. We're passed x and we we need to compute y, where x = getVIntSize(y) + y.
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.17.0
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3046:
----------------------------------
Status: Patch Available (was: Open)
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.17.0
>
> Attachments: 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3046:
----------------------------------
Status: Open (was: Patch Available)
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3046-2.patch, 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3046) Text and BytesWritable's raw
comparators should use the lengths provided instead of rebuilding them from
scratch using readInt
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3046:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Text and BytesWritable's raw comparators should use the lengths provided instead of rebuilding them from scratch using readInt
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3046
> URL: https://issues.apache.org/jira/browse/HADOOP-3046
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: 3046-2.patch, 3046.patch
>
>
> It would be much faster to use the key length provided by the raw compare function rather than rebuilding the integer lengths back up from bytes twice for every comparison in the sort.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.