You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/05/20 23:44:41 UTC
[jira] [Created] (MAPREDUCE-4274) MapOutputBuffer should use native
byte order for kvmeta
Todd Lipcon created MAPREDUCE-4274:
--------------------------------------
Summary: MapOutputBuffer should use native byte order for kvmeta
Key: MAPREDUCE-4274
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279880#comment-13279880 ]
Hadoop QA commented on MAPREDUCE-4274:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528357/mapreduce-4274.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2402//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2402//console
This message is automatically generated.
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280375#comment-13280375 ]
Todd Lipcon commented on MAPREDUCE-4274:
----------------------------------------
Thanks Bobby!
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4274) MapOutputBuffer should use native
byte order for kvmeta
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-4274:
-----------------------------------
Attachment: mapreduce-4274.txt
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280223#comment-13280223 ]
Robert Joseph Evans commented on MAPREDUCE-4274:
------------------------------------------------
I am not an expert on this code. I dug into it, but the code is somewhat complex, so I want to check one thing with you first before giving it a +1. kvmeta is a wrapper around kvbuffer, but it only is used to store the offsets into kvbuffer where the data is stored not the keys and values themselves. Those are written into kvbuffer bypassing kvmeta. So, even though kvbuffer is handed directly to the user supplied RawComparator code, the bytes in between the offsets given to RawComparator are the same as they were before the change. Is this analysis correct?
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280474#comment-13280474 ]
Hudson commented on MAPREDUCE-4274:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2296 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2296/])
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280973#comment-13280973 ]
Hudson commented on MAPREDUCE-4274:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1087 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1087/])
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280391#comment-13280391 ]
Hudson commented on MAPREDUCE-4274:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2350 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2350/])
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280382#comment-13280382 ]
Hudson commented on MAPREDUCE-4274:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2277 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2277/])
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280258#comment-13280258 ]
Todd Lipcon commented on MAPREDUCE-4274:
----------------------------------------
Hey Robert. Yep, that is my understanding as well. Of course I could have made a mistake :) But from what I know of this code (I helped review MAPREDUCE-64 a couple years back) this should be perfectly compatible.
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280894#comment-13280894 ]
Hudson commented on MAPREDUCE-4274:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1053 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1053/])
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. (todd via bobby) (Revision 1341151)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341151
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4274) MapOutputBuffer should use native
byte order for kvmeta
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4274:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
2.0.0
Status: Resolved (was: Patch Available)
Thanks Todd,
I put this into trunk and branch-2.
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 2.0.0, 3.0.0
>
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4274) MapOutputBuffer should use
native byte order for kvmeta
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280364#comment-13280364 ]
Robert Joseph Evans commented on MAPREDUCE-4274:
------------------------------------------------
Thanks, I just wanted a sanity check on that. +1. I'll merge it into trunk, and branch-2.
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4274) MapOutputBuffer should use native
byte order for kvmeta
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-4274:
-----------------------------------
Status: Patch Available (was: Open)
> MapOutputBuffer should use native byte order for kvmeta
> -------------------------------------------------------
>
> Key: MAPREDUCE-4274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4274
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task
> Affects Versions: 2.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mapreduce-4274.txt
>
>
> I don't have a benchmark to support this, but this should give a small CPU improvement on the map output buffer: currently, we create {{kvmeta}} as {{ByteBuffer.wrap(kvbuffer).asIntBuffer()}}. According to the javadocs, the resulting int buffer will inherit its byte order from the ByteBuffer it comes from, and the byte buffer defaults to BIG_ENDIAN. Thus, all of our int access to/from the buffer will require byte-swapping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira