You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2006/11/18 00:59:37 UTC
[jira] Created: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Jute record is inappropriate
The underlying data structure, ByteArrayOutputStream, for buffer type of Jute record is inappropriate
------------------------------------------------------------------------------------------------------
Key: HADOOP-735
URL: http://issues.apache.org/jira/browse/HADOOP-735
Project: Hadoop
Issue Type: Bug
Reporter: Runping Qi
With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465603 ]
Doug Cutting commented on HADOOP-735:
-------------------------------------
You've also added bounds checking to all of the raw comparator i/o primitives. That might be a good thing, but I'm not sure why its part of this patch. It seems like a separate issue that should be benchmarked, to make sure that it doesn't affect performance, as these comparators are the heart of our innermost sort loops.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Work started: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-735 started by Milind Bhandarkar.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-735:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Milind!
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465608 ]
Milind Bhandarkar commented on HADOOP-735:
------------------------------------------
Another option was to have a separate serialization format for BytesWritable (one that uses VInt instead of Int to denote length of the buffer) when used inside a Record. That would not require the above change, but it would be ugly to have diffferent serialization formats for the same datatype in two diffferent contexts.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Jute record is inappropriate
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-735?page=all ]
Owen O'Malley updated HADOOP-735:
---------------------------------
Component/s: record
(was: io)
Description:
With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
It seems BytesWritable is a better replacement.
was:
With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
It seems BytesWritable is a better replacement.
Affects Version/s: 0.9.2
Assignee: Milind Bhandarkar (was: Owen O'Malley)
> The underlying data structure, ByteArrayOutputStream, for buffer type of Jute record is inappropriate
> ------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: http://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-735:
-------------------------------------
Fix Version/s: 0.11.0
Status: Patch Available (was: In Progress)
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-735:
-------------------------------------
Status: Open (was: Patch Available)
I have removed this patch, because it creates incompatibility between serialization formats for buffer in c++ and java. I will upload the correct patch soon. The new patch will also eliminate the need to modify the WritableComparator's read* static methods.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465597 ]
Milind Bhandarkar commented on HADOOP-735:
------------------------------------------
Attached a patch that replaces ByteArrayOutputStream as a buffer-representation in Hadoop record I/O by BytesWritable. Waring, this would break any existing code that uses records containing buffer type in Hadoop Record I/O.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-735:
-------------------------------------
Attachment: (was: BytesWritable.patch)
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-735:
-------------------------------------
Attachment: BytesWritable.patch
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465622 ]
Milind Bhandarkar commented on HADOOP-735:
------------------------------------------
I think throwing IOException in the rest of the read* methods (even if one is reading from byte[]) is the right thing to do. If there are performance implications of doing an explicit length check, I could remove the explicit checkLength call, and replace it by try-catch, and throw an IOException on catching ArrayIndexOutOfBounds exception. This will still have some performance implications, but it will be smaller than an explicit method call and bounds check.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465621 ]
Doug Cutting commented on HADOOP-735:
-------------------------------------
Okay, now I get it.
Mightn't it be easier to change the exception thrown by readVInt to a RuntimeException like ArrayIndexOutOfBounds? That's a back-compatible change. Then, separately, we can consider whether to do more bounds checking in all of the other raw comparator methods. Would that make sense?
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-735:
-------------------------------------
Summary: The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate (was: The underlying data structure, ByteArrayOutputStream, for buffer type of Jute record is inappropriate)
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465617 ]
Milind Bhandarkar commented on HADOOP-735:
------------------------------------------
The public static methods readInt and readVInt of WritableComparator had different contracts as far as exceptions are concerned. readVInt throws IOException if the supplied byte[] is smaller than expected, whereas readInt does not (it will throw ArrayIndexOutOfBounds exception, which being a RuntimeException, need not be declared.) The generated raw compare method contract will differ (or we will have to catch alll exceptions, log them and ignore them) if we are either using readVInt or readInt. That is the issue.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465609 ]
Doug Cutting commented on HADOOP-735:
-------------------------------------
> the generated code was catching IOException, and resulted in a compile-error for trying to catch an exception that was not thrown
So the fix is to cause an exception to be thrown? I still don't follow. Code which processes a buffer only is technically not doing i/o, so IOException is also a strange one to add to these methods.
I also still don't completely see the issue. Can you provide some examples of the generated code, or point to where in the generated test cases I might look? Thanks.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-735:
-------------------------------------
Status: Patch Available (was: Open)
I have uploaded a patch that uses BytesWritable to represent Hadoop Record I/O's buffer type instead of ByteArrayOutputStream.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465607 ]
Milind Bhandarkar commented on HADOOP-735:
------------------------------------------
Indeed. This was necessary because the variable sized integer and longs were throwing IOException in their read methods. The serialization of BytesWritable uses int. For a record that contains only a buffer, the generated code was catching IOException, and resulted in a compile-error for trying to catch an exception that was not thrown. That's why it is part of this patch.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465886 ]
Hadoop QA commented on HADOOP-735:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12349207/BytesWritable.patch applied and successfully tested against trunk revision r497583.
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-735) The underlying data structure,
ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-735:
-------------------------------------
Attachment: BytesWritable.patch
> The underlying data structure, ByteArrayOutputStream, for buffer type of Hadoop record is inappropriate
> --------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-735
> URL: https://issues.apache.org/jira/browse/HADOOP-735
> Project: Hadoop
> Issue Type: Bug
> Components: record
> Affects Versions: 0.9.2
> Reporter: Runping Qi
> Assigned To: Milind Bhandarkar
> Fix For: 0.11.0
>
> Attachments: BytesWritable.patch
>
>
> With ByteArrayOutputStream as the underlying data structure for a buffer, the user is forced to convert it into a byte [] object in order to do any operations other than sequence append on the buffer. The convertion will create a new copy of bytes. That will cause huge performance problem.
> It seems BytesWritable is a better replacement.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira