You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Gordon Sommers (JIRA)" <ji...@apache.org> on 2010/07/27 16:05:16 UTC
[jira] Created: (HADOOP-6883) Text.toString violates its
abstraction
Text.toString violates its abstraction
--------------------------------------
Key: HADOOP-6883
URL: https://issues.apache.org/jira/browse/HADOOP-6883
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.20.1
Environment: Linux
Reporter: Gordon Sommers
I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
byte [] decoded = b64.decode(val.getBytes())
//this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
byte [] decoded = b64.decode(val.toString().getBytes());
//YES, toString() FIXES IT
Elsewhere in my code I also have:
Text curline = new Text(values.next().toString());
byte [] raw = base64.decode(curline.getBytes());
//This does work.
It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
RE: [jira] Commented: (HADOOP-6883) Text.toString violates its abstraction
Posted by ga...@thomsonreuters.com.
Please remove me from the list - thanks
Gary.Lutz@thomson.com
-----Original Message-----
From: Owen O'Malley (JIRA) [mailto:jira@apache.org]
Sent: Tuesday, July 27, 2010 1:27 PM
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6883) Text.toString violates its abstraction
[ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892880#action_12892880 ]
Owen O'Malley commented on HADOOP-6883:
---------------------------------------
No, it is never useful to look at the trailing bytes. They are just random bytes left over from a previous value.
> Text.toString violates its abstraction
> --------------------------------------
>
> Key: HADOOP-6883
> URL: https://issues.apache.org/jira/browse/HADOOP-6883
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: Linux
> Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have:
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6883) Text.toString violates its
abstraction
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-6883.
-----------------------------------
Resolution: Invalid
The proper call is:
{code}
b64.decode(val.getBytes(), 0, val.getLength());
{code}
Yes, it is confusing, but doing anything else would not perform acceptably. If you look at the javadoc for getBytes(), you'll see why your call fails.
> Text.toString violates its abstraction
> --------------------------------------
>
> Key: HADOOP-6883
> URL: https://issues.apache.org/jira/browse/HADOOP-6883
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: Linux
> Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have:
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6883) Text.toString violates its
abstraction
Posted by "Gordon Sommers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892862#action_12892862 ]
Gordon Sommers commented on HADOOP-6883:
----------------------------------------
Ah, just found the javadocs hosted by apache -- now I see what the issue was. Thanks. Is there a case where it's useful to dump the entire byte array and not just the valid ones?
> Text.toString violates its abstraction
> --------------------------------------
>
> Key: HADOOP-6883
> URL: https://issues.apache.org/jira/browse/HADOOP-6883
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: Linux
> Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have:
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6883) Text.toString violates its
abstraction
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892880#action_12892880 ]
Owen O'Malley commented on HADOOP-6883:
---------------------------------------
No, it is never useful to look at the trailing bytes. They are just random bytes left over from a previous value.
> Text.toString violates its abstraction
> --------------------------------------
>
> Key: HADOOP-6883
> URL: https://issues.apache.org/jira/browse/HADOOP-6883
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: Linux
> Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have:
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6883) Text.toString violates its
abstraction
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892820#action_12892820 ]
Owen O'Malley commented on HADOOP-6883:
---------------------------------------
I should also comment that I'm almost done with a patch that re-designs the generic serialization framework that includes native support for ProtocolBuffers, Thrift, Avro, and Writables. I already have all of the serializations working with both SequenceFiles and TFiles (with serialization layered on top named OFiles).
> Text.toString violates its abstraction
> --------------------------------------
>
> Key: HADOOP-6883
> URL: https://issues.apache.org/jira/browse/HADOOP-6883
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: Linux
> Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have:
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6883) Text.toString violates its
abstraction
Posted by "Gordon Sommers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892861#action_12892861 ]
Gordon Sommers commented on HADOOP-6883:
----------------------------------------
Thanks for the quick feedback. I'm not sure I understand why calling it like that would make a difference -- if the bytes we want to decode amount to the entire string that should have the same effect as b64.decode(val.getBytes()), no? I looked at the api here: http://www.docjar.com/docs/api/org/apache/hadoop/io/Text.html#getBytes and it didn't specify anything unusual. Is Text encoding the bytes differently from String?
In either case, I'm glad to hear that you're working on improving the serialization framework -- I think a lot of people, myself included, will really appreciate that when it gets released. Thanks!
> Text.toString violates its abstraction
> --------------------------------------
>
> Key: HADOOP-6883
> URL: https://issues.apache.org/jira/browse/HADOOP-6883
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.20.1
> Environment: Linux
> Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have:
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.