You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Thejas M Nair (JIRA)" <ji...@apache.org> on 2012/06/21 04:54:43 UTC
[jira] [Created] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Thejas M Nair created HIVE-3168:
-----------------------------------
Summary: LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
Key: HIVE-3168
URL: https://issues.apache.org/jira/browse/HIVE-3168
Project: Hive
Issue Type: Bug
Reporter: Thejas M Nair
LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413923#comment-13413923 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
Ashutosh,
Can you please apply this fix to 0.9 branch as well ?
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch, HIVE-3168.3.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3168:
---------------------------------
Component/s: Serializers/Deserializers
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-3168:
-----------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.9.1)
Status: Resolved (was: Patch Available)
Committed. Thanks, Thejas!
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch, HIVE-3168.3.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407375#comment-13407375 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
The command for clearing ivy cache for hive is {code} "rm rf ~/.ivy2/local/org.apache.hive/hive* ; rm rf ~/.ivy2/cache/org.apache.hive/hive*" {code}
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Neha Tomar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409264#comment-13409264 ]
Neha Tomar commented on HIVE-3168:
----------------------------------
Thanks a lot, Thejas. It is working fine now.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Neha Tomar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407893#comment-13407893 ]
Neha Tomar commented on HIVE-3168:
----------------------------------
Tried the given suggestion. Replaced everything, but still same issue. Not sure where I am making the mistake. I am using hcatalog released version 0.4.0. Can you please confirm if there is no change in this hcatalog code.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408135#comment-13408135 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
Neha,
Try 'ant clean' on the hive dir before building it.
To verify that the patched hive code is getting used, try adding a debug message in LazyBinaryObjectInspector around line 43 (after applying patch) (note the debug message would come out of map task). Or change the data setting the first byte, and see if changes your result -
{code}
System.arraycopy(bWritable.getBytes(), 0, data, 0, bWritable.getLength());
ba.setData(data);
{code}
to
{code}
System.arraycopy(bWritable.getBytes(), 0, data, 0, bWritable.getLength());
data[0]='Z';
ba.setData(data);
{code}
I am not sure if I tested with hcat 0.4 branch or trunk. But I don't think there has been any change that should affect this behavior, it should work with hcat 0.4 release as well.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411927#comment-13411927 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
New patch is ready for reivew.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch, HIVE-3168.3.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Neha Tomar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407404#comment-13407404 ]
Neha Tomar commented on HIVE-3168:
----------------------------------
Thanks a lot, Thejas. Will try this.
The data that I have pasted above is the hive output. And, the data actually stored is in base64 encoding only.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3168:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
@Thejas: I left some comments on phabricator. Thanks.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated HIVE-3168:
--------------------------------
Status: Patch Available (was: Reopened)
Yes, the updated patch on phabricator is ready for review.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413447#comment-13413447 ]
Hudson commented on HIVE-3168:
------------------------------
Integrated in Hive-trunk-h0.21 #1539 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1539/])
HIVE-3168: LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable (Thejas Nair via Ashutosh Chauhan) (Revision 1360812)
Result = SUCCESS
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1360812
Files :
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyBinaryObjectInspector.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaBinaryObjectInspector.java
* /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableBinaryObjectInspector.java
* /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/lazybinary/TestLazyBinarySerDe.java
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch, HIVE-3168.3.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach reopened HIVE-3168:
----------------------------------
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410576#comment-13410576 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
Can any committer please review the 2nd patch ? (also in phabricator link)
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402918#comment-13402918 ]
Carl Steinbach commented on HIVE-3168:
--------------------------------------
@Thejas: Is the patch ready for review?
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated HIVE-3168:
--------------------------------
Attachment: HIVE-3168.3.patch
HIVE-3168.3.patch - updated patch based on Ashutosh's comments in phabricator. Also updated patch on phabricator.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch, HIVE-3168.3.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Neha Tomar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405922#comment-13405922 ]
Neha Tomar commented on HIVE-3168:
----------------------------------
One more question here. Do we need to apply both the patches attached?
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated HIVE-3168:
--------------------------------
Attachment: HIVE-3168.1.patch
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Reporter: Thejas M Nair
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Neha Tomar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407286#comment-13407286 ]
Neha Tomar commented on HIVE-3168:
----------------------------------
Please let me know if anyone tried and these patches are working fine. I need to look into my setup if this works fine for you. Would be very helpful if I get any clue here. Thanks a lot.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated HIVE-3168:
--------------------------------
Fix Version/s: 0.9.1
0.10.0
Assignee: Thejas M Nair
Affects Version/s: 0.9.0
Status: Patch Available (was: Open)
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Neha Tomar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406258#comment-13406258 ]
Neha Tomar commented on HIVE-3168:
----------------------------------
Thanks Thejas. I applied only the second patch and the build was fine. However, I am still seeing the same incorrect values coming back. Could you please let me know if you have verified and this works fine for you.
e.g. I have following data in table.
a
ab
abc
abcd
abcde
z
And the values (this is in hexadecimal) coming back from HCatalog is :
61
616200
616263
616263640000
616263646500
7A6263646500
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated HIVE-3168:
--------------------------------
Attachment: HIVE-3168.2.patch
HIVE-3168.2.patch - attaching patch that has already been published in phabricator.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399000#comment-13399000 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
review board link - https://reviews.facebook.net/D3777
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401991#comment-13401991 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
Updated patch on phabricator. Fixes JavaBinaryObjectInspector and WritableBinaryObjectInspector as well. Also, tries to avoid copy of byte[] in BytesWritable .
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407374#comment-13407374 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
Yes, I tried the patch and it worked for me with hcatalog. I tested it using MR job that uses hcat to read data.
You might want to make sure that the you have deployed the updated hive jars and that hcatalog is using those.
hive build has some issues now, and I had to do "rm -rf ~/.ivy2/local/org.apache.hive/hive-* ; rm -rf ~/.ivy2/cache/org.apache.hive/hive-*" to clear the old jars. I am not sure if this affects only unit tests, or if it affects the package creation as well. But you might want to give that a try.
Hive(ie hcat as well) expects the binary column be base64 encoded (at least for text format). The values you pasted don't have the padding with '==', and my understanding is that base64 is expected to be padded to 4 bytes. Can you try with valid base64 encoding of printable ascii strings ?
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3168)
LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of
underlying BytesWritable
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406012#comment-13406012 ]
Thejas M Nair commented on HIVE-3168:
-------------------------------------
The 2nd patch replaces the first, so you need to apply on the 2nd one. I tried applying it to a fresh checkout of hive 0.9 branch and I was able to build it fine. Please post the build error if you think it is related to the patch.
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies beyond length of underlying BytesWritable
> -------------------------------------------------------------------------------------------------
>
> Key: HIVE-3168
> URL: https://issues.apache.org/jira/browse/HIVE-3168
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0, 0.9.1
>
> Attachments: HIVE-3168.1.patch, HIVE-3168.2.patch
>
>
> LazyBinaryObjectInspector.getPrimitiveJavaObject copies the full capacity of the LazyBinary's underlying BytesWritable object, which can be greater than the size of the actual contents.
> This leads to additional characters at the end of the ByteArrayRef returned. When the LazyBinary object gets re-used, there can be remnants of the later portion of previous entry.
> This was not seen while reading through hive queries, which I think is because a copy elsewhere seems to create LazyBinary with length == capacity. (probably LazyBinary copy constructor). This was seen when MR or pig used Hcatalog to read the data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira