You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Yongzhi Chen (JIRA)" <ji...@apache.org> on 2015/06/25 20:49:05 UTC

[jira] [Commented] (HIVE-11112) ISO-8859-1 text output has fragments of previous longer rows appended

    [ https://issues.apache.org/jira/browse/HIVE-11112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601716#comment-14601716 ] 

Yongzhi Chen commented on HIVE-11112:
-------------------------------------

org.apache.hadoop.io.Text 's byte array is not necessarily null terminated. But 
SerDeUtils.transformTextToUTF8 method just use Text byte array to create String object which causes the unclean buffer issue.
Fix it by using String constructor with lengh info.


> ISO-8859-1 text output has fragments of previous longer rows appended
> ---------------------------------------------------------------------
>
>                 Key: HIVE-11112
>                 URL: https://issues.apache.org/jira/browse/HIVE-11112
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>    Affects Versions: 1.2.0
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>
> If a LazySimpleSerDe table is created using ISO 8859-1 encoding, query results for a string column are incorrect for any row that was preceded by a row containing a longer string.
> Example steps to reproduce:
> 1. Create a table using ISO 8859-1 encoding:
> CREATE TABLE person_lat1 (name STRING)
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ('serialization.encoding'='ISO8859_1');
> 2. Copy an ISO-8859-1 encoded text file into the appropriate warehouse folder in HDFS. I'll attach an example file containing the following text: 
> Müller,Thomas
> Jørgensen,Jørgen
> Peña,Andrés
> Nåm,Fæk
> 3. Execute SELECT * FROM person_lat1
> Result - The following output appears:
> +-------------------+--+
> | person_lat1.name |
> +-------------------+--+
> | Müller,Thomas |
> | Jørgensen,Jørgen |
> | Peña,Andrésørgen |
> | Nåm,Fækdrésørgen |
> +-------------------+--+



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)