[jira] [Commented] (HIVE-7777) Add CSV Serde based on OpenCSV

["Alon Goldshuv (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-7777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200053#comment-14200053 ] 

Alon Goldshuv commented on HIVE-7777:
-------------------------------------

While the serde works fine, it has an issue, which is quite serious IMO - It forces all the column types to String. This means that running a query on data that isn't all string type can return wrong query results. In the unit tests I see a single example of a table using all string columns, and in the tests linked here there are many tables with non-string types, but all the queries seem to be simple COUNT(*), which won't catch the problem.

Consider the following example:

{noformat}
CREATE EXTERNAL TABLE test (totalprice DECIMAL(38,10)) 
ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with 
serdeproperties ("separatorChar" = ",","quoteChar"= "'","escapeChar"= "\\") 
STORED AS TEXTFILE 
LOCATION '<some location>' 
tblproperties ("skip.header.line.count"="1");
{noformat}

Now consider this sql:

hive> select min(totalprice) from test;

in this case given my data, the result should have been 874.89, but the actual result became 100001.57 (as it is first according to byte ordering of a string type). this is a wrong result.

hive> desc extended test;
OK
o_totalprice        	string              	from deserializer
...

I apologize if it's a false alarm and I'm misusing the DDL somehow. Otherwise - this is a concern as wrong query results is a bad thing...


> Add CSV Serde based on OpenCSV
> ------------------------------
>
>                 Key: HIVE-7777
>                 URL: https://issues.apache.org/jira/browse/HIVE-7777
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Ferdinand Xu
>            Assignee: Ferdinand Xu
>              Labels: TODOC14
>             Fix For: 0.14.0
>
>         Attachments: HIVE-7777.1.patch, HIVE-7777.2.patch, HIVE-7777.3.patch, HIVE-7777.patch, csv-serde-master.zip
>
>
> There is no official support for csvSerde for hive while there is an open source project in github(https://github.com/ogrodnek/csv-serde). CSV is of high frequency in use as a data format.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)