You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Daniel Doubrovkine (JIRA)" <ji...@apache.org> on 2017/04/02 13:53:41 UTC

[jira] [Commented] (HIVE-16351) Hive confused by CR/LFs

    [ https://issues.apache.org/jira/browse/HIVE-16351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952685#comment-15952685 ] 

Daniel Doubrovkine commented on HIVE-16351:
-------------------------------------------

Those NULLs are actually from an empty record. Is that right? Should the last empty line be ignored since these are text files and if so which components responsibility is it? 

{code}
~$ cat /tmp/test.json 
{"text":"foo\nbar","number":123}
{"text":"bar\nfoo","number":345}
~$ perl -pe 'chomp if eof' /tmp/test.json > /tmp/test2.json
~$ cat /tmp/test2.json 
{"text":"foo\nbar","number":123}
{"text":"bar\nfoo","number":345}~$ 
$ hadoop fs -put -f /tmp/test2.json /user/data/test.json
$ hive
hive> SELECT * FROM test;
OK
foo
bar	123
bar
foo	345
{code}

> Hive confused by CR/LFs
> -----------------------
>
>                 Key: HIVE-16351
>                 URL: https://issues.apache.org/jira/browse/HIVE-16351
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, Serializers/Deserializers
>    Affects Versions: 1.2.1
>         Environment: Hadoop 2.7.3
>            Reporter: Daniel Doubrovkine
>
> From https://github.com/rcongiu/Hive-JSON-Serde/issues/65
> This happens with both JSON and MongoDB connector Serde, so I don't believe this is a Serde bug.
> Using http://www.congiu.net/hive-json-serde/1.3.6/cdh4/json-serde-1.3.6-jar-with-dependencies.jar placed into /usr/local/Cellar/apache-hive-1.2.1/lib
> A dummy test.json with a CR/LF
> {code}
> $ cat /tmp/test.json
> {"text":"foo\nbar","number":123}
> $ hadoop fs -mkdir /user/data
> $ hadoop fs -put -f /tmp/test.json /user/data/test.json
> $ hive
> hive> CREATE DATABASE test;
> hive> CREATE EXTERNAL TABLE test ( text string )
>     > ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
>     > LOCATION '/user/data';
> hive> SELECT * FROM test;
> foo
> bar	123
> NULL	NULL
> {code}
> You can see how that's totally wrong, there's only one row of data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)