[jira] [Commented] (HIVE-12718) skip.footer.line.count misbehaves on larger text files

["Charles Bernard (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859141#comment-15859141 ] 

Charles Bernard commented on HIVE-12718:
----------------------------------------

We are experiencing the same issue running CDH 5.8.0.

Our problem is that the wrong line (not the last one) is being skipped. Forcing one mapper only does not help.

> skip.footer.line.count misbehaves on larger text files
> ------------------------------------------------------
>
>                 Key: HIVE-12718
>                 URL: https://issues.apache.org/jira/browse/HIVE-12718
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 1.1.0
>         Environment: The bug was discovered and reproduced on a Cloudera Hadoop 5.4 distribution running on CentOS 6.4.
>            Reporter: Gergely Nagy
>            Priority: Minor
>
> We noticed that when working on a table backed by a larger (large enough to require splitting) text file, the {{skip.footer.line.count}} property of the table misbehaves: the footer is not being ignored.
> To reproduce, follow these steps:
> 1) Create a large file: {{for i in $(seq 1 100); do cat /usr/share/dict/words; done >large.txt}}
> 2) Upload it to HDFS (eg, as {{/tmp/words}})
> 3) Create an external table with {{skip.footer.line.count}} set: 
> {quote}
> CREATE EXTERNAL TABLE ext_words (word STRING)
>   ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
>   LINES TERMINATED BY '\n'
>   STORED AS TEXTFILE LOCATION '/tmp/words'
>   tblproperties("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {quote}
> 4) Count the number of times the last line (in this example, I assume that to be {{ZZZ}}) appears: {{SELECT COUNT( * ) FROM ext_words WHERE word = 'ZZZ';}}
> 5) Observe that it returns 100 instead of 99.
> Investigation showed that this happens when there are more than one mappers used for the job. If we increase the split size, to force using one mapper only, the problem did not occur.
> There may be other related issues as well, like the wrong line being skipped -- but we did not reproduce those yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)