You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Gergely Nagy (JIRA)" <ji...@apache.org> on 2015/12/21 11:46:46 UTC
[jira] [Created] (HIVE-12718) skip.footer.line.count misbehaves on
larger text files
Gergely Nagy created HIVE-12718:
-----------------------------------
Summary: skip.footer.line.count misbehaves on larger text files
Key: HIVE-12718
URL: https://issues.apache.org/jira/browse/HIVE-12718
Project: Hive
Issue Type: Bug
Affects Versions: 1.1.0
Environment: The bug was discovered and reproduced on a Cloudera Hadoop 5.4 distribution running on CentOS 6.4.
Reporter: Gergely Nagy
Priority: Minor
We noticed that when working on a table backed by a larger (large enough to require splitting) text file, the {{skip.footer.line.count}} property of the table misbehaves: the footer is not being ignored.
To reproduce, follow these steps:
1) Create a large file: {{for i in $(seq 1 100); do cat /usr/share/dict/words; done >large.txt}}
2) Upload it to HDFS (eg, as {{/tmp/words}})
3) Create an external table with {{skip.footer.line.count}} set:
{quote}
CREATE EXTERNAL TABLE ext_words (word STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE LOCATION '/tmp/words'
tblproperties("skip.header.line.count"="1", "skip.footer.line.count"="1");
{quote}
4) Count the number of times the last line (in this example, I assume that to be {{ZZZ}}) appears: {{SELECT COUNT( * ) FROM ext_words WHERE word = 'ZZZ';}}
5) Observe that it returns 100 instead of 99.
Investigation showed that this happens when there are more than one mappers used for the job. If we increase the split size, to force using one mapper only, the problem did not occur.
There may be other related issues as well, like the wrong line being skipped -- but we did not reproduce those yet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)