You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Prasanth Jayachandran (JIRA)" <ji...@apache.org> on 2017/07/13 21:42:00 UTC
[jira] [Updated] (HIVE-17085) ORC file merge/concatenation should
do full schema check
[ https://issues.apache.org/jira/browse/HIVE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasanth Jayachandran updated HIVE-17085:
-----------------------------------------
Description:
ORC merging/concatenation compatibility check just looks for column count match at outer level. ORC schema evolution now supports inner structs as well. With that outer level column count will match but inner column level will not match. Compatibility check should do full schema match before merging/concatenation. This issue will not cause data loss but will cause task failures with exception like below
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to close OrcFileMergeOperator
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:247)
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.processKeyValuePairs(OrcFileMergeOperator.java:172)
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.process(OrcFileMergeOperator.java:72)
at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:212)
... 16 more
Caused by: java.lang.IllegalArgumentException: Column has wrong number of index entries found: 0 expected: 1
at org.apache.orc.impl.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:695)
at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:2147)
at org.apache.orc.impl.WriterImpl.flushStripe(WriterImpl.java:2661)
at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:2834)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:321)
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:243)
... 19 more
{code}
Concatenation should also make sure writer version is matching (it currently checks only file version match).
was:
ORC merging/concatenation compatibility check just looks for column count match at outer level. ORC schema evolution now supports inner structs as well. With that outer level column count will match but inner column level will not match. Compatibility check should do full schema match before merging/concatenation. This issue will not cause data loss but will cause task failures with exception like below
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to close OrcFileMergeOperator
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:247)
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.processKeyValuePairs(OrcFileMergeOperator.java:172)
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.process(OrcFileMergeOperator.java:72)
at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:212)
... 16 more
Caused by: java.lang.IllegalArgumentException: Column has wrong number of index entries found: 0 expected: 1
at org.apache.orc.impl.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:695)
at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:2147)
at org.apache.orc.impl.WriterImpl.flushStripe(WriterImpl.java:2661)
at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:2834)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:321)
at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:243)
... 19 more
{code}
> ORC file merge/concatenation should do full schema check
> --------------------------------------------------------
>
> Key: HIVE-17085
> URL: https://issues.apache.org/jira/browse/HIVE-17085
> Project: Hive
> Issue Type: Bug
> Components: ORC
> Affects Versions: 2.2.0, 2.3.0, 3.0.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
>
> ORC merging/concatenation compatibility check just looks for column count match at outer level. ORC schema evolution now supports inner structs as well. With that outer level column count will match but inner column level will not match. Compatibility check should do full schema match before merging/concatenation. This issue will not cause data loss but will cause task failures with exception like below
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to close OrcFileMergeOperator
> at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:247)
> at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.processKeyValuePairs(OrcFileMergeOperator.java:172)
> at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.process(OrcFileMergeOperator.java:72)
> at org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.processRow(MergeFileRecordProcessor.java:212)
> ... 16 more
> Caused by: java.lang.IllegalArgumentException: Column has wrong number of index entries found: 0 expected: 1
> at org.apache.orc.impl.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:695)
> at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:2147)
> at org.apache.orc.impl.WriterImpl.flushStripe(WriterImpl.java:2661)
> at org.apache.orc.impl.WriterImpl.close(WriterImpl.java:2834)
> at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:321)
> at org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:243)
> ... 19 more
> {code}
> Concatenation should also make sure writer version is matching (it currently checks only file version match).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)