[jira] [Commented] (HIVE-13285) Orc concatenation may drop old files from moving to final path

["Prasanth Jayachandran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-13285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194628#comment-15194628 ] 

Prasanth Jayachandran commented on HIVE-13285:
----------------------------------------------

[~gopalv]/[~daijy] Could someone please take a look at this patch? This patch makes sure closeOp() is called even when outWriter is null as we might have some incompatible files to move to final path.

> Orc concatenation may drop old files from moving to final path
> --------------------------------------------------------------
>
>                 Key: HIVE-13285
>                 URL: https://issues.apache.org/jira/browse/HIVE-13285
>             Project: Hive
>          Issue Type: Bug
>          Components: ORC
>    Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.3.0, 2.0.0, 2.1.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Critical
>         Attachments: HIVE-13285.1.patch
>
>
> ORC concatenation uses combine hive input format for merging files. Under specific case where all files within a combine split are incompatible for merge (old files without stripe statistics) then these files are added to incompatible file set. But this file set is not processed as closeOp() will not be called (no output file writer will exist which will skip super.closeOp()). As a result, these incompatible files are not moved to final path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)