You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hive.apache.org by "Panagiotis Garefalakis (Jira)" <ji...@apache.org> on 2021/10/08 20:24:00 UTC

[jira] [Resolved] (HIVE-25521) Data corruption when concatenating files with different compressions in same table/partition

     [ https://issues.apache.org/jira/browse/HIVE-25521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Panagiotis Garefalakis resolved HIVE-25521.
-------------------------------------------
    Resolution: Fixed

> Data corruption when concatenating files with different compressions in same table/partition
> --------------------------------------------------------------------------------------------
>
>                 Key: HIVE-25521
>                 URL: https://issues.apache.org/jira/browse/HIVE-25521
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Harish JP
>            Assignee: Harish JP
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently if files of different compressions are in same directory then concatenate can fail and cause data corruption. This happens because file can be moved by one task as incompatible file and the other tasks will fail after this.
>  
> This issue is addressed in this Jira by only processing a file in one task where offset 0 is process and ignoring the the file in all other tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)