[jira] [Created] (HIVE-21266) Issue with single delta file

["Eugene Koifman (JIRA)" <ji...@apache.org>]

Eugene Koifman created HIVE-21266:
-------------------------------------

             Summary: Issue with single delta file
                 Key: HIVE-21266
                 URL: https://issues.apache.org/jira/browse/HIVE-21266
             Project: Hive
          Issue Type: Sub-task
    Affects Versions: 4.0.0
            Reporter: Eugene Koifman
            Assignee: Vaibhav Gumashta


[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]

 
{noformat}
if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1) {
      LOG.debug("Not compacting {}; current base is {} and there are {} deltas and {} originals", sd.getLocation(), dir
          .getBaseDirectory(), deltaCount, origCount);
      return;
    }
 {noformat}

Is problematic.
Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where {{txnid:13}} was aborted.  The code above will not rewrite the delta (which drops anything that belongs to the aborted txn) and transition the compaction to "ready_for_cleaning" which will drop the metadata about the aborted txn.  Now aborted data will come back as committed.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)