[jira] [Work logged] (HIVE-24852) Add support for Snapshots during external table replication

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-24852?focusedWorklogId=592937&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-592937 ]

ASF GitHub Bot logged work on HIVE-24852:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 04/May/21 18:52
            Start Date: 04/May/21 18:52
    Worklog Time Spent: 10m 
      Work Description: ayushtkn commented on a change in pull request #2043:
URL: https://github.com/apache/hive/pull/2043#discussion_r626025516



##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -1197,6 +1224,103 @@ public boolean runDistCp(List<Path> srcPaths, Path dst, Configuration conf) thro
     }
   }
 
+  public boolean runDistCpWithSnapshots(String snap1, String snap2, List<Path> srcPaths, Path dst, Configuration conf)
+      throws IOException {
+    DistCpOptions options = new DistCpOptions.Builder(srcPaths, dst).withSyncFolder(true).withUseDiff(snap1, snap2)
+        .preserve(FileAttribute.BLOCKSIZE).preserve(FileAttribute.XATTR).build();
+
+    List<String> params = constructDistCpWithSnapshotParams(srcPaths, dst, snap1, snap2, conf, "-diff");
+    try {
+      conf.setBoolean("mapred.mapper.new-api", true);
+      DistCp distcp = new DistCp(conf, options);
+      int returnCode = distcp.run(params.toArray(new String[0]));
+      if (returnCode == 0) {
+        return true;
+      } else if (returnCode == DistCpConstants.INVALID_ARGUMENT) {
+        // Handling FileNotFoundException, if source got deleted, in that case we don't want to copy either, So it is
+        // like a success case, we didn't had anything to copy and we copied nothing, so, we need not to fail.
+        LOG.warn("Copy failed with INVALID_ARGUMENT for source: {} to target: {} snapshot1: {} snapshot2: {} "
+            + "params: {}", srcPaths, dst, snap1, snap2, params);
+        return true;
+      } else if (returnCode == DistCpConstants.UNKNOWN_ERROR && conf
+          .getBoolean("hive.repl.externaltable.snapshot.overwrite.target", true)) {
+        // Check if this error is due to target modified.
+        if (shouldRdiff(dst, conf, snap1)) {
+          LOG.warn("Copy failed due to target modified. Attempting to restore back the target. source: {} target: {} "
+              + "snapshot: {}", srcPaths, dst, snap1);
+          List<String> rParams = constructDistCpWithSnapshotParams(srcPaths, dst, ".", snap1, conf, "-rdiff");
+          DistCp rDistcp = new DistCp(conf, options);
+          returnCode = rDistcp.run(rParams.toArray(new String[0]));
+          if (returnCode == 0) {
+            LOG.info("Target restored to previous state.  source: {} target: {} snapshot: {}. Reattempting to copy.",
+                srcPaths, dst, snap1);
+            dst.getFileSystem(conf).deleteSnapshot(dst, snap1);
+            dst.getFileSystem(conf).createSnapshot(dst, snap1);
+            returnCode = distcp.run(params.toArray(new String[0]));
+            if (returnCode == 0) {
+              return true;
+            } else {
+              LOG.error("Copy failed with after target restore for source: {} to target: {} snapshot1: {} snapshot2: "
+                  + "{} params: {}. Return code: {}", srcPaths, dst, snap1, snap2, params, returnCode);
+              return false;
+            }
+          }
+        }
+      }
+    } catch (Exception e) {
+      throw new IOException("Cannot execute DistCp process: " + e, e);

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 592937)
    Time Spent: 3.5h  (was: 3h 20m)

> Add support for Snapshots during external table replication
> -----------------------------------------------------------
>
>                 Key: HIVE-24852
>                 URL: https://issues.apache.org/jira/browse/HIVE-24852
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Ayush Saxena
>            Assignee: Ayush Saxena
>            Priority: Critical
>              Labels: pull-request-available
>         Attachments: Design Doc HDFS Snapshots for External Table Replication-01.pdf
>
>          Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Add support for use of snapshot diff for external table replication.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)