You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/12/13 07:57:00 UTC
[jira] [Work logged] (HIVE-25397) Snapshot support for controlled failover
[ https://issues.apache.org/jira/browse/HIVE-25397?focusedWorklogId=694887&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-694887 ]
ASF GitHub Bot logged work on HIVE-25397:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Dec/21 07:56
Start Date: 13/Dec/21 07:56
Worklog Time Spent: 10m
Work Description: ArkoSharma commented on a change in pull request #2539:
URL: https://github.com/apache/hive/pull/2539#discussion_r767480983
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -695,6 +695,10 @@ private static void populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
+ "data copy, the target data is overwritten and the modifications are removed and the copy is again "
+ "attempted using the snapshot based approach. If disabled, the replication will fail in case the target is "
+ "modified."),
+ REPL_REUSE_SNAPSHOTS("hive.repl.reuse.snapshots", false,
+ "If enabled,reusing snapshots is attempted in case of controlled failover(B->A) when same paths are"
+ + "used for external table replication on src and target. Also in cases of failed incremental where re-bootstrap is required."
+ + "If set to true and snapshots exist in some paths, it creates/reuses new snapshots in those paths using the same name as exisiting snapshots."),
Review comment:
Corrected
##########
File path: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
##########
@@ -1237,25 +1238,42 @@ public boolean runDistCpWithSnapshots(String oldSnapshot, String newSnapshot, Li
LOG.warn("Copy failed with INVALID_ARGUMENT for source: {} to target: {} snapshot1: {} snapshot2: {} "
+ "params: {}", srcPaths, dst, oldSnapshot, newSnapshot, params);
return true;
- } else if (returnCode == DistCpConstants.UNKNOWN_ERROR && overwriteTarget) {
+ } else if (returnCode == DistCpConstants.UNKNOWN_ERROR) {
// Check if this error is due to target modified.
- if (shouldRdiff(dst, conf, oldSnapshot, overwriteTarget)) {
- LOG.warn("Copy failed due to target modified. Attempting to restore back the target. source: {} target: {} "
- + "snapshot: {}", srcPaths, dst, oldSnapshot);
- List<String> rParams = constructDistCpWithSnapshotParams(srcPaths, dst, ".", oldSnapshot, conf, "-rdiff");
- DistCp rDistcp = new DistCp(conf, null);
- returnCode = rDistcp.run(rParams.toArray(new String[0]));
- if (returnCode == 0) {
- LOG.info("Target restored to previous state. source: {} target: {} snapshot: {}. Reattempting to copy.",
- srcPaths, dst, oldSnapshot);
- dst.getFileSystem(conf).deleteSnapshot(dst, oldSnapshot);
- dst.getFileSystem(conf).createSnapshot(dst, oldSnapshot);
- returnCode = distcp.run(params.toArray(new String[0]));
+ if (targetModified(dst, conf, oldSnapshot)) {
+ if (overwriteTarget) {
+ LOG.warn("Copy failed due to target modified. Attempting to restore back the target. source: {} target: {} "
+ + "snapshot: {}", srcPaths, dst, oldSnapshot);
+ List<String> rParams = constructDistCpWithSnapshotParams(srcPaths, dst, ".", oldSnapshot, conf, "-rdiff");
+ DistCp rDistcp = new DistCp(conf, null);
+ returnCode = rDistcp.run(rParams.toArray(new String[0]));
if (returnCode == 0) {
+ LOG.info("Target restored to previous state. source: {} target: {} snapshot: {}. Reattempting to copy.",
+ srcPaths, dst, oldSnapshot);
+ dst.getFileSystem(conf).deleteSnapshot(dst, oldSnapshot);
+ dst.getFileSystem(conf).createSnapshot(dst, oldSnapshot);
+ returnCode = distcp.run(params.toArray(new String[0]));
+ if (returnCode == 0) {
+ return true;
+ } else {
+ LOG.error("Copy failed with after target restore for source: {} to target: {} snapshot1: {} snapshot2: "
+ + "{} params: {}. Return code: {}", srcPaths, dst, oldSnapshot, newSnapshot, params, returnCode);
+ return false;
+ }
+ }
+ } else {
+ //in case overwriteTarget is false, and we encounter an exception due to targetFs getting modified
+ // since last snapshot, then fall back to full distcp
+ LOG.warn("Copy failed due to target modified and overwrite is false. Attempting full distcp." +
+ "Source:{}, target: {}",srcPaths, dst);
+ // Get the path relative to the initial snapshot for copy.
+ Path snapRelPath = new Path(srcPaths.get(0), HdfsConstants.DOT_SNAPSHOT_DIR + "/" + newSnapshot);
+ // Copy from the initial snapshot path.
+ if(runDistCp(Collections.singletonList(snapRelPath), dst, conf)) {
return true;
} else {
- LOG.error("Copy failed with after target restore for source: {} to target: {} snapshot1: {} snapshot2: "
- + "{} params: {}. Return code: {}", srcPaths, dst, oldSnapshot, newSnapshot, params, returnCode);
+ LOG.error("Copy failed with full distcp for source: {} to target: {} snapshot:"
+ + "{} params: {}. Return code: {}", srcPaths, dst, newSnapshot, returnCode);
Review comment:
Corrected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 694887)
Time Spent: 4h 40m (was: 4.5h)
> Snapshot support for controlled failover
> ----------------------------------------
>
> Key: HIVE-25397
> URL: https://issues.apache.org/jira/browse/HIVE-25397
> Project: Hive
> Issue Type: Bug
> Reporter: Arko Sharma
> Assignee: Arko Sharma
> Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> In case the same locations are used for external tables on the source and target, then the snapshots created during replication can be re-used during reverse replication. This patch enables re-using the snapshots during reverse replication using a configuration.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)