You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/07/26 05:22:23 UTC

[GitHub] [hive] hmangla98 opened a new pull request #2529: HIVE-24946: Handle failover case during repl load

hmangla98 opened a new pull request #2529:
URL: https://github.com/apache/hive/pull/2529


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678752527



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);
+    Map<String, String> dbParameters = db.getParameters();
+    return Failover_Point.SOURCE.toString().equalsIgnoreCase(dbParameters.get(ReplConst.REPL_FAILOVER_ENABLED));

Review comment:
       Refractored.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678750819



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -204,10 +207,11 @@ private void testTargetDbReplIncompatible(boolean setReplIncompProp) throws Thro
   }
 
   @Test
-  public void testFailoverDuringDump() throws Throwable {
+  public void testCompleteFailoverWithReverseBootstrap() throws Throwable {
     HiveConf primaryConf = primary.getConf();
     TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
     List<String> failoverConfigs = Arrays.asList("'" + HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+    Database db;

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pkumarsinha commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r680692744



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -723,6 +734,75 @@ public void testFailoverFailureInReverseReplication() throws Throwable {
     assertTrue(fs.exists(new Path(dumpPath, LOAD_ACKNOWLEDGEMENT.toString())));
   }
 
+  @Test
+  public void testFailoverFailoverAndRollback() throws Throwable {

Review comment:
       typo in name?

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -723,6 +734,75 @@ public void testFailoverFailureInReverseReplication() throws Throwable {
     assertTrue(fs.exists(new Path(dumpPath, LOAD_ACKNOWLEDGEMENT.toString())));
   }
 
+  @Test
+  public void testFailoverFailoverAndRollback() throws Throwable {
+    List<String> failoverConfigs = Arrays.asList("'" + HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'",
+            "'" + HiveConf.ConfVars.REPL_RETAIN_PREV_DUMP_DIR + "'='true'",
+            "'" + HiveConf.ConfVars.REPL_RETAIN_PREV_DUMP_DIR_COUNT + "'='1'");
+    WarehouseInstance.Tuple dumpData = primary.run("use " + primaryDbName)
+            .run("create table t1 (id int) clustered by(id) into 3 buckets stored as orc " +
+                    "tblproperties (\"transactional\"=\"true\")")
+            .run("create table t2 (rank int) partitioned by (name string) tblproperties(\"transactional\"=\"true\", " +
+                    "\"transactional_properties\"=\"insert_only\")")
+            .dump(primaryDbName, failoverConfigs);
+
+    //This dump is not failover ready as target db can be used for replication only after first incremental load.
+    FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+    Path dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    assertFalse(fs.exists(new Path(dumpPath, FailoverMetaData.FAILOVER_METADATA)));
+    assertFalse(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
+    assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+    replica.load(replicatedDbName, primaryDbName)
+            .run("use " + replicatedDbName)
+            .run("show tables")
+            .verifyResults(new String[]{"t1", "t2"})
+            .run("repl status " + replicatedDbName)
+            .verifyResult(dumpData.lastReplicationId);
+
+    assertTrue(MetaStoreUtils.isTargetOfReplication(replica.getDatabase(replicatedDbName)));
+    dumpData = primary.run("use " + primaryDbName)
+            .run("insert into t1 values(1)")
+            .run("insert into t2 partition(name='Bob') values(11)")
+            .dump(primaryDbName, failoverConfigs);
+
+    dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    Path dumpAckFile = new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString());
+    assertTrue(fs.exists(dumpAckFile));
+    assertTrue(fs.exists(new Path(dumpPath, FailoverMetaData.FAILOVER_METADATA)));
+    assertTrue(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
+    assertTrue(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(primary.getDatabase(primaryDbName),
+            MetaStoreUtils.FailoverEndpoint.SOURCE));
+
+    fs.delete(dumpAckFile, false);
+
+    dumpData = primary.run("use " + primaryDbName)
+            .run("insert into t2 partition(name='Carl') values(10)")
+            .run("create table t3 (id int)")
+            .run("insert into t3 values (2)")
+            .dump(primaryDbName);
+
+    dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    assertTrue(fs.exists(new Path(dumpPath, FailoverMetaData.FAILOVER_METADATA)));
+    assertFalse(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
+    assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString())));
+    assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+    replica.load(replicatedDbName, primaryDbName)
+            .run("use " + replicatedDbName)
+            .run("show tables")
+            .verifyResults(new String[]{"t1", "t2", "t3"})
+            .run("repl status " + replicatedDbName)
+            .verifyResult(dumpData.lastReplicationId)
+            .run("select id from t1")
+            .verifyResults(new String[]{"1"})
+            .run("select rank from t2 order by rank")
+            .verifyResults(new String[]{"11", "10"})
+            .run("select id from t3")
+            .verifyResults(new String[]{"2"});
+    assertTrue(fs.exists(new Path(dumpPath, LOAD_ACKNOWLEDGEMENT.toString())));

Review comment:
       assert for isDbBeingFailedOver




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pkumarsinha commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r677196856



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -554,6 +554,24 @@ public void run() throws SemanticException {
           }
         }
       });
+      if (work.shouldFailover()) {
+        listOfPreAckTasks.add(new PreAckTask() {
+          @Override
+          public void run() throws SemanticException {
+            try {
+              Database db = getHive().getDatabase(work.getTargetDatabase());
+              if (!MetaStoreUtils.isTargetOfReplication(db)) {
+                return;
+              }
+              LOG.info("Removing property: {} from database: {}", ReplConst.TARGET_OF_REPLICATION, db.getName());
+              db.getParameters().remove(ReplConst.TARGET_OF_REPLICATION);

Review comment:
       If you remove the property TARGET_OF_REPLICATION at this point of time which will allow the background threads like  Partition Management/ Stat Updater etc. Can we still rollback in that case? Wondering if we should delay the removal of this property until first dump in reverse direction?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pkumarsinha commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r680517039



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -554,6 +554,24 @@ public void run() throws SemanticException {
           }
         }
       });
+      if (work.shouldFailover()) {
+        listOfPreAckTasks.add(new PreAckTask() {
+          @Override
+          public void run() throws SemanticException {
+            try {
+              Database db = getHive().getDatabase(work.getTargetDatabase());
+              if (!MetaStoreUtils.isTargetOfReplication(db)) {
+                return;
+              }
+              LOG.info("Removing property: {} from database: {}", ReplConst.TARGET_OF_REPLICATION, db.getName());
+              db.getParameters().remove(ReplConst.TARGET_OF_REPLICATION);

Review comment:
       We can remove this property as the first dump in reverse direction




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pkumarsinha merged pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
pkumarsinha merged pull request #2529:
URL: https://github.com/apache/hive/pull/2529


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678826183



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -308,6 +316,9 @@ public void testFailoverDuringDump() throws Throwable {
             .run("select rank from t2 order by rank")
             .verifyResults(new String[]{"10", "11"});
 
+    db = replica.getDatabase(replicatedDbName);
+    assertFalse(MetaStoreUtils.isTargetOfReplication(db));

Review comment:
       repl.failover.endpoint db prop would restrict backgrounds threads to run for such db if rollback is initiated.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678751529



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -554,6 +555,29 @@ public void run() throws SemanticException {
           }
         }
       });
+      if (work.shouldFailover()) {
+        listOfPreAckTasks.add(new PreAckTask() {
+          @Override
+          public void run() throws SemanticException {
+            try {
+              Database db = getHive().getDatabase(work.getTargetDatabase());
+              Map<String, String> params = db.getParameters();
+              if (params == null) {
+                params = new HashMap<>();
+                db.setParameters(params);
+              } else if (MetaStoreUtils.isTargetOfReplication(db)) {
+                params.remove(ReplConst.TARGET_OF_REPLICATION);

Review comment:
       In rollback, this prop would be again set in incremental load.

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java
##########
@@ -129,16 +130,20 @@ public ReplLoadWork(HiveConf hiveConf, String dumpDirectory,
       if (metricCollector != null) {
         metricCollector.setMetricsMBean(name);
       }
+      Path failoverReadyMarker = new Path(dumpDirectory, ReplAck.FAILOVER_READY_MARKER.toString());
+      FileSystem fs = failoverReadyMarker.getFileSystem(hiveConf);
+      shouldFailover = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_REPL_FAILOVER_START)
+              && fs.exists(failoverReadyMarker);
       incrementalLoadTasksBuilder = new IncrementalLoadTasksBuilder(dbNameToLoadIn, dumpDirectory,
           new IncrementalLoadEventsIterator(dumpDirectory, hiveConf), hiveConf, eventTo, metricCollector,
-          replStatsTracker);
+          replStatsTracker, shouldFailover);
 
       /*
        * If the current incremental dump also includes bootstrap for some tables, then create iterator
        * for the same.
        */
       Path incBootstrapDir = new Path(dumpDirectory, ReplUtils.INC_BOOTSTRAP_ROOT_DIR_NAME);
-      FileSystem fs = incBootstrapDir.getFileSystem(hiveConf);
+      fs = incBootstrapDir.getFileSystem(hiveConf);

Review comment:
       Removed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r677297935



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -554,6 +554,24 @@ public void run() throws SemanticException {
           }
         }
       });
+      if (work.shouldFailover()) {
+        listOfPreAckTasks.add(new PreAckTask() {
+          @Override
+          public void run() throws SemanticException {
+            try {
+              Database db = getHive().getDatabase(work.getTargetDatabase());
+              if (!MetaStoreUtils.isTargetOfReplication(db)) {
+                return;
+              }
+              LOG.info("Removing property: {} from database: {}", ReplConst.TARGET_OF_REPLICATION, db.getName());
+              db.getParameters().remove(ReplConst.TARGET_OF_REPLICATION);

Review comment:
       Yes, rollback may be impacted. But we can't delay this removal until first dump also. As dump operation will itself fail if db is target of replication.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678751441



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -655,17 +679,25 @@ private int executeIncrementalLoad(long loadStartTime) throws Exception {
     if (work.replScopeModified) {
       dropTablesExcludedInReplScope(work.currentReplScope);
     }
-    if (!MetaStoreUtils.isTargetOfReplication(getHive().getDatabase(work.dbNameToLoadIn))) {
+    if (!work.shouldFailover()) {
+      Database targetDb = getHive().getDatabase(work.dbNameToLoadIn);
       Map<String, String> props = new HashMap<>();
-      props.put(ReplConst.TARGET_OF_REPLICATION, "true");
-      AlterDatabaseSetPropertiesDesc setTargetDesc = new AlterDatabaseSetPropertiesDesc(work.dbNameToLoadIn, props, null);
-      Task<?> addReplTargetPropTask =
-              TaskFactory.get(new DDLWork(new HashSet<>(), new HashSet<>(), setTargetDesc, true,
-                      work.dumpDirectory, work.getMetricCollector()), conf);
-      if (this.childTasks == null) {
-        this.childTasks = new ArrayList<>();
+      if (!MetaStoreUtils.isTargetOfReplication(targetDb)) {
+        props.put(ReplConst.TARGET_OF_REPLICATION, ReplConst.TRUE);
+      }
+      if (ReplUtils.isDbBeingFailedOverAtTarget(targetDb)) {
+        props.put(ReplConst.REPL_FAILOVER_ENABLED, "");

Review comment:
       Yes




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678757046



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);

Review comment:
       For repl dump * , execution would not come here ever.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678750058



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -319,31 +330,92 @@ public void testFailoverDuringDump() throws Throwable {
     assertTrue(fs.exists(new Path(dumpPath, ReplAck.LOAD_ACKNOWLEDGEMENT.toString())));
 
     assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
-    dumpData = primary.dump(primaryDbName);
-    dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
-    Assert.assertEquals(new DumpMetaData(dumpPath, conf).getDumpType(), DumpType.INCREMENTAL);
-    Path failoverReadyFile = new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString());
-    Path failoverMdFile = new Path(dumpPath, FailoverMetaData.FAILOVER_METADATA);
-    assertFalse(fs.exists(failoverReadyFile));
-    assertFalse(fs.exists(failoverMdFile));
-    assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
-    replica.load(replicatedDbName, primaryDbName);
 
-    fs.create(failoverReadyFile);
-    fs.create(failoverMdFile);
-    assertTrue(fs.exists(failoverReadyFile));
-    assertTrue(fs.exists(failoverMdFile));
+    primary.run("drop database if exists " + primaryDbName + " cascade");
 
-    //Since the failover start config is disabled and previous valid dump directory contains _failover_ready marker file
-    //So, this dump iteration will perform bootstrap dump instead of incremental and last dump directory also should not
-    //deleted.
-    WarehouseInstance.Tuple newDumpData = primary.dump(primaryDbName);
-    assertNotEquals(newDumpData.dumpLocation, dumpData.dumpLocation);
+    assertTrue(primary.getDatabase(primaryDbName) == null);
+
+    assertTrue(ReplChangeManager.getReplPolicyIdString(replica.getDatabase(replicatedDbName)) == null);
+    WarehouseInstance.Tuple reverseDumpData = replica.dump(replicatedDbName);
+    assertNotEquals(reverseDumpData.dumpLocation, dumpData.dumpLocation);
     assertTrue(fs.exists(dumpPath));
     assertTrue(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
-    dumpPath = new Path(newDumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    dumpPath = new Path(reverseDumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
     assertFalse(fs.exists(new Path(dumpPath, FailoverMetaData.FAILOVER_METADATA)));
     assertTrue(new DumpMetaData(dumpPath, conf).getDumpType() == DumpType.BOOTSTRAP);
+    assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString())));
+    assertTrue(ReplUtils.isDbBeingFailedOverAtTarget(replica.getDatabase(replicatedDbName)));

Review comment:
       Already done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678751309



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -308,6 +316,9 @@ public void testFailoverDuringDump() throws Throwable {
             .run("select rank from t2 order by rank")
             .verifyResults(new String[]{"10", "11"});
 
+    db = replica.getDatabase(replicatedDbName);
+    assertFalse(MetaStoreUtils.isTargetOfReplication(db));

Review comment:
       In case of rollback, repl.targer.for would be again set in incremental load.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pkumarsinha commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678808131



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -308,6 +316,9 @@ public void testFailoverDuringDump() throws Throwable {
             .run("select rank from t2 order by rank")
             .verifyResults(new String[]{"10", "11"});
 
+    db = replica.getDatabase(replicatedDbName);
+    assertFalse(MetaStoreUtils.isTargetOfReplication(db));

Review comment:
       It might take a while before rollback can kick in, meanwhile if background threads run, wouldn't they mess up the state.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pkumarsinha commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678577106



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -155,6 +156,10 @@
     LOAD_NEW, LOAD_SKIP, LOAD_REPLACE
   }
 
+  public static enum Failover_Point {

Review comment:
       add javadoc comment for what this class is for

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -155,6 +156,10 @@
     LOAD_NEW, LOAD_SKIP, LOAD_REPLACE
   }
 
+  public static enum Failover_Point {

Review comment:
       nit: Rename to FailoverEndpoint

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);
+    Map<String, String> dbParameters = db.getParameters();
+    return Failover_Point.SOURCE.toString().equalsIgnoreCase(dbParameters.get(ReplConst.REPL_FAILOVER_ENABLED));

Review comment:
       db.getParameters() is Nullable, don't you require a null check like you did in other places?

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
##########
@@ -172,7 +179,7 @@ public IncrementalLoadTasksBuilder(String dbName, String loadPath, IncrementalLo
 
       Map<String, String> dbProps = new HashMap<>();
       dbProps.put(ReplicationSpec.KEY.CURR_STATE_ID.toString(), String.valueOf(lastReplayedEvent));
-      ReplStateLogWork replStateLogWork = new ReplStateLogWork(replLogger, dbProps, dumpDirectory, metricCollector);
+      ReplStateLogWork replStateLogWork = new ReplStateLogWork(replLogger, dbProps, dumpDirectory, metricCollector, shouldFailover);

Review comment:
       nit: format

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -655,17 +679,25 @@ private int executeIncrementalLoad(long loadStartTime) throws Exception {
     if (work.replScopeModified) {
       dropTablesExcludedInReplScope(work.currentReplScope);
     }
-    if (!MetaStoreUtils.isTargetOfReplication(getHive().getDatabase(work.dbNameToLoadIn))) {
+    if (!work.shouldFailover()) {
+      Database targetDb = getHive().getDatabase(work.dbNameToLoadIn);
       Map<String, String> props = new HashMap<>();
-      props.put(ReplConst.TARGET_OF_REPLICATION, "true");
-      AlterDatabaseSetPropertiesDesc setTargetDesc = new AlterDatabaseSetPropertiesDesc(work.dbNameToLoadIn, props, null);
-      Task<?> addReplTargetPropTask =
-              TaskFactory.get(new DDLWork(new HashSet<>(), new HashSet<>(), setTargetDesc, true,
-                      work.dumpDirectory, work.getMetricCollector()), conf);
-      if (this.childTasks == null) {
-        this.childTasks = new ArrayList<>();
+      if (!MetaStoreUtils.isTargetOfReplication(targetDb)) {
+        props.put(ReplConst.TARGET_OF_REPLICATION, ReplConst.TRUE);
+      }
+      if (ReplUtils.isDbBeingFailedOverAtTarget(targetDb)) {
+        props.put(ReplConst.REPL_FAILOVER_ENABLED, "");

Review comment:
       Is this a rollback use case?

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -204,10 +207,11 @@ private void testTargetDbReplIncompatible(boolean setReplIncompProp) throws Thro
   }
 
   @Test
-  public void testFailoverDuringDump() throws Throwable {
+  public void testCompleteFailoverWithReverseBootstrap() throws Throwable {
     HiveConf primaryConf = primary.getConf();
     TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
     List<String> failoverConfigs = Arrays.asList("'" + HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+    Database db;

Review comment:
       nit:Move this down to where you need it

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);
+    Map<String, String> dbParameters = db.getParameters();
+    return Failover_Point.SOURCE.toString().equalsIgnoreCase(dbParameters.get(ReplConst.REPL_FAILOVER_ENABLED));

Review comment:
       Also, ReplConst.REPL_FAILOVER_ENABLED -> ReplConst.REPL_FAILOVER_ENDPOINT
   Does this make sense?

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -308,6 +316,9 @@ public void testFailoverDuringDump() throws Throwable {
             .run("select rank from t2 order by rank")
             .verifyResults(new String[]{"10", "11"});
 
+    db = replica.getDatabase(replicatedDbName);
+    assertFalse(MetaStoreUtils.isTargetOfReplication(db));

Review comment:
       Wouldn't this mean that there will not be any rollback?

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);

Review comment:
       What do you gain by this assertion that would result in Error, rather NPE will be thrown

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadWork.java
##########
@@ -129,16 +130,20 @@ public ReplLoadWork(HiveConf hiveConf, String dumpDirectory,
       if (metricCollector != null) {
         metricCollector.setMetricsMBean(name);
       }
+      Path failoverReadyMarker = new Path(dumpDirectory, ReplAck.FAILOVER_READY_MARKER.toString());
+      FileSystem fs = failoverReadyMarker.getFileSystem(hiveConf);
+      shouldFailover = hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_REPL_FAILOVER_START)
+              && fs.exists(failoverReadyMarker);
       incrementalLoadTasksBuilder = new IncrementalLoadTasksBuilder(dbNameToLoadIn, dumpDirectory,
           new IncrementalLoadEventsIterator(dumpDirectory, hiveConf), hiveConf, eventTo, metricCollector,
-          replStatsTracker);
+          replStatsTracker, shouldFailover);
 
       /*
        * If the current incremental dump also includes bootstrap for some tables, then create iterator
        * for the same.
        */
       Path incBootstrapDir = new Path(dumpDirectory, ReplUtils.INC_BOOTSTRAP_ROOT_DIR_NAME);
-      FileSystem fs = incBootstrapDir.getFileSystem(hiveConf);
+      fs = incBootstrapDir.getFileSystem(hiveConf);

Review comment:
       Why do you need this? You already have fs

##########
File path: standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
##########
@@ -249,7 +249,7 @@ public static boolean isDbReplIncompatible(Database db) {
   public static boolean isDbBeingFailedOver(Database db) {
     assert (db != null);
     Map<String, String> dbParameters = db.getParameters();
-    return dbParameters != null && ReplConst.TRUE.equalsIgnoreCase(dbParameters.get(ReplConst.REPL_FAILOVER_ENABLED));
+    return dbParameters != null && !StringUtils.isEmpty(dbParameters.get(ReplConst.REPL_FAILOVER_ENABLED));

Review comment:
       It can't be any random string, it has to be a valid Failover_Point (i.e FailoverEndpoint as per the other comment)

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadTask.java
##########
@@ -554,6 +555,29 @@ public void run() throws SemanticException {
           }
         }
       });
+      if (work.shouldFailover()) {
+        listOfPreAckTasks.add(new PreAckTask() {
+          @Override
+          public void run() throws SemanticException {
+            try {
+              Database db = getHive().getDatabase(work.getTargetDatabase());
+              Map<String, String> params = db.getParameters();
+              if (params == null) {
+                params = new HashMap<>();
+                db.setParameters(params);
+              } else if (MetaStoreUtils.isTargetOfReplication(db)) {
+                params.remove(ReplConst.TARGET_OF_REPLICATION);

Review comment:
       Will this not break the rollback logic?

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);

Review comment:
       Does this hold good for 'REPL DUMP *' use case?

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -319,31 +330,92 @@ public void testFailoverDuringDump() throws Throwable {
     assertTrue(fs.exists(new Path(dumpPath, ReplAck.LOAD_ACKNOWLEDGEMENT.toString())));
 
     assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
-    dumpData = primary.dump(primaryDbName);
-    dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
-    Assert.assertEquals(new DumpMetaData(dumpPath, conf).getDumpType(), DumpType.INCREMENTAL);
-    Path failoverReadyFile = new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString());
-    Path failoverMdFile = new Path(dumpPath, FailoverMetaData.FAILOVER_METADATA);
-    assertFalse(fs.exists(failoverReadyFile));
-    assertFalse(fs.exists(failoverMdFile));
-    assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
-    replica.load(replicatedDbName, primaryDbName);
 
-    fs.create(failoverReadyFile);
-    fs.create(failoverMdFile);
-    assertTrue(fs.exists(failoverReadyFile));
-    assertTrue(fs.exists(failoverMdFile));
+    primary.run("drop database if exists " + primaryDbName + " cascade");
 
-    //Since the failover start config is disabled and previous valid dump directory contains _failover_ready marker file
-    //So, this dump iteration will perform bootstrap dump instead of incremental and last dump directory also should not
-    //deleted.
-    WarehouseInstance.Tuple newDumpData = primary.dump(primaryDbName);
-    assertNotEquals(newDumpData.dumpLocation, dumpData.dumpLocation);
+    assertTrue(primary.getDatabase(primaryDbName) == null);
+
+    assertTrue(ReplChangeManager.getReplPolicyIdString(replica.getDatabase(replicatedDbName)) == null);
+    WarehouseInstance.Tuple reverseDumpData = replica.dump(replicatedDbName);
+    assertNotEquals(reverseDumpData.dumpLocation, dumpData.dumpLocation);
     assertTrue(fs.exists(dumpPath));
     assertTrue(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
-    dumpPath = new Path(newDumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    dumpPath = new Path(reverseDumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
     assertFalse(fs.exists(new Path(dumpPath, FailoverMetaData.FAILOVER_METADATA)));
     assertTrue(new DumpMetaData(dumpPath, conf).getDumpType() == DumpType.BOOTSTRAP);
+    assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString())));
+    assertTrue(ReplUtils.isDbBeingFailedOverAtTarget(replica.getDatabase(replicatedDbName)));

Review comment:
       Assert at this point that the _dumpmetadat has marked the dump as bootstrap one 

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
##########
@@ -80,9 +79,11 @@ public ReplLogger getReplLogger() {
   private final Long eventTo;
   private String dumpDirectory;
   private final ReplicationMetricCollector metricCollector;
+  private boolean shouldFailover;
 
   public IncrementalLoadTasksBuilder(String dbName, String loadPath, IncrementalLoadEventsIterator iterator,
-      HiveConf conf, Long eventTo, ReplicationMetricCollector metricCollector, ReplStatsTracker replStatsTracker)
+      HiveConf conf, Long eventTo, ReplicationMetricCollector metricCollector, ReplStatsTracker replStatsTracker,

Review comment:
       nit: format this

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplStateLogWork.java
##########
@@ -115,19 +116,13 @@ public ReplStateLogWork(ReplLogger replLogger, String functionName, String dumpD
     this.metricCollector = metricCollector;
   }
 
-  public ReplStateLogWork(ReplLogger replLogger, Map<String, String> dbProps, ReplicationMetricCollector collector) {
-    this.logType = LOG_TYPE.END;
-    this.replLogger = replLogger;
-    this.lastReplId = ReplicationSpec.getLastReplicatedStateFromParameters(dbProps);
-    this.metricCollector = collector;
-  }
-
-  public ReplStateLogWork(ReplLogger replLogger, Map<String, String> dbProps, String dumpDirectory, ReplicationMetricCollector collector) {
+  public ReplStateLogWork(ReplLogger replLogger, Map<String, String> dbProps, String dumpDirectory, ReplicationMetricCollector collector, boolean shouldFailover) {

Review comment:
       nit: format this

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);
+    Map<String, String> dbParameters = db.getParameters();
+    return Failover_Point.SOURCE.toString().equalsIgnoreCase(dbParameters.get(ReplConst.REPL_FAILOVER_ENABLED));

Review comment:
       May be we can just have a single method like this?
   public static boolean isDbBeingFailedOverAtEndpoint(Database db, Failover_Point endPoint) {
       assert (db != null);
       Map<String, String> dbParameters = db.getParameters();
       return endPoint.toString().equalsIgnoreCase(dbParameters.get(ReplConst.REPL_FAILOVER_ENABLED));
     }




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] hmangla98 commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
hmangla98 commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r678760787



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##########
@@ -219,16 +224,16 @@
     return TaskFactory.get(replLogWork, conf);
   }
 
+  public static boolean isDbBeingFailedOverAtSource(Database db) {
+    assert (db != null);

Review comment:
       Removed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] pkumarsinha commented on a change in pull request #2529: HIVE-24946: Handle failover case during repl load

Posted by GitBox <gi...@apache.org>.
pkumarsinha commented on a change in pull request #2529:
URL: https://github.com/apache/hive/pull/2529#discussion_r680520961



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -592,7 +577,8 @@ public void testFailoverDuringDumpWithPreviousFailed() throws Throwable {
     fs.delete(dumpAckFile, false);
 
     dumpData = primary.run("use " + primaryDbName)
-            .run("insert into t2 partition(name='Carl') values(10)")
+            .run("create table t3 (id int)")
+            .run("insert into t3 values (2)")

Review comment:
       insert into existing table as well. The table which has already part of repl and assert that the change is not there

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -621,6 +607,72 @@ public void testFailoverDuringDumpWithPreviousFailed() throws Throwable {
     assertTrue(MetaStoreUtils.isTargetOfReplication(db));
     assertTrue(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(db, MetaStoreUtils.FailoverEndpoint.TARGET));
     assertTrue(fs.exists(new Path(dumpPath, ReplAck.LOAD_ACKNOWLEDGEMENT.toString())));
+
+    primary.run("drop database if exists " + primaryDbName + " cascade");
+
+    WarehouseInstance.Tuple reverseDumpData = replica.run("use " + replicatedDbName)
+            .run("create table t3 (id int)")
+            .run("insert into t3 values (2)")
+            .dump(replicatedDbName, retainPrevDumpDir);

Review comment:
       Add a test where hive.repl.retain.prev.dump.dir=true with retain count being 1

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -621,6 +607,72 @@ public void testFailoverDuringDumpWithPreviousFailed() throws Throwable {
     assertTrue(MetaStoreUtils.isTargetOfReplication(db));
     assertTrue(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(db, MetaStoreUtils.FailoverEndpoint.TARGET));
     assertTrue(fs.exists(new Path(dumpPath, ReplAck.LOAD_ACKNOWLEDGEMENT.toString())));
+
+    primary.run("drop database if exists " + primaryDbName + " cascade");
+
+    WarehouseInstance.Tuple reverseDumpData = replica.run("use " + replicatedDbName)
+            .run("create table t3 (id int)")
+            .run("insert into t3 values (2)")

Review comment:
       insert different value as the last attempted one. e.g 10 or anything other than 2

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##########
@@ -621,6 +607,72 @@ public void testFailoverDuringDumpWithPreviousFailed() throws Throwable {
     assertTrue(MetaStoreUtils.isTargetOfReplication(db));
     assertTrue(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(db, MetaStoreUtils.FailoverEndpoint.TARGET));
     assertTrue(fs.exists(new Path(dumpPath, ReplAck.LOAD_ACKNOWLEDGEMENT.toString())));
+
+    primary.run("drop database if exists " + primaryDbName + " cascade");
+
+    WarehouseInstance.Tuple reverseDumpData = replica.run("use " + replicatedDbName)
+            .run("create table t3 (id int)")
+            .run("insert into t3 values (2)")
+            .dump(replicatedDbName, retainPrevDumpDir);
+    assertNotEquals(reverseDumpData.dumpLocation, dumpData.dumpLocation);
+    assertTrue(fs.exists(dumpPath));
+    assertTrue(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
+    dumpPath = new Path(reverseDumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    dumpAckFile = new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString());
+    assertTrue(fs.exists(dumpAckFile));
+    assertFalse(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
+    assertTrue(new DumpMetaData(dumpPath, conf).getDumpType() == DumpType.BOOTSTRAP);
+    db = replica.getDatabase(replicatedDbName);
+    assertTrue(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(db, MetaStoreUtils.FailoverEndpoint.TARGET));
+    assertFalse(MetaStoreUtils.isTargetOfReplication(db));
+
+    fs.delete(dumpAckFile, false);
+    assertFalse(fs.exists(dumpAckFile));
+    dumpData = replica.dump(replicatedDbName, retainPrevDumpDir);
+    assertEquals(reverseDumpData.dumpLocation, dumpData.dumpLocation);
+    assertFalse(fs.exists(new Path(dumpPath, ReplAck.FAILOVER_READY_MARKER.toString())));
+    assertTrue(new DumpMetaData(dumpPath, conf).getDumpType() == DumpType.BOOTSTRAP);
+    assertTrue(fs.exists(dumpAckFile));
+    db = replica.getDatabase(replicatedDbName);
+    assertTrue(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(db, MetaStoreUtils.FailoverEndpoint.TARGET));
+    assertFalse(MetaStoreUtils.isTargetOfReplication(db));
+
+    primary.load(primaryDbName, replicatedDbName)
+            .run("use " + primaryDbName)
+            .run("show tables")
+            .verifyResults(new String[]{"t1", "t2", "t3"})
+            .run("repl status " + primaryDbName)
+            .verifyResult(dumpData.lastReplicationId)
+            .run("select id from t1")
+            .verifyResults(new String[]{"1"})
+            .run("select rank from t2 order by rank")
+            .verifyResults(new String[]{"11"})
+            .run("select id from t3")
+            .verifyResults(new String[]{"2"});
+
+    assertTrue(fs.exists(new Path(dumpPath, LOAD_ACKNOWLEDGEMENT.toString())));
+
+    reverseDumpData = replica.run("insert into t3 values (3)")
+            .dump(replicatedDbName, retainPrevDumpDir);
+    dumpPath = new Path(reverseDumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    dumpAckFile = new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString());
+    assertFalse(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(replica.getDatabase(replicatedDbName),
+            MetaStoreUtils.FailoverEndpoint.TARGET));
+    fs.delete(dumpAckFile);
+    replica.run("ALTER DATABASE " + replicatedDbName + " SET DBPROPERTIES('" + ReplConst.REPL_FAILOVER_ENDPOINT + "'='"
+            + MetaStoreUtils.FailoverEndpoint.TARGET + "')");
+    assertTrue(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(replica.getDatabase(replicatedDbName),
+            MetaStoreUtils.FailoverEndpoint.TARGET));
+    assertFalse(fs.exists(dumpAckFile));
+    assertEquals(reverseDumpData.dumpLocation, replica.dump(replicatedDbName, retainPrevDumpDir).dumpLocation);
+    fs.exists(dumpAckFile);
+    assertFalse(MetaStoreUtils.isDbBeingFailedOverAtEndpoint(replica.getDatabase(replicatedDbName),
+            MetaStoreUtils.FailoverEndpoint.TARGET));
+
+    primary.load(primaryDbName, replicatedDbName)
+            .run("select id from t3")
+            .verifyResults(new String[]{"2", "3"});;
+    assertTrue(fs.exists(new Path(dumpPath, LOAD_ACKNOWLEDGEMENT.toString())));

Review comment:
       Can you please handle different failure use case in different tests.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org