[GitHub] [hive] aasha commented on a change in pull request #1898: HIVE-24675: Handle external table replication for HA with same NS and…

[GitBox <gi...@apache.org>]

aasha commented on a change in pull request #1898:
URL: https://github.com/apache/hive/pull/1898#discussion_r563482079



##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##########
@@ -1707,6 +1738,50 @@ public void testHdfsNameserviceLazyCopyIncr() throws Throwable {
     }
   }
 
+  @Test
+  public void testHdfsNSLazyCopyIncrExtTbls() throws Throwable {

Review comment:
       both tests can be combined?

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##########
@@ -2122,4 +2221,21 @@ private void setupUDFJarOnHDFS(Path identityUdfLocalPath, Path identityUdfHdfsPa
             + NS_REMOTE + "'");
     return withClause;
   }
+
+  /*
+   * Method used from TestReplicationScenariosExclusiveReplica
+   */
+  private void assertExternalFileInfo(List<String> expected, String dumplocation, boolean isIncremental,
+                                      WarehouseInstance warehouseInstance)
+          throws IOException {
+    Path hivePath = new Path(dumplocation, ReplUtils.REPL_HIVE_BASE_DIR);
+    Path metadataPath = new Path(hivePath, EximUtil.METADATA_PATH_NAME);
+    Path externalTableInfoFile;
+    if (isIncremental) {
+      externalTableInfoFile = new Path(hivePath, FILE_NAME);

Review comment:
       Is this the actual file used by code or the deprecated one?

##########
File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##########
@@ -1668,6 +1671,34 @@ public void testHdfsNameserviceLazyCopy() throws Throwable {
     }
   }
 
+  @Test
+  public void testHdfsNSLazyCopyBootStrapExtTbls() throws Throwable {
+    List<String> clause = getHdfsNameserviceClause();
+    clause.add("'" + HiveConf.ConfVars.REPL_DUMP_METADATA_ONLY_FOR_EXTERNAL_TABLE.varname + "'='false'");
+    clause.add("'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'");

Review comment:
       this is true by default

##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplExternalTables.java
##########
@@ -183,6 +184,22 @@ private void dirLocationToCopy(FileList fileList, Path sourcePath, HiveConf conf
             throws HiveException {
       Path basePath = getExternalTableBaseDir(conf);
       Path targetPath = externalTableDataPath(conf, basePath, sourcePath);
+      //Here, when src and target are HA clusters with same NS, then sourcePath would have the correct host
+      //whereas the targetPath would have an host that refers to the target cluster. This is fine for
+      //data-copy running during dump as the correct logical locations would be used. But if data-copy runs during
+      //load, then the remote location needs to point to the src cluster from where the data would be copied and
+      //the common original NS would suffice for targetPath.
+      if(hiveConf.getBoolVar(HiveConf.ConfVars.REPL_HA_DATAPATH_REPLACE_REMOTE_NAMESERVICE) &&
+              hiveConf.getBoolVar(HiveConf.ConfVars.REPL_RUN_DATA_COPY_TASKS_ON_TARGET)) {
+        String remoteNS = hiveConf.get(HiveConf.ConfVars.REPL_HA_DATAPATH_REPLACE_REMOTE_NAMESERVICE_NAME.varname);
+        if (StringUtils.isEmpty(remoteNS)) {
+          throw new SemanticException(ErrorMsg.REPL_INVALID_CONFIG_FOR_SERVICE

Review comment:
       non recoverable error?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org