[GitHub] [hive] ashutosh-bapat commented on a change in pull request #587: HIVE-21213 : Acid table bootstrap replication needs to handle directory created by compaction with txn id

[GitBox <gi...@apache.org>]

ashutosh-bapat commented on a change in pull request #587: HIVE-21213 : Acid table bootstrap replication needs to handle directory created by compaction with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r354223809
 
 

 ##########
 File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
 ##########
 @@ -463,7 +464,29 @@ public static Path getCopyDestination(ReplChangeManager.FileInfo fileInfo, Path
     String[] subDirs = fileInfo.getSubDir().split(Path.SEPARATOR);
     Path destination = destRoot;
     for (String subDir: subDirs) {
-      destination = new Path(destination, subDir);
+      // If the directory is created by compactor, then the directory will have the transaction id also.
+      // In case of replication, the same txn id can not be used at target, as the txn with same id might be a
+      // aborted or live txn at target.
+      // In case of bootstrap load, we copy only the committed data, so the directory with only write id
+      // can be created. The validity txn id can be removed from the directory name.
+      // TODO : Support for incremental load flow. This can be done once replication of compaction is decided.
 
 Review comment:
   Is this TODO resolved? If yes, please remove it. If it needs to be resolved, may be we should resolve it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org