You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/01/10 07:52:13 UTC

[GitHub] [incubator-hudi] harveyyue opened a new pull request #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…

harveyyue opened a new pull request #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…
URL: https://github.com/apache/incubator-hudi/pull/1206
 
 
   ## What is the purpose of the pull request
   This code change fix the correct checkpoints to deltacommit/commit metadata
   1. choose the correct completed timeline for MOR/COW
   2. set empty checkpoint metadata for compaction
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…
URL: https://github.com/apache/incubator-hudi/pull/1206#discussion_r365561070
 
 

 ##########
 File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
 ##########
 @@ -378,7 +387,7 @@ private void refreshTimeline() throws IOException {
 
         // Schedule compaction if needed
         if (cfg.isAsyncCompactionEnabled()) {
-          scheduledCompactionInstant = writeClient.scheduleCompaction(Option.of(checkpointCommitMetadata));
+          scheduledCompactionInstant = writeClient.scheduleCompaction(Option.empty());
 
 Review comment:
   Seems this was being done so that when compaction is run inline (default), we would get the checkpoints? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] bvaradar merged pull request #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…

Posted by GitBox <gi...@apache.org>.
bvaradar merged pull request #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…
URL: https://github.com/apache/incubator-hudi/pull/1206
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] harveyyue commented on issue #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…

Posted by GitBox <gi...@apache.org>.
harveyyue commented on issue #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…
URL: https://github.com/apache/incubator-hudi/pull/1206#issuecomment-573397616
 
 
   Yes Vinoth, since I have no jira permission to assign HUDI-322 to me

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-hudi] vinothchandar commented on issue #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1206: [HUDI-322] DeltaSteamer should pick checkpoints off only deltacommits…
URL: https://github.com/apache/incubator-hudi/pull/1206#issuecomment-573388767
 
 
   Do you also may be want to assign HUDI-322 to yourself? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services