You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/01/15 08:04:27 UTC
[GitHub] [incubator-hudi] bvaradar opened a new pull request #1229:
[HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues
bvaradar opened a new pull request #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229
## What is the purpose of the pull request
PR #1009 introduces table layout version which is used to determine if rename must be allowed or not. Renames are disabled by default in 0.5.1 but users can enable them in ingestion through write config. Along with this feature, another related change to avoid writing compaction plans in .aux folder is added. This change could result in race conditions when we use different versions for reader and writer. This PR effectively reverts the compaction plan storage change allowing it to be also stored in .aux folder.
## Brief change log
*(for example:)*
- *Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues*
## Verify this pull request
*(Please pick either of the following options)*
Existing unit-tests covers the code change that is done
## Committer checklist
- [X] Has a corresponding JIRA in PR title & commit
- [X] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on a change in pull request
#1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder
to avoid 0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#discussion_r367229596
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##########
@@ -369,8 +366,6 @@ public HoodieInstant transitionCleanInflightToComplete(HoodieInstant inflightIns
Preconditions.checkArgument(inflightInstant.getAction().equals(HoodieTimeline.CLEAN_ACTION));
Preconditions.checkArgument(inflightInstant.isInflight());
HoodieInstant commitInstant = new HoodieInstant(State.COMPLETED, CLEAN_ACTION, inflightInstant.getTimestamp());
- // First write metadata to aux folder
Review comment:
Unlike compaction plan, Cleaner Metadata is never read concurrently read and written. Hence, there is no need to write to .aux folder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1229: [HUDI-535]
Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#issuecomment-575491975
@bvaradar any updates?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux
folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#discussion_r367167688
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##########
@@ -369,8 +366,6 @@ public HoodieInstant transitionCleanInflightToComplete(HoodieInstant inflightIns
Preconditions.checkArgument(inflightInstant.getAction().equals(HoodieTimeline.CLEAN_ACTION));
Preconditions.checkArgument(inflightInstant.isInflight());
HoodieInstant commitInstant = new HoodieInstant(State.COMPLETED, CLEAN_ACTION, inflightInstant.getTimestamp());
- // First write metadata to aux folder
Review comment:
why is this ok?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on issue #1229: [HUDI-535]
Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#issuecomment-575720079
@vinothchandar : updated the PR to handle compatibility. Jira : https://issues.apache.org/jira/browse/HUDI-546 to do the follow-up of stopping the write to aux folder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on issue #1229: [HUDI-535]
Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#issuecomment-575751729
LGTM .. merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar merged pull request #1229:
[HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull
request #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux
folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#discussion_r367471314
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##########
@@ -279,18 +280,28 @@ private void deleteInstantFile(HoodieInstant instant) {
return readDataFromPath(detailPath);
}
+ public Option<byte[]> readCleanerInfoAsBytes(HoodieInstant instant) {
+ // Cleaner metadata are always stored only in timeline .hoodie
+ return readDataFromPath(new Path(metaClient.getMetaPath(), instant.getFileName()));
+ }
+
//-----------------------------------------------------------------
// BEGIN - COMPACTION RELATED META-DATA MANAGEMENT.
//-----------------------------------------------------------------
- public Option<byte[]> readPlanAsBytes(HoodieInstant instant) {
- Path detailPath = null;
- if (metaClient.getTimelineLayoutVersion().isNullVersion()) {
- detailPath = new Path(metaClient.getMetaAuxiliaryPath(), instant.getFileName());
- } else {
- detailPath = new Path(metaClient.getMetaPath(), instant.getFileName());
+ public Option<byte[]> readCompactionPlanAsBytes(HoodieInstant instant) {
+ try {
+ // This is going to be the common case in future when 0.5.1 is deployed.
Review comment:
but then in the meantime, every reader will be doing two RPCs for this method right? I am thinking about flipping the order.. read from aux first, then fallback to meta path...
This way, when we switch to writing in metapath, only the older readers will incur this additional RPC... Does it make sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on issue #1229: [HUDI-535]
Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#issuecomment-574981683
@vinothchandar : I have made changes to handle the compatibility issue when reading compaction plan instead of double writing. Can you take one more look at this and review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-hudi] bvaradar commented on issue #1229: [HUDI-535]
Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1229: [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid 0.5.0/0.5.1 reader-writer compatibility issues
URL: https://github.com/apache/incubator-hudi/pull/1229#issuecomment-574758128
@vinothchandar : when you get a chance, please take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services