You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/18 09:32:29 UTC
[GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1633: [HUDI-858] Allow multiple operations to be executed within a single commit
vinothchandar commented on a change in pull request #1633:
URL: https://github.com/apache/incubator-hudi/pull/1633#discussion_r426492576
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -99,6 +99,20 @@
private static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
private static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
+ /**
+ * HUDI-858 : There are users who had been directly using RDD APIs and have relied on a behavior in 0.4.x to allow
+ * multiple write operations (upsert/buk-insert/...) to be executed within a single commit.
+ *
+ * Given Hudi commit protocol, these are generally unsafe operations and user need to handle failure scenarios. It
+ * only works with COW table. Hudi 0.5.x had stopped this behavior.
+ *
+ * Given the importance of supporting such cases for the user's migration to 0.5.x, we are proposing a safety flag
+ * (disabled by default) which will allow this old behavior.
+ */
+ private static final String ALLOW_UNSAFE_MULTI_OPERATIONS_PER_COMMIT =
+ "hoodie.allow.unsafe.multi_operations_per_commit";
Review comment:
@n3nash was also thinking about adding one of these.. Let's give these a consistent name.prefix.. `_.hoodie.allow.multi.write.on.same.instant` .. beginning with underscore can mean (spark also has similar undoced params.. ) its undoc-ed/hidden and to be used with full understanding..
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -99,6 +99,20 @@
private static final String MAX_CONSISTENCY_CHECKS_PROP = "hoodie.consistency.check.max_checks";
private static int DEFAULT_MAX_CONSISTENCY_CHECKS = 7;
+ /**
+ * HUDI-858 : There are users who had been directly using RDD APIs and have relied on a behavior in 0.4.x to allow
+ * multiple write operations (upsert/buk-insert/...) to be executed within a single commit.
+ *
+ * Given Hudi commit protocol, these are generally unsafe operations and user need to handle failure scenarios. It
+ * only works with COW table. Hudi 0.5.x had stopped this behavior.
+ *
+ * Given the importance of supporting such cases for the user's migration to 0.5.x, we are proposing a safety flag
+ * (disabled by default) which will allow this old behavior.
+ */
+ private static final String ALLOW_UNSAFE_MULTI_OPERATIONS_PER_COMMIT =
+ "hoodie.allow.unsafe.multi_operations_per_commit";
Review comment:
lets not mix underscore and dot notation? :)
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -723,6 +741,11 @@ public Builder withEmbeddedTimelineServerEnabled(boolean enabled) {
return this;
}
+ public Builder withAllowUnsafeMultiOperationsPerCommit(boolean allow) {
+ props.setProperty(ALLOW_UNSAFE_MULTI_OPERATIONS_PER_COMMIT, String.valueOf(allow));
Review comment:
rename based on property name suggested above?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org