You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zhangyue19921010 (via GitHub)" <gi...@apache.org> on 2023/02/16 03:10:03 UTC
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #7941: [HUDI-5786] Add a new config to specific spark write rdd storage level
zhangyue19921010 commented on code in PR #7941:
URL: https://github.com/apache/hudi/pull/7941#discussion_r1107957756
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -127,6 +127,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.noDefaultValue()
.withDocumentation("Table name that will be used for registering with metastores like HMS. Needs to be same across runs.");
+ public static final ConfigProperty<String> SPARK_WRITE_STORAGE_LEVEL_VALUE = ConfigProperty
+ .key("hoodie.spark.write.storage.level")
Review Comment:
nit: could we use `hoodie.write.tagged.record.storage.level`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -127,6 +127,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.noDefaultValue()
.withDocumentation("Table name that will be used for registering with metastores like HMS. Needs to be same across runs.");
+ public static final ConfigProperty<String> SPARK_WRITE_STORAGE_LEVEL_VALUE = ConfigProperty
+ .key("hoodie.spark.write.storage.level")
+ .defaultValue("MEMORY_AND_DISK_SER")
+ .withDocumentation("Determine what level of persistence is used to cache write RDDs. "
Review Comment:
is used to cache the tagged records
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -127,6 +127,12 @@ public class HoodieWriteConfig extends HoodieConfig {
.noDefaultValue()
.withDocumentation("Table name that will be used for registering with metastores like HMS. Needs to be same across runs.");
+ public static final ConfigProperty<String> SPARK_WRITE_STORAGE_LEVEL_VALUE = ConfigProperty
Review Comment:
TAGGED_RECORD_STORAGE_LEVEL_VALUE
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1069,6 +1075,10 @@ public String getWriteSchema() {
return getSchema();
}
+ public String getSparkWriteStorageLevel() {
Review Comment:
getTaggedRecordStorageLevel()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org