You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/29 08:23:13 UTC
[GitHub] [hudi] xushiyan commented on a diff in pull request #5664: [HUDI-4140] Fixing hive style partitioning and default partition with bulk insert row writer with SimpleKeyGen and virtual keys
xushiyan commented on code in PR #5664:
URL: https://github.com/apache/hudi/pull/5664#discussion_r884231341
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java:
##########
@@ -128,7 +133,11 @@ public void write(InternalRow record) throws IOException {
if (!keyGeneratorOpt.isPresent()) { // NoPartitionerKeyGen
partitionPath = "";
} else if (simpleKeyGen) { // SimpleKeyGen
- partitionPath = (record.get(simplePartitionFieldIndex, simplePartitionFieldDataType)).toString();
+ Object parititionPathValue = record.get(simplePartitionFieldIndex, simplePartitionFieldDataType);
+ partitionPath = parititionPathValue != null ? parititionPathValue.toString() : PartitionPathEncodeUtils.DEFAULT_PARTITION_PATH;
+ if (isHiveStylePartitioning) {
+ partitionPath = (keyGeneratorOpt.get()).getPartitionPathFields().get(0) + "=" + partitionPath;
Review Comment:
`(keyGeneratorOpt.get())` the bracket is redundant
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java:
##########
@@ -87,6 +89,9 @@ public BulkInsertDataInternalWriterHelper(HoodieTable hoodieTable, HoodieWriteCo
this.populateMetaFields = populateMetaFields;
this.arePartitionRecordsSorted = arePartitionRecordsSorted;
this.fileIdPrefix = UUID.randomUUID().toString();
+ this.isHiveStylePartitioning = writeConfig.getProps().containsKey(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING().key())
+ ? Boolean.parseBoolean((String) writeConfig.getProps().get(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING().key()))
+ : Boolean.parseBoolean(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING().defaultValue());
Review Comment:
this is very verbose.. why not add an API in write config say `isHiveStylePartitioning()` ?
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java:
##########
@@ -128,7 +133,11 @@ public void write(InternalRow record) throws IOException {
if (!keyGeneratorOpt.isPresent()) { // NoPartitionerKeyGen
partitionPath = "";
} else if (simpleKeyGen) { // SimpleKeyGen
- partitionPath = (record.get(simplePartitionFieldIndex, simplePartitionFieldDataType)).toString();
+ Object parititionPathValue = record.get(simplePartitionFieldIndex, simplePartitionFieldDataType);
+ partitionPath = parititionPathValue != null ? parititionPathValue.toString() : PartitionPathEncodeUtils.DEFAULT_PARTITION_PATH;
+ if (isHiveStylePartitioning) {
+ partitionPath = (keyGeneratorOpt.get()).getPartitionPathFields().get(0) + "=" + partitionPath;
Review Comment:
why only get the first partition path field and add `=` sign? what about multi-level partition like `year=2020/month=01/day=01` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org