You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/07/09 14:40:36 UTC
[PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
codope opened a new pull request, #11597:
URL: https://github.com/apache/hudi/pull/11597
### Change Logs
Extract metadata record type to `MetadataPartitionType` enum. Note that we use two different record types for `FILES` partition, so `getRecordType` accepts a key as an argument. For most MDT partitions, except `FILES` key is passed as empty string.
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1705604341
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -137,6 +327,37 @@ public String getFileIdPrefix() {
return fileIdPrefix;
}
+ public int getRecordType() {
+ return recordType;
+ }
+
+ /**
+ * Construct metadata payload from the given record.
+ */
+ public void constructMetadataPayload(HoodieMetadataPayload payload, GenericRecord record) {
+ throw new UnsupportedOperationException("MetadataPayload construction not supported for partition type: " + this);
+ }
+
+ /**
+ * Merge old and new metadata payloads. By default, it returns the newer payload.
+ * Implementations can override this method to merge the payloads depending on the partition type.
+ */
+ public HoodieMetadataPayload combineMetadataPayloads(HoodieMetadataPayload older, HoodieMetadataPayload newer) {
+ return newer;
+ }
+
+ /**
+ * Get the metadata partition type for the given record type.
+ */
+ public static MetadataPartitionType get(int type) {
+ for (MetadataPartitionType partitionType : values()) {
+ if (partitionType.getRecordType() == type) {
+ return partitionType;
+ }
+ }
Review Comment:
makes sense but at the moment this loop is very cheap as ooposed ot constructing map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2218031207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec339e004d71a16e0b038bc7008385e1005f2367 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1704793858
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -99,18 +210,96 @@ public boolean isMetadataPartitionAvailable(HoodieTableMetaClient metaClient) {
}
return false;
}
+
+ @Override
+ public void constructMetadataPayload(HoodieMetadataPayload payload, GenericRecord record) {
+ GenericRecord secondaryIndexRecord = getNestedFieldValue(record, SCHEMA_FIELD_ID_SECONDARY_INDEX);
+ checkState(secondaryIndexRecord != null, "Valid SecondaryIndexMetadata record expected for type: " + MetadataPartitionType.SECONDARY_INDEX.getRecordType());
+ payload.secondaryIndexMetadata = new HoodieSecondaryIndexInfo(
+ secondaryIndexRecord.get(SECONDARY_INDEX_FIELD_RECORD_KEY).toString(),
+ (Boolean) secondaryIndexRecord.get(SECONDARY_INDEX_FIELD_IS_DELETED));
+ }
},
- PARTITION_STATS(HoodieTableMetadataUtil.PARTITION_NAME_PARTITION_STATS, "partition-stats-") {
+ PARTITION_STATS(HoodieTableMetadataUtil.PARTITION_NAME_PARTITION_STATS, "partition-stats-", 6) {
Review Comment:
Does PARTITION_STATS index needs custom combine logic in `#combineMetadataPayloads`?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2141,6 +2147,118 @@ private static List<HoodieColumnRangeMetadata<Comparable>> translateWriteStatToF
return getFileStatsRangeMetadata(writeStat.getPartitionPath(), writeStat.getPath(), datasetMetaClient, columnsToIndex, false);
}
+ public static String getPartitionStatsIndexKey(String partitionPath, String columnName) {
Review Comment:
In the refactoring of the indexing code to come, we can move these index-specific util methods to index subclass.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -137,6 +327,37 @@ public String getFileIdPrefix() {
return fileIdPrefix;
}
+ public int getRecordType() {
+ return recordType;
+ }
+
+ /**
+ * Construct metadata payload from the given record.
+ */
+ public void constructMetadataPayload(HoodieMetadataPayload payload, GenericRecord record) {
+ throw new UnsupportedOperationException("MetadataPayload construction not supported for partition type: " + this);
+ }
+
+ /**
+ * Merge old and new metadata payloads. By default, it returns the newer payload.
+ * Implementations can override this method to merge the payloads depending on the partition type.
+ */
+ public HoodieMetadataPayload combineMetadataPayloads(HoodieMetadataPayload older, HoodieMetadataPayload newer) {
+ return newer;
+ }
+
+ /**
+ * Get the metadata partition type for the given record type.
+ */
+ public static MetadataPartitionType get(int type) {
+ for (MetadataPartitionType partitionType : values()) {
+ if (partitionType.getRecordType() == type) {
+ return partitionType;
+ }
+ }
Review Comment:
Maybe we can construct a map for faster retrieval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1671384805
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -137,6 +148,10 @@ public String getFileIdPrefix() {
return fileIdPrefix;
}
+ public int getRecordType(String key) {
+ return recordType;
Review Comment:
The key is never used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1705606229
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -99,18 +210,96 @@ public boolean isMetadataPartitionAvailable(HoodieTableMetaClient metaClient) {
}
return false;
}
+
+ @Override
+ public void constructMetadataPayload(HoodieMetadataPayload payload, GenericRecord record) {
+ GenericRecord secondaryIndexRecord = getNestedFieldValue(record, SCHEMA_FIELD_ID_SECONDARY_INDEX);
+ checkState(secondaryIndexRecord != null, "Valid SecondaryIndexMetadata record expected for type: " + MetadataPartitionType.SECONDARY_INDEX.getRecordType());
+ payload.secondaryIndexMetadata = new HoodieSecondaryIndexInfo(
+ secondaryIndexRecord.get(SECONDARY_INDEX_FIELD_RECORD_KEY).toString(),
+ (Boolean) secondaryIndexRecord.get(SECONDARY_INDEX_FIELD_IS_DELETED));
+ }
},
- PARTITION_STATS(HoodieTableMetadataUtil.PARTITION_NAME_PARTITION_STATS, "partition-stats-") {
+ PARTITION_STATS(HoodieTableMetadataUtil.PARTITION_NAME_PARTITION_STATS, "partition-stats-", 6) {
Review Comment:
we reuse that for colstats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2255504032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4d4081b1d60551ae60f57f10163d0d2a63f0378 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995)
* 6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2250703652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4d4081b1d60551ae60f57f10163d0d2a63f0378 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2267390786
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
}, {
"hash" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063",
"triggerID" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017)
* 21eaff0cc54ac7a1c3a800eca278c7584b52b992 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1691466848
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -137,6 +148,10 @@ public String getFileIdPrefix() {
return fileIdPrefix;
}
+ public int getRecordType(String key) {
+ return recordType;
Review Comment:
only used for FILES because FILES partition has two different record types. I have added another method without the key argument as suggested by @bvaradar so that we don't have to pass empty string all the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2217930443
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec339e004d71a16e0b038bc7008385e1005f2367 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1695000091
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -288,19 +277,19 @@ private HoodieMetadataPayload(String key, int type, Map<String, HoodieMetadataFi
}
private HoodieMetadataPayload(String key, HoodieMetadataBloomFilter metadataBloomFilter) {
- this(key, METADATA_TYPE_BLOOM_FILTER, null, metadataBloomFilter, null, null, null);
+ this(key, MetadataPartitionType.BLOOM_FILTERS.getRecordType(), null, metadataBloomFilter, null, null, null);
}
private HoodieMetadataPayload(String key, HoodieMetadataColumnStats columnStats) {
- this(key, METADATA_TYPE_COLUMN_STATS, null, null, columnStats, null, null);
+ this(key, MetadataPartitionType.COLUMN_STATS.getRecordType(), null, null, columnStats, null, null);
}
private HoodieMetadataPayload(String key, HoodieRecordIndexInfo recordIndexMetadata) {
- this(key, METADATA_TYPE_RECORD_INDEX, null, null, null, recordIndexMetadata, null);
+ this(key, MetadataPartitionType.RECORD_INDEX.getRecordType(), null, null, null, recordIndexMetadata, null);
}
private HoodieMetadataPayload(String key, HoodieSecondaryIndexInfo secondaryIndexMetadata) {
- this(key, METADATA_TYPE_SECONDARY_INDEX, null, null, null, null, secondaryIndexMetadata);
+ this(key, MetadataPartitionType.SECONDARY_INDEX.getRecordType(), null, null, null, null, secondaryIndexMetadata);
Review Comment:
Added `ALL_PARTITIONS` with same partition path but different record type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1691875001
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option<GenericRecord> recordOpt) {
key = record.get(KEY_FIELD_NAME).toString();
type = (int) record.get(SCHEMA_FIELD_NAME_TYPE);
- if (type == METADATA_TYPE_FILE_LIST || type == METADATA_TYPE_PARTITION_LIST) {
+ if (type == MetadataPartitionType.FILES.getRecordType() || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
Review Comment:
If we follow this, there is no need to add `getRecordType()`.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option<GenericRecord> recordOpt) {
key = record.get(KEY_FIELD_NAME).toString();
type = (int) record.get(SCHEMA_FIELD_NAME_TYPE);
- if (type == METADATA_TYPE_FILE_LIST || type == METADATA_TYPE_PARTITION_LIST) {
+ if (type == MetadataPartitionType.FILES.getRecordType() || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
Review Comment:
The idea is to keep all custom logic of each index type in a single class as much as possible, so it's easy to add new index in the future.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option<GenericRecord> recordOpt) {
key = record.get(KEY_FIELD_NAME).toString();
type = (int) record.get(SCHEMA_FIELD_NAME_TYPE);
- if (type == METADATA_TYPE_FILE_LIST || type == METADATA_TYPE_PARTITION_LIST) {
+ if (type == MetadataPartitionType.FILES.getRecordType() || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
Review Comment:
Could we avoid if-else handling here and make it extensible by incorporating the payload handing through `MetadataPartitionType` enum? One way is to introduce `static MetadataPartitionType get(int type)` to get the enum based on the type ID and `void constructMetadataPayload(HoodieMetadataPayload payload, GenericRecord record)` so that each `MetadataPartitionType` handles the payload construction differently in `MetadataPartitionType`. Here simply calls `MetadataPartitionType.get(type).constructMetadataPayload(this, record)`. HoodieMetadataPayload may need to add new setters on different members like `recordIndexMetadata` for `MetadataPartitionType#constructMetadataPayload` to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2267451597
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
}, {
"hash" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063",
"triggerID" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21eaff0cc54ac7a1c3a800eca278c7584b52b992 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #11597:
URL: https://github.com/apache/hudi/pull/11597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2255623657
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1695002939
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -890,45 +877,35 @@ public String toString() {
sb.append(KEY_FIELD_NAME + "=").append(key).append(", ");
sb.append(SCHEMA_FIELD_NAME_TYPE + "=").append(type).append(", ");
- switch (type) {
- case METADATA_TYPE_PARTITION_LIST:
- case METADATA_TYPE_FILE_LIST:
- sb.append("Files: {");
- sb.append("creations=").append(Arrays.toString(getFilenames().toArray())).append(", ");
- sb.append("deletions=").append(Arrays.toString(getDeletions().toArray())).append(", ");
- sb.append("}");
- break;
- case METADATA_TYPE_BLOOM_FILTER:
- checkState(getBloomFilterMetadata().isPresent());
- sb.append("BloomFilter: {");
- sb.append("bloom size: ").append(getBloomFilterMetadata().get().getBloomFilter().array().length).append(", ");
- sb.append("timestamp: ").append(getBloomFilterMetadata().get().getTimestamp()).append(", ");
- sb.append("deleted: ").append(getBloomFilterMetadata().get().getIsDeleted());
- sb.append("}");
- break;
- case METADATA_TYPE_COLUMN_STATS:
- checkState(getColumnStatMetadata().isPresent());
- sb.append("ColStats: {");
- sb.append(getColumnStatMetadata().get());
- sb.append("}");
- break;
- case METADATA_TYPE_RECORD_INDEX:
- sb.append("RecordIndex: {");
- sb.append("location=").append(getRecordGlobalLocation());
- sb.append("}");
- break;
- default:
- break;
+ if (type == MetadataPartitionType.FILES.getRecordType() || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
+ sb.append("Files: {");
Review Comment:
I have kept this as is. There are quite a few private methods on which each partition string builder is dependent e.g. getFileNames, filterFileInfoEntries, getDeletions, etc. As discussed, enum may not be very extensible. Instead, i will work on a separate HoodieMetadataIndex API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2218151172
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec339e004d71a16e0b038bc7008385e1005f2367 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2269459201
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
}, {
"hash" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063",
"triggerID" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"triggerType" : "PUSH"
}, {
"hash" : "01982f5c69b5fdd2df3f5c7effa62c2e4f1da948",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "01982f5c69b5fdd2df3f5c7effa62c2e4f1da948",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21eaff0cc54ac7a1c3a800eca278c7584b52b992 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063)
* 01982f5c69b5fdd2df3f5c7effa62c2e4f1da948 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2250373412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec339e004d71a16e0b038bc7008385e1005f2367 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789)
* c4d4081b1d60551ae60f57f10163d0d2a63f0378 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2250481596
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec339e004d71a16e0b038bc7008385e1005f2367 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789)
* c4d4081b1d60551ae60f57f10163d0d2a63f0378 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1691884566
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -288,19 +277,19 @@ private HoodieMetadataPayload(String key, int type, Map<String, HoodieMetadataFi
}
private HoodieMetadataPayload(String key, HoodieMetadataBloomFilter metadataBloomFilter) {
- this(key, METADATA_TYPE_BLOOM_FILTER, null, metadataBloomFilter, null, null, null);
+ this(key, MetadataPartitionType.BLOOM_FILTERS.getRecordType(), null, metadataBloomFilter, null, null, null);
}
private HoodieMetadataPayload(String key, HoodieMetadataColumnStats columnStats) {
- this(key, METADATA_TYPE_COLUMN_STATS, null, null, columnStats, null, null);
+ this(key, MetadataPartitionType.COLUMN_STATS.getRecordType(), null, null, columnStats, null, null);
}
private HoodieMetadataPayload(String key, HoodieRecordIndexInfo recordIndexMetadata) {
- this(key, METADATA_TYPE_RECORD_INDEX, null, null, null, recordIndexMetadata, null);
+ this(key, MetadataPartitionType.RECORD_INDEX.getRecordType(), null, null, null, recordIndexMetadata, null);
}
private HoodieMetadataPayload(String key, HoodieSecondaryIndexInfo secondaryIndexMetadata) {
- this(key, METADATA_TYPE_SECONDARY_INDEX, null, null, null, null, secondaryIndexMetadata);
+ this(key, MetadataPartitionType.SECONDARY_INDEX.getRecordType(), null, null, null, null, secondaryIndexMetadata);
Review Comment:
I think we should keep using the `MetadataPartitionType` enum instead of the type number. We can introduce a new type named `PARTITIONS` with the same partition path and different record key. We may want to rename `MetadataPartitionType` to `MetadataIndexType` if the new name fits better into the code structure.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -890,45 +877,35 @@ public String toString() {
sb.append(KEY_FIELD_NAME + "=").append(key).append(", ");
sb.append(SCHEMA_FIELD_NAME_TYPE + "=").append(type).append(", ");
- switch (type) {
- case METADATA_TYPE_PARTITION_LIST:
- case METADATA_TYPE_FILE_LIST:
- sb.append("Files: {");
- sb.append("creations=").append(Arrays.toString(getFilenames().toArray())).append(", ");
- sb.append("deletions=").append(Arrays.toString(getDeletions().toArray())).append(", ");
- sb.append("}");
- break;
- case METADATA_TYPE_BLOOM_FILTER:
- checkState(getBloomFilterMetadata().isPresent());
- sb.append("BloomFilter: {");
- sb.append("bloom size: ").append(getBloomFilterMetadata().get().getBloomFilter().array().length).append(", ");
- sb.append("timestamp: ").append(getBloomFilterMetadata().get().getTimestamp()).append(", ");
- sb.append("deleted: ").append(getBloomFilterMetadata().get().getIsDeleted());
- sb.append("}");
- break;
- case METADATA_TYPE_COLUMN_STATS:
- checkState(getColumnStatMetadata().isPresent());
- sb.append("ColStats: {");
- sb.append(getColumnStatMetadata().get());
- sb.append("}");
- break;
- case METADATA_TYPE_RECORD_INDEX:
- sb.append("RecordIndex: {");
- sb.append("location=").append(getRecordGlobalLocation());
- sb.append("}");
- break;
- default:
- break;
+ if (type == MetadataPartitionType.FILES.getRecordType() || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
+ sb.append("Files: {");
Review Comment:
Similar here.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -409,31 +398,29 @@ public HoodieMetadataPayload preCombine(HoodieMetadataPayload previousRecord) {
checkArgument(previousRecord.key.equals(key),
"Cannot combine " + previousRecord.key + " with " + key + " as the keys differ");
- switch (type) {
- case METADATA_TYPE_PARTITION_LIST:
- case METADATA_TYPE_FILE_LIST:
- Map<String, HoodieMetadataFileInfo> combinedFileInfo = combineFileSystemMetadata(previousRecord);
- return new HoodieMetadataPayload(key, type, combinedFileInfo);
- case METADATA_TYPE_BLOOM_FILTER:
- HoodieMetadataBloomFilter combineBloomFilterMetadata = combineBloomFilterMetadata(previousRecord);
- return new HoodieMetadataPayload(key, combineBloomFilterMetadata);
- case METADATA_TYPE_COLUMN_STATS:
- return new HoodieMetadataPayload(key, combineColumnStatsMetadata(previousRecord));
- case METADATA_TYPE_RECORD_INDEX:
- // There is always a single mapping and the latest mapping is maintained.
- // Mappings in record index can change in two scenarios:
- // 1. A key deleted from dataset and then added again (new filedID)
- // 2. A key moved to a different file due to clustering
-
- // No need to merge with previous record index, always pick the latest payload.
- return this;
- case METADATA_TYPE_SECONDARY_INDEX:
- // Secondary Index combine()/merge() always returns the current (*this*)
- // record and discards the prevRecord. Based on the 'isDeleted' marker in the payload,
- // the merger running on top takes the right action (discard current or retain current record).
- return this;
- default:
- throw new HoodieMetadataException("Unknown type of HoodieMetadataPayload: " + type);
+ if (type == MetadataPartitionType.FILES.getRecordType() || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
+ Map<String, HoodieMetadataFileInfo> combinedFileInfo = combineFileSystemMetadata(previousRecord);
Review Comment:
Similar here on removing the if-else logic by replacing it with logic in each enum in `MetadataPartitionType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1680168106
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option<GenericRecord> recordOpt) {
key = record.get(KEY_FIELD_NAME).toString();
type = (int) record.get(SCHEMA_FIELD_NAME_TYPE);
- if (type == METADATA_TYPE_FILE_LIST || type == METADATA_TYPE_PARTITION_LIST) {
+ if (type == MetadataPartitionType.FILES.getRecordType(EMPTY_STRING) || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
Review Comment:
For better readability, Can you introduce a method getRecordType() in MetadatPartitionType enum without any key argument and pass the correct type number back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2255487064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4d4081b1d60551ae60f57f10163d0d2a63f0378 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995)
* 6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2269474158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
}, {
"hash" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063",
"triggerID" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"triggerType" : "PUSH"
}, {
"hash" : "01982f5c69b5fdd2df3f5c7effa62c2e4f1da948",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25067",
"triggerID" : "01982f5c69b5fdd2df3f5c7effa62c2e4f1da948",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 21eaff0cc54ac7a1c3a800eca278c7584b52b992 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063)
* 01982f5c69b5fdd2df3f5c7effa62c2e4f1da948 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25067)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2267385043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
}, {
"hash" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017)
* 21eaff0cc54ac7a1c3a800eca278c7584b52b992 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11597:
URL: https://github.com/apache/hudi/pull/11597#issuecomment-2269643250
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24789",
"triggerID" : "ec339e004d71a16e0b038bc7008385e1005f2367",
"triggerType" : "PUSH"
}, {
"hash" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24995",
"triggerID" : "c4d4081b1d60551ae60f57f10163d0d2a63f0378",
"triggerType" : "PUSH"
}, {
"hash" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25017",
"triggerID" : "6bcd88040daa36a1dadd0f89bf3852f5dc1b25c7",
"triggerType" : "PUSH"
}, {
"hash" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25063",
"triggerID" : "21eaff0cc54ac7a1c3a800eca278c7584b52b992",
"triggerType" : "PUSH"
}, {
"hash" : "01982f5c69b5fdd2df3f5c7effa62c2e4f1da948",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25067",
"triggerID" : "01982f5c69b5fdd2df3f5c7effa62c2e4f1da948",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 01982f5c69b5fdd2df3f5c7effa62c2e4f1da948 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=25067)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1705602504
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2141,6 +2147,118 @@ private static List<HoodieColumnRangeMetadata<Comparable>> translateWriteStatToF
return getFileStatsRangeMetadata(writeStat.getPartitionPath(), writeStat.getPath(), datasetMetaClient, columnsToIndex, false);
}
+ public static String getPartitionStatsIndexKey(String partitionPath, String columnName) {
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1691467321
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option<GenericRecord> recordOpt) {
key = record.get(KEY_FIELD_NAME).toString();
type = (int) record.get(SCHEMA_FIELD_NAME_TYPE);
- if (type == METADATA_TYPE_FILE_LIST || type == METADATA_TYPE_PARTITION_LIST) {
+ if (type == MetadataPartitionType.FILES.getRecordType(EMPTY_STRING) || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7692] Extract metadata record type to MetadataPartitionType enum [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1694999684
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataPayload.java:
##########
@@ -211,7 +200,7 @@ public HoodieMetadataPayload(Option<GenericRecord> recordOpt) {
key = record.get(KEY_FIELD_NAME).toString();
type = (int) record.get(SCHEMA_FIELD_NAME_TYPE);
- if (type == METADATA_TYPE_FILE_LIST || type == METADATA_TYPE_PARTITION_LIST) {
+ if (type == MetadataPartitionType.FILES.getRecordType() || type == MetadataPartitionType.FILES.getRecordType(RECORDKEY_PARTITION_LIST)) {
Review Comment:
Agree with the general idea. For this PR, I have refactored `MetadataPartitionType` to only construct payload and combine payloads for multiple partition types. Will followup with more refactoring in subsequent PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org