You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "usberkeley (via GitHub)" <gi...@apache.org> on 2024/10/27 09:03:57 UTC
[PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
usberkeley opened a new pull request, #12171:
URL: https://github.com/apache/hudi/pull/12171
### Change Logs
Fix log block footer incorrectly using HeaderMetadataType
### Impact
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2440462621
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427)
* 41a4c89b1657959f694e4c67c6f73e82d3f12457 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2439916485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bea9d371ffd9addbab67e4333e2403155a56f33d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2440989357
> Needs to avoid code duplication for `getHeaderMetadataBytes, getHeaderMetadata` and `getFooterMetadataBytes, getFooterMetadata` in `HoodieLogBlock` class somehow.
Thank you for the review. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2439972490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2440975009
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
}, {
"hash" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 41a4c89b1657959f694e4c67c6f73e82d3f12457 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431)
* 2f7d540ab6c9b3d0940bb9a51403f401098018b8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1819036827
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -255,42 +256,31 @@ public long getBlockEndPos() {
}
/**
- * Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ * Convert header metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
*/
- public static byte[] getLogMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- DataOutputStream output = new DataOutputStream(baos);
- output.writeInt(metadata.size());
- for (Map.Entry<HeaderMetadataType, String> entry : metadata.entrySet()) {
- output.writeInt(entry.getKey().ordinal());
- byte[] bytes = getUTF8Bytes(entry.getValue());
- output.writeInt(bytes.length);
- output.write(bytes);
- }
- return baos.toByteArray();
+ public static byte[] getHeaderMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
}
/**
- * Convert bytes to LogMetadata, follow the same order as {@link HoodieLogBlock#getLogMetadataBytes}.
+ * Convert bytes to Header Metadata, follow the same order as {@link HoodieLogBlock#getHeaderMetadataBytes}.
*/
- public static Map<HeaderMetadataType, String> getLogMetadata(SeekableDataInputStream dis) throws IOException {
+ public static Map<HeaderMetadataType, String> getHeaderMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> HeaderMetadataType.values()[index]);
+ }
- Map<HeaderMetadataType, String> metadata = new HashMap<>();
- // 1. Read the metadata written out
- int metadataCount = dis.readInt();
- try {
- while (metadataCount > 0) {
- int metadataEntryIndex = dis.readInt();
- int metadataEntrySize = dis.readInt();
- byte[] metadataEntry = new byte[metadataEntrySize];
- dis.readFully(metadataEntry, 0, metadataEntrySize);
- metadata.put(HeaderMetadataType.values()[metadataEntryIndex], new String(metadataEntry));
- metadataCount--;
- }
- return metadata;
- } catch (EOFException eof) {
- throw new IOException("Could not read metadata fields ", eof);
- }
+ /**
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ */
+ public static byte[] getFooterMetadataBytes(Map<FooterMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
+ }
+
+ /**
+ * Convert bytes to Footer Metadata, follow the same order as {@link HoodieLogBlock#getFooterMetadataBytes}.
+ */
+ public static Map<FooterMetadataType, String> getFooterMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> FooterMetadataType.values()[index]);
Review Comment:
will this trigger NPE because the `FooterMetadataType.values()` returns empty array?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2440534765
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 41a4c89b1657959f694e4c67c6f73e82d3f12457 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2441036797
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
}, {
"hash" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433",
"triggerID" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 41a4c89b1657959f694e4c67c6f73e82d3f12457 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431)
* 2f7d540ab6c9b3d0940bb9a51403f401098018b8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1818636192
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -271,9 +271,25 @@ public static byte[] getLogMetadataBytes(Map<HeaderMetadataType, String> metadat
}
/**
- * Convert bytes to LogMetadata, follow the same order as {@link HoodieLogBlock#getLogMetadataBytes}.
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
*/
- public static Map<HeaderMetadataType, String> getLogMetadata(SeekableDataInputStream dis) throws IOException {
+ public static byte[] getFooterMetadataBytes(Map<FooterMetadataType, String> metadata) throws IOException {
Review Comment:
The `FooterMetadataType` is empty, is this just a code usage fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2440985541
> Needs to avoid code duplication for `getHeaderMetadataBytes, getHeaderMetadata` and `getFooterMetadataBytes, getFooterMetadata` in `HoodieLogBlock` class somehow.
Thank you for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2439917043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bea9d371ffd9addbab67e4333e2403155a56f33d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2441500596
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
}, {
"hash" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433",
"triggerID" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"triggerType" : "PUSH"
}, {
"hash" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1435",
"triggerID" : "2441341447",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2f7d540ab6c9b3d0940bb9a51403f401098018b8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1435)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1819056185
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -255,42 +256,31 @@ public long getBlockEndPos() {
}
/**
- * Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ * Convert header metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
*/
- public static byte[] getLogMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- DataOutputStream output = new DataOutputStream(baos);
- output.writeInt(metadata.size());
- for (Map.Entry<HeaderMetadataType, String> entry : metadata.entrySet()) {
- output.writeInt(entry.getKey().ordinal());
- byte[] bytes = getUTF8Bytes(entry.getValue());
- output.writeInt(bytes.length);
- output.write(bytes);
- }
- return baos.toByteArray();
+ public static byte[] getHeaderMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
}
/**
- * Convert bytes to LogMetadata, follow the same order as {@link HoodieLogBlock#getLogMetadataBytes}.
+ * Convert bytes to Header Metadata, follow the same order as {@link HoodieLogBlock#getHeaderMetadataBytes}.
*/
- public static Map<HeaderMetadataType, String> getLogMetadata(SeekableDataInputStream dis) throws IOException {
+ public static Map<HeaderMetadataType, String> getHeaderMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> HeaderMetadataType.values()[index]);
+ }
- Map<HeaderMetadataType, String> metadata = new HashMap<>();
- // 1. Read the metadata written out
- int metadataCount = dis.readInt();
- try {
- while (metadataCount > 0) {
- int metadataEntryIndex = dis.readInt();
- int metadataEntrySize = dis.readInt();
- byte[] metadataEntry = new byte[metadataEntrySize];
- dis.readFully(metadataEntry, 0, metadataEntrySize);
- metadata.put(HeaderMetadataType.values()[metadataEntryIndex], new String(metadataEntry));
- metadataCount--;
- }
- return metadata;
- } catch (EOFException eof) {
- throw new IOException("Could not read metadata fields ", eof);
- }
+ /**
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ */
+ public static byte[] getFooterMetadataBytes(Map<FooterMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
+ }
+
+ /**
+ * Convert bytes to Footer Metadata, follow the same order as {@link HoodieLogBlock#getFooterMetadataBytes}.
+ */
+ public static Map<FooterMetadataType, String> getFooterMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> FooterMetadataType.values()[index]);
Review Comment:
actually, this function `index -> FooterMetadataType.values()[index]` is not executed if there is no metadata in footer.
And to write metadata to footer - at firts, you need to add some values to FooterMetadataType enum. So, i think NPE is impossible here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2439937482
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bea9d371ffd9addbab67e4333e2403155a56f33d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426)
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2439919205
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bea9d371ffd9addbab67e4333e2403155a56f33d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426)
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1819056185
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -255,42 +256,31 @@ public long getBlockEndPos() {
}
/**
- * Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ * Convert header metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
*/
- public static byte[] getLogMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- DataOutputStream output = new DataOutputStream(baos);
- output.writeInt(metadata.size());
- for (Map.Entry<HeaderMetadataType, String> entry : metadata.entrySet()) {
- output.writeInt(entry.getKey().ordinal());
- byte[] bytes = getUTF8Bytes(entry.getValue());
- output.writeInt(bytes.length);
- output.write(bytes);
- }
- return baos.toByteArray();
+ public static byte[] getHeaderMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
}
/**
- * Convert bytes to LogMetadata, follow the same order as {@link HoodieLogBlock#getLogMetadataBytes}.
+ * Convert bytes to Header Metadata, follow the same order as {@link HoodieLogBlock#getHeaderMetadataBytes}.
*/
- public static Map<HeaderMetadataType, String> getLogMetadata(SeekableDataInputStream dis) throws IOException {
+ public static Map<HeaderMetadataType, String> getHeaderMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> HeaderMetadataType.values()[index]);
+ }
- Map<HeaderMetadataType, String> metadata = new HashMap<>();
- // 1. Read the metadata written out
- int metadataCount = dis.readInt();
- try {
- while (metadataCount > 0) {
- int metadataEntryIndex = dis.readInt();
- int metadataEntrySize = dis.readInt();
- byte[] metadataEntry = new byte[metadataEntrySize];
- dis.readFully(metadataEntry, 0, metadataEntrySize);
- metadata.put(HeaderMetadataType.values()[metadataEntryIndex], new String(metadataEntry));
- metadataCount--;
- }
- return metadata;
- } catch (EOFException eof) {
- throw new IOException("Could not read metadata fields ", eof);
- }
+ /**
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ */
+ public static byte[] getFooterMetadataBytes(Map<FooterMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
+ }
+
+ /**
+ * Convert bytes to Footer Metadata, follow the same order as {@link HoodieLogBlock#getFooterMetadataBytes}.
+ */
+ public static Map<FooterMetadataType, String> getFooterMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> FooterMetadataType.values()[index]);
Review Comment:
actually, this function `index -> FooterMetadataType.values()[index]` is not executed if there is no metadata in footer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2440607399
Needs to avoid code duplication for `getHeaderMetadataBytes, getHeaderMetadata` and `getFooterMetadataBytes, getFooterMetadata` in `HoodieLogBlock` class somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "usberkeley (via GitHub)" <gi...@apache.org>.
usberkeley commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1818647683
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -271,9 +271,25 @@ public static byte[] getLogMetadataBytes(Map<HeaderMetadataType, String> metadat
}
/**
- * Convert bytes to LogMetadata, follow the same order as {@link HoodieLogBlock#getLogMetadataBytes}.
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
*/
- public static Map<HeaderMetadataType, String> getLogMetadata(SeekableDataInputStream dis) throws IOException {
+ public static byte[] getFooterMetadataBytes(Map<FooterMetadataType, String> metadata) throws IOException {
Review Comment:
> The `FooterMetadataType` is empty, is this just a code usage fix?
Yes, currently it is just a code usage fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1819056185
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -255,42 +256,31 @@ public long getBlockEndPos() {
}
/**
- * Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ * Convert header metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
*/
- public static byte[] getLogMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- DataOutputStream output = new DataOutputStream(baos);
- output.writeInt(metadata.size());
- for (Map.Entry<HeaderMetadataType, String> entry : metadata.entrySet()) {
- output.writeInt(entry.getKey().ordinal());
- byte[] bytes = getUTF8Bytes(entry.getValue());
- output.writeInt(bytes.length);
- output.write(bytes);
- }
- return baos.toByteArray();
+ public static byte[] getHeaderMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
}
/**
- * Convert bytes to LogMetadata, follow the same order as {@link HoodieLogBlock#getLogMetadataBytes}.
+ * Convert bytes to Header Metadata, follow the same order as {@link HoodieLogBlock#getHeaderMetadataBytes}.
*/
- public static Map<HeaderMetadataType, String> getLogMetadata(SeekableDataInputStream dis) throws IOException {
+ public static Map<HeaderMetadataType, String> getHeaderMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> HeaderMetadataType.values()[index]);
+ }
- Map<HeaderMetadataType, String> metadata = new HashMap<>();
- // 1. Read the metadata written out
- int metadataCount = dis.readInt();
- try {
- while (metadataCount > 0) {
- int metadataEntryIndex = dis.readInt();
- int metadataEntrySize = dis.readInt();
- byte[] metadataEntry = new byte[metadataEntrySize];
- dis.readFully(metadataEntry, 0, metadataEntrySize);
- metadata.put(HeaderMetadataType.values()[metadataEntryIndex], new String(metadataEntry));
- metadataCount--;
- }
- return metadata;
- } catch (EOFException eof) {
- throw new IOException("Could not read metadata fields ", eof);
- }
+ /**
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ */
+ public static byte[] getFooterMetadataBytes(Map<FooterMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
+ }
+
+ /**
+ * Convert bytes to Footer Metadata, follow the same order as {@link HoodieLogBlock#getFooterMetadataBytes}.
+ */
+ public static Map<FooterMetadataType, String> getFooterMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> FooterMetadataType.values()[index]);
Review Comment:
actually, this function `index -> FooterMetadataType.values()[index]` is not executed if there is no metadata in footer.
And to write metadata to footer - at first, you need to add some values to FooterMetadataType enum. So, i think NPE is impossible here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2439919643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bea9d371ffd9addbab67e4333e2403155a56f33d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426)
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12171:
URL: https://github.com/apache/hudi/pull/12171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2441244819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
}, {
"hash" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433",
"triggerID" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2f7d540ab6c9b3d0940bb9a51403f401098018b8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2441344396
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1431",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
}, {
"hash" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433",
"triggerID" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"triggerType" : "PUSH"
}, {
"hash" : "2f7d540ab6c9b3d0940bb9a51403f401098018b8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1435",
"triggerID" : "2441341447",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2f7d540ab6c9b3d0940bb9a51403f401098018b8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1433) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1435)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2441341447
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1819938842
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -255,42 +256,31 @@ public long getBlockEndPos() {
}
/**
- * Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ * Convert header metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
*/
- public static byte[] getLogMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- DataOutputStream output = new DataOutputStream(baos);
- output.writeInt(metadata.size());
- for (Map.Entry<HeaderMetadataType, String> entry : metadata.entrySet()) {
- output.writeInt(entry.getKey().ordinal());
- byte[] bytes = getUTF8Bytes(entry.getValue());
- output.writeInt(bytes.length);
- output.write(bytes);
- }
- return baos.toByteArray();
+ public static byte[] getHeaderMetadataBytes(Map<HeaderMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
}
/**
- * Convert bytes to LogMetadata, follow the same order as {@link HoodieLogBlock#getLogMetadataBytes}.
+ * Convert bytes to Header Metadata, follow the same order as {@link HoodieLogBlock#getHeaderMetadataBytes}.
*/
- public static Map<HeaderMetadataType, String> getLogMetadata(SeekableDataInputStream dis) throws IOException {
+ public static Map<HeaderMetadataType, String> getHeaderMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> HeaderMetadataType.values()[index]);
+ }
- Map<HeaderMetadataType, String> metadata = new HashMap<>();
- // 1. Read the metadata written out
- int metadataCount = dis.readInt();
- try {
- while (metadataCount > 0) {
- int metadataEntryIndex = dis.readInt();
- int metadataEntrySize = dis.readInt();
- byte[] metadataEntry = new byte[metadataEntrySize];
- dis.readFully(metadataEntry, 0, metadataEntrySize);
- metadata.put(HeaderMetadataType.values()[metadataEntryIndex], new String(metadataEntry));
- metadataCount--;
- }
- return metadata;
- } catch (EOFException eof) {
- throw new IOException("Could not read metadata fields ", eof);
- }
+ /**
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ */
+ public static byte[] getFooterMetadataBytes(Map<FooterMetadataType, String> metadata) throws IOException {
+ return getLogMetadataBytes(metadata);
+ }
+
+ /**
+ * Convert bytes to Footer Metadata, follow the same order as {@link HoodieLogBlock#getFooterMetadataBytes}.
+ */
+ public static Map<FooterMetadataType, String> getFooterMetadata(SeekableDataInputStream dis) throws IOException {
+ return getLogMetadata(dis, index -> FooterMetadataType.values()[index]);
Review Comment:
Makes sense somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12171:
URL: https://github.com/apache/hudi/pull/12171#issuecomment-2440436035
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1426",
"triggerID" : "bea9d371ffd9addbab67e4333e2403155a56f33d",
"triggerType" : "PUSH"
}, {
"hash" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6593267714ae85c2545d0c467fbf30f5fb2a56a2",
"triggerType" : "PUSH"
}, {
"hash" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427",
"triggerID" : "2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e",
"triggerType" : "PUSH"
}, {
"hash" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "41a4c89b1657959f694e4c67c6f73e82d3f12457",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6593267714ae85c2545d0c467fbf30f5fb2a56a2 UNKNOWN
* 2d7c4150a18ddbbc57b10c94d34f0e75aa95fe9e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1427)
* 41a4c89b1657959f694e4c67c6f73e82d3f12457 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8448] Fix log block footer incorrectly using HeaderMetadataType [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #12171:
URL: https://github.com/apache/hudi/pull/12171#discussion_r1818406390
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -255,9 +255,9 @@ public long getBlockEndPos() {
}
/**
- * Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
+ * Convert footer metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes
Review Comment:
Convert **header**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org