You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/07/31 03:05:13 UTC
[GitHub] [hudi] yihua opened a new pull request, #9315: [HUDI-5760] Use Avro as serde for delete log blocks
yihua opened a new pull request, #9315:
URL: https://github.com/apache/hudi/pull/9315
### Change Logs
This PR changes the format of delete log blocks and upgrades the log format from version 2 to 3. After this PR, the delete records (record key, partition path, and ordering value) are serialized in Avro and then written into the delete log block, instead using Kyro which is a serialization framework for Java. This makes delete blocks deserializable in programming languages other than Java, as Avro is supported by common programming languages such as [Java](https://github.com/apache/avro/tree/master/lang/java), [C/C++](https://github.com/apache/avro/tree/master/lang/c), [Python](https://github.com/apache/avro/tree/master/lang/py), and [Rust](https://github.com/apache/avro/tree/master/lang/rust).
Changes are:
- New avro schema for delete record: `HoodieDeleteRecordList` and `HoodieDeleteRecord`, reusing avro type wrappers used by column stats in metadata;
- Moves avro type wrappers and util methods to `HoodieAvroUtils` for reuse;
- Upgrades log block format version from version 2 to 3;
- Changes the logic in `HoodieDeleteBlock` to serialize delete records using Avro and deserialize v3 content using Avro. Deserialization of v2 content still uses Kryo to be backwards compatible;
- Adds unit tests of Avro serde of the delete block in `TestHoodieDeleteBlock`.
The following shows the benchmark result comparing the serde of Avro (v3) vs Kryo (v2) for a delete block with varying number of deletes. Avro serde is moderately slower than Kryo, except for serializing 1 million entries. The difference is only at 100-millisecond level at most for 1 million entries. The bytes generated by Avro are 10% to 20% larger than that by Kryo.
```
@Benchmark
public void serializeDeleteRecords(BenchState bs, Blackhole bh) throws IOException {
HoodieDeleteBlock deleteBlock = new HoodieDeleteBlock(bs.deleteRecordArray, new HashMap<>());
deleteBlock.getContentBytes();
}
@Benchmark
public void deserializeDeleteRecords(BenchState bs, Blackhole bh) throws IOException {
HoodieDeleteBlock deleteBlock = new HoodieDeleteBlock(
Option.of(bs.deleteContentBytes), null, true, Option.empty(), new HashMap<>(), new HashMap<>());
deleteBlock.getRecordsToDelete();
}
```
| numDeletes | Avro serialization | Kryo serialization | Avro deserialization | Kryo deserialization | Avro bytes | Kryo bytes |
| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |
|1 |0.001 ± 0.001 ms/op |0.001 ± 0.001 ms/op |0.001 ± 0.001 ms/op |0.001 ± 0.001 ms/op | 82| 316|
|10 |0.005 ± 0.001 ms/op |0.003 ± 0.001 ms/op |0.008 ± 0.008 ms/op |0.003 ± 0.001 ms/op | 634| 686|
|100 |0.054 ± 0.027 ms/op |0.025 ± 0.003 ms/op |0.060 ± 0.009 ms/op |0.022 ± 0.004 ms/op | 6,171| 5,546|
|1000 |0.514 ± 0.071 ms/op |0.357 ± 0.048 ms/op |0.624 ± 0.171 ms/op |0.214 ± 0.044 ms/op | 61,523| 54,146|
|10000 |5.665 ± 1.370 ms/op |4.003 ± 1.201 ms/op |6.591 ± 1.152 ms/op |2.051 ± 0.181 ms/op | 614,990| 540,100|
|100000 |84.675 ± 31.536 ms/op |86.890 ± 152.199 ms/op |54.503 ± 32.649 ms/op |21.312 ± 3.302 ms/op | 6,149,725| 5,399,763|
|1000000 |613.506 ± 85.529 ms/op |1231.790 ± 23.968 ms/op |500.381 ± 50.868 ms/op | 312.019 ± 768.724| 61,496,183| 53,996,274|
### Impact
Makes delete blocks deserializable in programming languages other than Java, as Avro is supported by common programming languages.
### Risk level
low
### Documentation Update
Update delete block format in Hudi Tech Spec and docs: HUDI-6616
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1657443751
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8da1ecd693c283a6dd04b3919115a30e8d1e8d4d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659471755
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
}, {
"hash" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931",
"triggerID" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"triggerType" : "PUSH"
}, {
"hash" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18935",
"triggerID" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"triggerType" : "PUSH"
}, {
"hash" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8366afa4faf338c62f28d3277e246a754e7a6a86 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931)
* 7aa72de28882934a1ae1d93f18822bd8edc80d9d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18935)
* 4016bdb10065fb83579c9d0488e558e2c70e023a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659457425
We also need to add a downgrade step: for downgrading a table from v6 to v5, we need to check any v3 delete blocks using the new format and ask user to manually restore to a commit before any file slice with a v3 delete block. HUDI-6621 to track the work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1661283012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943",
"triggerID" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4016bdb10065fb83579c9d0488e558e2c70e023a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1657455514
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8da1ecd693c283a6dd04b3919115a30e8d1e8d4d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1279972598
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -51,7 +51,7 @@ public abstract class HoodieLogBlock {
* corresponding changes need to be made to {@link HoodieLogBlockVersion} TODO : Change this to a class, something
* like HoodieLogBlockVersionV1/V2 and implement/override operations there
*/
- public static int version = 2;
+ public static int version = 3;
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1661187733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943",
"triggerID" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4016bdb10065fb83579c9d0488e558e2c70e023a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659379124
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
}, {
"hash" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931",
"triggerID" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae048ad8f890264db36b52091ce639747f2a9f2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912)
* 8366afa4faf338c62f28d3277e246a754e7a6a86 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1279970361
##########
hudi-common/src/main/avro/HoodieDeleteRecordList.avsc:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+ "namespace": "org.apache.hudi.avro.model",
+ "type": "record",
+ "name": "HoodieDeleteRecordList",
+ "doc": "A list of delete records stored in the delete block in log block version 3",
+ "fields": [
+ {
+ "name": "deleteRecordList",
+ "type": {
+ "type": "array",
+ "items": {
+ "name": "HoodieDeleteRecord",
+ "type": "record",
+ "fields": [
+ {
+ "name": "recordKey",
+ "type": [
+ "null",
+ "string"
+ ],
+ "default": null
+ },
+ {
+ "name": "partitionPath",
+ "type": [
+ "null",
+ "string"
+ ],
+ "default": null
+ },
+ {
+ "name": "orderingVal",
+ "type": [
+ "null",
+ "org.apache.hudi.avro.model.BooleanWrapper",
+ "org.apache.hudi.avro.model.IntWrapper",
+ "org.apache.hudi.avro.model.LongWrapper",
+ "org.apache.hudi.avro.model.FloatWrapper",
+ "org.apache.hudi.avro.model.DoubleWrapper",
+ "org.apache.hudi.avro.model.BytesWrapper",
+ "org.apache.hudi.avro.model.StringWrapper",
+ "org.apache.hudi.avro.model.DateWrapper",
+ "org.apache.hudi.avro.model.DecimalWrapper",
+ "org.apache.hudi.avro.model.TimeMicrosWrapper",
+ "org.apache.hudi.avro.model.TimestampMicrosWrapper"
Review Comment:
It looks like `TimeMicrosWrapper` is no longer used but kept for backwards compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659453598
> Do we also need to upgrade table version? I know we're already handling different versions. But, jsut asking what's the convention here? Did we update table version last time?
We need to upgrade table version. In fact, the table version is already upgrade for 0.14.0 so the new delete block format can ride on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #9315:
URL: https://github.com/apache/hudi/pull/9315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659347100
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
}, {
"hash" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae048ad8f890264db36b52091ce639747f2a9f2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912)
* 8366afa4faf338c62f28d3277e246a754e7a6a86 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659516558
cc @vinothchandar @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1661178847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4016bdb10065fb83579c9d0488e558e2c70e023a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659428849
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
}, {
"hash" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931",
"triggerID" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"triggerType" : "PUSH"
}, {
"hash" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae048ad8f890264db36b52091ce639747f2a9f2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912)
* 8366afa4faf338c62f28d3277e246a754e7a6a86 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931)
* 7aa72de28882934a1ae1d93f18822bd8edc80d9d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1657643457
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8da1ecd693c283a6dd04b3919115a30e8d1e8d4d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907)
* ae048ad8f890264db36b52091ce639747f2a9f2c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1657657636
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8da1ecd693c283a6dd04b3919115a30e8d1e8d4d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907)
* ae048ad8f890264db36b52091ce639747f2a9f2c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1658090354
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ae048ad8f890264db36b52091ce639747f2a9f2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1280013405
##########
hudi-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieDeleteBlock.java:
##########
Review Comment:
I wrote a test by writing both V2 and V3 delete blocks in the same file group and scanning the logs using `HoodieMergedLogRecordScanner` (`TestHoodieLogFormat#testLogReaderWithDifferentVersionsOfDeleteBlocks`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659917570
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
}, {
"hash" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931",
"triggerID" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"triggerType" : "PUSH"
}, {
"hash" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18935",
"triggerID" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"triggerType" : "PUSH"
}, {
"hash" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943",
"triggerID" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4016bdb10065fb83579c9d0488e558e2c70e023a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1278800755
##########
hudi-common/src/main/avro/HoodieDeleteRecordList.avsc:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+ "namespace": "org.apache.hudi.avro.model",
+ "type": "record",
+ "name": "HoodieDeleteRecordList",
+ "doc": "A list of delete records stored in the delete block in log block version 3",
+ "fields": [
+ {
+ "name": "deleteRecordList",
+ "type": {
+ "type": "array",
+ "items": {
+ "name": "HoodieDeleteRecord",
+ "type": "record",
+ "fields": [
+ {
+ "name": "recordKey",
+ "type": [
+ "null",
+ "string"
+ ],
+ "default": null
Review Comment:
let's also add doc to these fields.
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1149,4 +1192,98 @@ public static boolean gteqAvro1_9() {
public static boolean gteqAvro1_10() {
return VersionUtil.compareVersions(AVRO_VERSION, "1.10") >= 0;
}
+
+ /**
+ * Wraps a value into Avro type wrapper.
+ *
+ * @param value Java value.
+ * @return A wrapped value with Avro type wrapper.
+ */
+ public static Object wrapValueIntoAvro(Comparable<?> value) {
+ if (value == null) {
+ return null;
+ } else if (value instanceof Date || value instanceof LocalDate) {
+ // NOTE: Due to breaking changes in code-gen b/w Avro 1.8.2 and 1.10, we can't
+ // rely on logical types to do proper encoding of the native Java types,
+ // and hereby have to encode value manually
+ LocalDate localDate = value instanceof LocalDate
+ ? (LocalDate) value
+ : ((Date) value).toLocalDate();
+ return DateWrapper.newBuilder(DATE_WRAPPER_BUILDER_STUB.get())
+ .setValue((int) localDate.toEpochDay())
+ .build();
+ } else if (value instanceof BigDecimal) {
+ Schema valueSchema = DecimalWrapper.SCHEMA$.getField("value").schema();
+ BigDecimal upcastDecimal = tryUpcastDecimal((BigDecimal) value, (LogicalTypes.Decimal) valueSchema.getLogicalType());
+ return DecimalWrapper.newBuilder(DECIMAL_WRAPPER_BUILDER_STUB.get())
+ .setValue(AVRO_DECIMAL_CONVERSION.toBytes(upcastDecimal, valueSchema, valueSchema.getLogicalType()))
+ .build();
+ } else if (value instanceof Timestamp) {
+ // NOTE: Due to breaking changes in code-gen b/w Avro 1.8.2 and 1.10, we can't
+ // rely on logical types to do proper encoding of the native Java types,
+ // and hereby have to encode value manually
+ Instant instant = ((Timestamp) value).toInstant();
+ return TimestampMicrosWrapper.newBuilder(TIMESTAMP_MICROS_WRAPPER_BUILDER_STUB.get())
+ .setValue(instantToMicros(instant))
Review Comment:
Just curious why don't we need something similar timestamp-millis? Because that is covered by LongWrapper?
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1149,4 +1192,98 @@ public static boolean gteqAvro1_9() {
public static boolean gteqAvro1_10() {
return VersionUtil.compareVersions(AVRO_VERSION, "1.10") >= 0;
}
+
+ /**
+ * Wraps a value into Avro type wrapper.
+ *
+ * @param value Java value.
+ * @return A wrapped value with Avro type wrapper.
+ */
+ public static Object wrapValueIntoAvro(Comparable<?> value) {
Review Comment:
Let's add a test for wrapping/unwrapping especially for timestamp types? I know this is merely moving an existing code to the right util class. But, would be good for future-proofing. We never know when can things break again in some other AVro minor version.
##########
hudi-common/src/main/avro/HoodieDeleteRecordList.avsc:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+ "namespace": "org.apache.hudi.avro.model",
+ "type": "record",
+ "name": "HoodieDeleteRecordList",
+ "doc": "A list of delete records stored in the delete block in log block version 3",
+ "fields": [
+ {
+ "name": "deleteRecordList",
+ "type": {
+ "type": "array",
+ "items": {
+ "name": "HoodieDeleteRecord",
+ "type": "record",
+ "fields": [
+ {
+ "name": "recordKey",
+ "type": [
+ "null",
+ "string"
+ ],
+ "default": null
+ },
+ {
+ "name": "partitionPath",
+ "type": [
+ "null",
+ "string"
+ ],
+ "default": null
+ },
+ {
+ "name": "orderingVal",
+ "type": [
+ "null",
+ "org.apache.hudi.avro.model.BooleanWrapper",
+ "org.apache.hudi.avro.model.IntWrapper",
+ "org.apache.hudi.avro.model.LongWrapper",
+ "org.apache.hudi.avro.model.FloatWrapper",
+ "org.apache.hudi.avro.model.DoubleWrapper",
+ "org.apache.hudi.avro.model.BytesWrapper",
+ "org.apache.hudi.avro.model.StringWrapper",
+ "org.apache.hudi.avro.model.DateWrapper",
+ "org.apache.hudi.avro.model.DecimalWrapper",
+ "org.apache.hudi.avro.model.TimeMicrosWrapper",
+ "org.apache.hudi.avro.model.TimestampMicrosWrapper"
Review Comment:
What's the difference between `TimeMicrosWrapper` and `TimestampMicrosWrapper`?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -51,7 +51,7 @@ public abstract class HoodieLogBlock {
* corresponding changes need to be made to {@link HoodieLogBlockVersion} TODO : Change this to a class, something
* like HoodieLogBlockVersionV1/V2 and implement/override operations there
*/
- public static int version = 2;
+ public static int version = 3;
Review Comment:
let's update the above javadoc too?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/log/block/TestHoodieDeleteBlock.java:
##########
Review Comment:
Can we also cover an e2e test (maybe in mor datasource or an upgrade test) which adds a delete block in v2, then merges and writes back in v3 and then reads the snapsphot again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1280190831
##########
hudi-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java:
##########
@@ -1809,6 +1810,130 @@ public void testAvroLogRecordReaderWithInsertsDeleteAndRollback(ExternalSpillabl
0, 0, Option.empty());
}
+ @ParameterizedTest
+ @MethodSource("testArguments")
+ public void testLogReaderWithDifferentVersionsOfDeleteBlocks(ExternalSpillableMap.DiskMapType diskMapType,
Review Comment:
+1 Thanks for adding this test!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1279971547
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1149,4 +1192,98 @@ public static boolean gteqAvro1_9() {
public static boolean gteqAvro1_10() {
return VersionUtil.compareVersions(AVRO_VERSION, "1.10") >= 0;
}
+
+ /**
+ * Wraps a value into Avro type wrapper.
+ *
+ * @param value Java value.
+ * @return A wrapped value with Avro type wrapper.
+ */
+ public static Object wrapValueIntoAvro(Comparable<?> value) {
+ if (value == null) {
+ return null;
+ } else if (value instanceof Date || value instanceof LocalDate) {
+ // NOTE: Due to breaking changes in code-gen b/w Avro 1.8.2 and 1.10, we can't
+ // rely on logical types to do proper encoding of the native Java types,
+ // and hereby have to encode value manually
+ LocalDate localDate = value instanceof LocalDate
+ ? (LocalDate) value
+ : ((Date) value).toLocalDate();
+ return DateWrapper.newBuilder(DATE_WRAPPER_BUILDER_STUB.get())
+ .setValue((int) localDate.toEpochDay())
+ .build();
+ } else if (value instanceof BigDecimal) {
+ Schema valueSchema = DecimalWrapper.SCHEMA$.getField("value").schema();
+ BigDecimal upcastDecimal = tryUpcastDecimal((BigDecimal) value, (LogicalTypes.Decimal) valueSchema.getLogicalType());
+ return DecimalWrapper.newBuilder(DECIMAL_WRAPPER_BUILDER_STUB.get())
+ .setValue(AVRO_DECIMAL_CONVERSION.toBytes(upcastDecimal, valueSchema, valueSchema.getLogicalType()))
+ .build();
+ } else if (value instanceof Timestamp) {
+ // NOTE: Due to breaking changes in code-gen b/w Avro 1.8.2 and 1.10, we can't
+ // rely on logical types to do proper encoding of the native Java types,
+ // and hereby have to encode value manually
+ Instant instant = ((Timestamp) value).toInstant();
+ return TimestampMicrosWrapper.newBuilder(TIMESTAMP_MICROS_WRAPPER_BUILDER_STUB.get())
+ .setValue(instantToMicros(instant))
Review Comment:
Based on the spec of Avro's logical types, all timestamp logical types are serialized as long values. Only if Java's `Timestamp` instance is used, we wrap it using `TimestampMicrosWrapper `.
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1149,4 +1192,98 @@ public static boolean gteqAvro1_9() {
public static boolean gteqAvro1_10() {
return VersionUtil.compareVersions(AVRO_VERSION, "1.10") >= 0;
}
+
+ /**
+ * Wraps a value into Avro type wrapper.
+ *
+ * @param value Java value.
+ * @return A wrapped value with Avro type wrapper.
+ */
+ public static Object wrapValueIntoAvro(Comparable<?> value) {
+ if (value == null) {
+ return null;
+ } else if (value instanceof Date || value instanceof LocalDate) {
+ // NOTE: Due to breaking changes in code-gen b/w Avro 1.8.2 and 1.10, we can't
+ // rely on logical types to do proper encoding of the native Java types,
+ // and hereby have to encode value manually
+ LocalDate localDate = value instanceof LocalDate
+ ? (LocalDate) value
+ : ((Date) value).toLocalDate();
+ return DateWrapper.newBuilder(DATE_WRAPPER_BUILDER_STUB.get())
+ .setValue((int) localDate.toEpochDay())
+ .build();
+ } else if (value instanceof BigDecimal) {
+ Schema valueSchema = DecimalWrapper.SCHEMA$.getField("value").schema();
+ BigDecimal upcastDecimal = tryUpcastDecimal((BigDecimal) value, (LogicalTypes.Decimal) valueSchema.getLogicalType());
+ return DecimalWrapper.newBuilder(DECIMAL_WRAPPER_BUILDER_STUB.get())
+ .setValue(AVRO_DECIMAL_CONVERSION.toBytes(upcastDecimal, valueSchema, valueSchema.getLogicalType()))
+ .build();
+ } else if (value instanceof Timestamp) {
+ // NOTE: Due to breaking changes in code-gen b/w Avro 1.8.2 and 1.10, we can't
+ // rely on logical types to do proper encoding of the native Java types,
+ // and hereby have to encode value manually
+ Instant instant = ((Timestamp) value).toInstant();
+ return TimestampMicrosWrapper.newBuilder(TIMESTAMP_MICROS_WRAPPER_BUILDER_STUB.get())
+ .setValue(instantToMicros(instant))
Review Comment:
Based on the spec of Avro's logical types, all timestamp logical types are serialized as long values. Only if Java's `Timestamp` instance is used, we wrap it using `TimestampMicrosWrapper`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659434838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
}, {
"hash" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931",
"triggerID" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"triggerType" : "PUSH"
}, {
"hash" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18935",
"triggerID" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8366afa4faf338c62f28d3277e246a754e7a6a86 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931)
* 7aa72de28882934a1ae1d93f18822bd8edc80d9d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18935)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9315:
URL: https://github.com/apache/hudi/pull/9315#issuecomment-1659477588
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18907",
"triggerID" : "8da1ecd693c283a6dd04b3919115a30e8d1e8d4d",
"triggerType" : "PUSH"
}, {
"hash" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18912",
"triggerID" : "ae048ad8f890264db36b52091ce639747f2a9f2c",
"triggerType" : "PUSH"
}, {
"hash" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18931",
"triggerID" : "8366afa4faf338c62f28d3277e246a754e7a6a86",
"triggerType" : "PUSH"
}, {
"hash" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18935",
"triggerID" : "7aa72de28882934a1ae1d93f18822bd8edc80d9d",
"triggerType" : "PUSH"
}, {
"hash" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943",
"triggerID" : "4016bdb10065fb83579c9d0488e558e2c70e023a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7aa72de28882934a1ae1d93f18822bd8edc80d9d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18935)
* 4016bdb10065fb83579c9d0488e558e2c70e023a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1279970081
##########
hudi-common/src/main/avro/HoodieDeleteRecordList.avsc:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+{
+ "namespace": "org.apache.hudi.avro.model",
+ "type": "record",
+ "name": "HoodieDeleteRecordList",
+ "doc": "A list of delete records stored in the delete block in log block version 3",
+ "fields": [
+ {
+ "name": "deleteRecordList",
+ "type": {
+ "type": "array",
+ "items": {
+ "name": "HoodieDeleteRecord",
+ "type": "record",
+ "fields": [
+ {
+ "name": "recordKey",
+ "type": [
+ "null",
+ "string"
+ ],
+ "default": null
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9315: [HUDI-5760] Use Avro as serde for delete log blocks
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9315:
URL: https://github.com/apache/hudi/pull/9315#discussion_r1279970649
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java:
##########
@@ -1149,4 +1192,98 @@ public static boolean gteqAvro1_9() {
public static boolean gteqAvro1_10() {
return VersionUtil.compareVersions(AVRO_VERSION, "1.10") >= 0;
}
+
+ /**
+ * Wraps a value into Avro type wrapper.
+ *
+ * @param value Java value.
+ * @return A wrapped value with Avro type wrapper.
+ */
+ public static Object wrapValueIntoAvro(Comparable<?> value) {
Review Comment:
New unit tests are added in `TestHoodieAvroUtils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org