You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "beyond1920 (via GitHub)" <gi...@apache.org> on 2023/08/11 13:10:24 UTC
[GitHub] [hudi] beyond1920 opened a new issue, #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
beyond1920 opened a new issue, #9430:
URL: https://github.com/apache/hudi/issues/9430
Dear community:
Some of our internal users are very interested in RFC-46, but they have some custom payloads. Therefore, I need to reconstruct these payloads using the new `HoodieRecordMerger` API. During this process, I found that some capabilities of the original payload seem to be missing in the new HoodieRecordMerger API.
An example is a custom payload that simulates the logic of a left outer join, which is named as `LeftOuterJoinPayload`. It's similar to the `PartialUpdateAvroPayload`, with the difference that the `LeftOuterJoinPayload` needs to discard rows that do not meet the left outer join condition before being written to storage.
Illustrating the functioning of this payload with an example, let's assume there are two writers working on writing to a HUDI wide table. Writer1 is responsible for reading data from source table1 and writing it to this wide table, while Writer2 handles data from source table2. The payload performs a left outer join between source table 1 and source table 2.
Schema of source table1 is
{name: id, type: string},
{name: ts, type : long},
{name: name, type: string}
Schema of source table2 is
{name: id, type: string},
{name: ts, type : long},
{name: price, type: string}
Schema of target wide table is
{name: id, type: string},
{name: ts, type : long},
{name: name, type: string},
{name: price, type: string}
Current data:
id ts name price
1 1 name_1 null
Insert data:
id ts name price
1 2 null price_1
2 2 null price_2
2 3 null price_3
The record of id 2 after `precombine` is '2, 3, null, proce_3', it should be **dropped** because it does not match left outer join syntax. In the previous implementation of `LeftOuterJoinPayload`, the **dropped** behavior is done in `getInsertValue` method by return `EMPTY` to **skip** writing this record.
However, when I tried to implement the logical of `LeftOuterJoinPayload` using the new `HoodieRecordMerger` API, I found that there is no way to achieve the aforementioned **dropped** logic. This is because, after the merge action of `precombine` is done, there is no callback API available to invoke custom logic to decide whether *dropped** the record or not before writing it to storage. I initially thought that `HoodieRecord#shouldIgnore` could help me, but I realized it's not the solution. This is because `HoodieRecord` is a generic implementation related to the compute engine, not something customized by the user based on their special business requirements.
I wonder if I missed anything, or if it's necessary to introduce a new API to implement this capability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1698034173
I've created tasks under the JIRA EPIC for RFC-46 (HUDI-3217) to capture the work around this. Filling the gaps of the functionality around the merge API are: HUDI-6702, HUDI-6765. @linliu-code is looking into these too. @beyond1920 are you creating a POC for your custom payload only, or you're also fixing the merge API and related functionality in the writer/reader paths?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1696712154
Sounds reasonable. Please give me some time, I would create a POC according to the idea, would respond later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1696612674
> @yihua Thanks for response. The solution could work, but not quite elegant, as the merge API takes on two responsibilities. Is there any better options?
Conceptually, the functionality to drop the record before writing to storage, implemented by the `getInsertValue`, is part of the "merging" too, because `getInsertValue` should only be called if the record key is not present on storage (in the table). This fits well into the new `merge` API which depends on `Option<HoodieRecord> older`. All custom logic can be implemented in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1698371622
@yihua Glad to hear that.
I have already fix the merge API and related functionality in the writer/reader paths of **Spark Engine**, not cover other engines yet. And it solves the problem in our internal version.
I originally intended to look into the writer/reader paths of other engines, e.g, Flink, to create a comprehensive POC and contribute it to the community.
If @linliu-code is looking into these, I'm OK with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1700302192
Great! @beyond1920 what's your Slack or Wechat handle? Since @linliu-code is fixing this and making it production ready, three of us should sync and see how we can divide the work here. You can still focus on support the new merger API for other engines like Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1679988009
@beyond1920 Thanks for highlighting the issue. I have created a JIRA to track it - HUDI-6702
> I initially thought that HoodieRecord#shouldIgnore could help me, but I realized it's not the solution. This is because HoodieRecord is a generic implementation related to the compute engine, not something customized by the user based on their special business requirements.
Can you elaborate a bit more on the above? In your custom implementation of `HoodieRecordMerger` aren't you also using a custom implementation of `HoodieRecord` where you can override `shouldIgnore`. This will ultimately be handled in one of `HoodieWriteHandle`s after merging. For example https://github.com/apache/hudi/blob/852bc7c5a650af839b4a0206d17bb8e08273bc79/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java#L286-L288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1695808484
@yihua Thanks for response. The solution could work, but not quite elegant, as the merge API takes on two responsibilities.
Is there any better options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1680204072
@codope I only add a custom implementation of `HoodieRecordMerger`, not introduce custom `HoodieRecord `. I think different `HoodieRecord` implementation is used for different engine-specific (spark, flink ), it's not something customized by the user based on their special business requirements. Please correct if I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1675597664
I agree after RFC-46, we have a function regression for custom payload, I'm +1 to fix it, maybe we can introduce another interface `PostMergingAwareness` and it has a method:
```java
boolean isValidRecord(HoodieRecord record);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1693807306
@beyond1920 IIUC the problem you have at high-level, if the record of a record key does not exist in the current data based on the input (e.g., `id=2` record does not exist in the current data, and `id=2` exists after legacy precombine logic), the new merger API is not called and no customization can be made. If this is the case, we can adjust the merge API in `HoodieRecordMerger ` to:
```
Option<HoodieRecord> merge(Option<HoodieRecord> older, Option<HoodieRecord> newer, Schema schema, Properties props) throws IOException;
```
The changes are to make `older` and `newer` records optional. So that besides calling this for `combineAndGetUpdateValue` logic, when only `newer` HoodieRecord exists (i.e., `older` is empty), we also call this API to allow custom logic, which solves your problem. This requires changes to the new merging logic but it opens up the same functionality and flexibility as before. Then you only need to implement this new record merger API without having to change other classes.
wdyt? Does this work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] beyond1920 commented on issue #9430: [SUPPORT] Problem when refactor a custom payload to new API defined in RFC-46
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on issue #9430:
URL: https://github.com/apache/hudi/issues/9430#issuecomment-1700346159
Cool. @yihua I ping you on Wechat and Slack. I'm Jing Zhang.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org