You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yongster (via GitHub)" <gi...@apache.org> on 2024/07/23 16:35:52 UTC
[I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
yongster opened a new issue, #11676:
URL: https://github.com/apache/hudi/issues/11676
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
-yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- yes
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Currently, the Flink engine does not support RECORD_INDEX. We are using the Flink engine for writing and the Spark engine for querying, requiring extremely high point differential efficiency. Is there any other way to add RECORD_INDEX to this table, or is it possible to enhance the Flink engine to support RECORD_INDEX? If so, what are the steps to submit this feature? This is my first time contributing code to the community. Thank you.
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.15.1
* Spark version : 3.3.1
* Flink Version: 1.17.1
* Hive version : 3.x
* Hadoop version : 3.1.3
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2246685843
The main reason that Flink does not support RLI is because RLI is kind of costly for committing to the Hudi table, while in streaming ingestion, the handling of the checkpoints ack (the Hudi instants are committed in this process) expect to be short in time, otherwise it may induce instability to the pipeline.
Do you want the RLI to speed up queries or the ingestion upsert?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2246963521
@yongster Do you have a use case to do point lookup queries? (select by filtering record keys) . RLI will only improve performance for that type of queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2248388418
@yongster Yes, RLI can definitely improve the performance for these type of queries. You can try running indexing separately with spark engine like here - https://hudi.apache.org/docs/metadata_indexing/#execute-indexing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "yongster (via GitHub)" <gi...@apache.org>.
yongster commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2247053310
> er Do you have a use case to do point lookup queries? (select by filtering record keys) . RLI will only improve performance for that type of queries.
select xxxx from table where user_id = xxx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "yongster (via GitHub)" <gi...@apache.org>.
yongster commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2246742149
> The main reason that Flink does not support RLI is because RLI is kind of costly for committing to the Hudi table, while in streaming ingestion, the handling of the checkpoints ack (the Hudi instants are committed in this process) expect to be short in time, otherwise it may induce instability to the pipeline.
>
> Do you want the RLI to speed up queries or the ingestion upsert?
Currently, I have a table in DW. It's written into by Flink consuming from Kafka. Users need to query it through Spark engine. The amount of data is very large, so I want to add RLI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2254797046
not started, you can contribute anytime you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "yongster (via GitHub)" <gi...@apache.org>.
yongster commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2254384416
Has the RLI functionality of Flink already started? How can I get involved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Flink engine support RECORD_INDEX [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11676:
URL: https://github.com/apache/hudi/issues/11676#issuecomment-2251768794
@yongster Got it, we have support primary key filtering for bucket index for both Flink and Spark query engine, as for RLI, the data pruning on query side is still on the way, might be supported in release 1.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org