You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Geser Dugarov (Jira)" <ji...@apache.org> on 2025/04/30 09:10:00 UTC

[jira] [Commented] (HUDI-9192) [RFC-87] RowData log handle supports writing record positions to log block

    [ https://issues.apache.org/jira/browse/HUDI-9192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17948406#comment-17948406 ] 

Geser Dugarov commented on HUDI-9192:
-------------------------------------

I've raised more questions then figured out how to implement it properly during research of this task.

 

Initially, Flink integration doesn't support record positions processing even for `HoodieRecord`s. But I didn't found any explicit description of how "log records positions" feature should work. So, we have to use Spark implementation as an example.

Writing of log records positions and using them are separated between different PRs, and were implemented by different developers:

1) additional property in log block header (without writing): [https://github.com/apache/hudi/pull/9376]

2) writing log record positions in log block header: [https://github.com/apache/hudi/pull/9581]

3) using of log record positions from log block headers in file group readers: [https://github.com/apache/hudi/pull/9819]

 

Confusing part here, that we write log record positions as a set in `HoodieLogBlock::addRecordPositionsIfRequired`:

[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java#L390-L392]

But later extract them as a list in `PositionBasedFileGroupRecordBuffer::extractRecordPositions`:

[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java#L305-L307]

and use them as a list in `PositionBasedFileGroupRecordBuffer::processDataBlock`:

[https://github.com/apache/hudi/blob/6f84c401b3a809997be1573b0d04e8106fd87fac/hudi-common/src/main/java/org/apache/hudi/common/table/read/PositionBasedFileGroupRecordBuffer.java#L132-L136]

 

> [RFC-87] RowData log handle supports writing record positions to log block
> --------------------------------------------------------------------------
>
>                 Key: HUDI-9192
>                 URL: https://issues.apache.org/jira/browse/HUDI-9192
>             Project: Apache Hudi
>          Issue Type: Sub-task
>          Components: flink-sql
>            Reporter: Shuo Cheng
>            Assignee: Geser Dugarov
>            Priority: Major
>
> Flink writer do not support record-position for updates and deletes yet, will support it later.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)