[jira] [Resolved] (HUDI-4769) Option read.streaming.skip_compaction skips delta commit

["Danny Chen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Danny Chen resolved HUDI-4769.
------------------------------

> Option read.streaming.skip_compaction skips delta commit
> --------------------------------------------------------
>
>                 Key: HUDI-4769
>                 URL: https://issues.apache.org/jira/browse/HUDI-4769
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: flink, flink-sql
>            Reporter: nonggia.liang
>            Assignee: Sagar Sumit
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 0.13.0
>
>
> Option read.streaming.skip_compaction was introduced to avoid consuming duplicate data from delta-commits and compactions in MOR table.
> But the option may cause delta-commits, here the case:
> Support we have a timeline (d for delta-commit, C for compaction/commit):
> d1 --> d2 --> C3 --> d4 --> d5 -->
> t1.......................................................t2..........
> Let's say scans for streaming read happen at time t1 and t2, when d1 and d5 is the latest instant seperately. 
> When we scan at t2 with read.streaming.skip_compaction=true, we get a latest merged fileslice with only log files containing d4+d5.  So d2 is skipped.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)