[jira] [Closed] (HUDI-5707) Support offset reset strategy w/ spark streaming read from hudi table

["kazdy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

kazdy closed HUDI-5707.
-----------------------
    Fix Version/s:     (was: 1.0.0)
       Resolution: Won't Fix

> Support offset reset strategy w/ spark streaming read from hudi table
> ---------------------------------------------------------------------
>
>                 Key: HUDI-5707
>                 URL: https://issues.apache.org/jira/browse/HUDI-5707
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: reader-core
>            Reporter: sivabalan narayanan
>            Assignee: kazdy
>            Priority: Major
>
> For users reading hudi table in a streaming manner, we need to support offset reset strategy if the commit of interest it archived or cleaned up. 
>  
> notes from the issue 
> In streaming read, user might want to get all incremental changes. from what I see, this is nothing but an incremental query on a hudi table. w/ incremental query, we do have fallback mechanism via {{{}hoodie.datasource.read.incr.fallback.fulltablescan.enable{}}}.
> But in streaming read, the amount of data read might spike up(if we plan to do the same) and the user may not have provisioned higher resources for the job.
> I am thinking, if we should add something like {{auto.offset.reset}} we have in kafka. If you know if we have something similar in streaming read from spark itself, we can leverage the same or add a new config in hoodie.
> So, users can configure what they want to do in such cases:
>  # whether they wish to resume reading from earliest valid commit from hudi.
> // impl might be involved. since we need to dedect the commit which hasn't been cleaned by the cleaner yet.
>  # Or do snapshot query w/ latest table state.
>  # Fail the streaming read.
>  #  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)