You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "kazdy (Jira)" <ji...@apache.org> on 2022/09/27 21:20:00 UTC

[jira] [Created] (HUDI-4933) sourceLimit and begin.instanttime for spark structured streaming

kazdy created HUDI-4933:
---------------------------

             Summary: sourceLimit and begin.instanttime for spark structured streaming
                 Key: HUDI-4933
                 URL: https://issues.apache.org/jira/browse/HUDI-4933
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: kazdy
            Assignee: kazdy
             Fix For: 0.13.0


Add config for sourceLimit for spark structured streaming to match deltastreamer sourceLimit option for incremental hudi source.

Deltastreamer implementation:
[https://github.com/apache/hudi/blob/801fdab55c97d1abd815f90ae64c0ecbe8e84f94/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java]

Add config for structured streaming so that users can pass begin.instanttime to start strudctred streaming query from given instanttime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)