You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2021/11/26 07:45:00 UTC

[jira] [Created] (HUDI-2865) Enable timeline-server-based markers for Spark structured streaming

Ethan Guo created HUDI-2865:
-------------------------------

             Summary: Enable timeline-server-based markers for Spark structured streaming
                 Key: HUDI-2865
                 URL: https://issues.apache.org/jira/browse/HUDI-2865
             Project: Apache Hudi
          Issue Type: Sub-task
            Reporter: Ethan Guo


For Spark structured streaming, after the end of first micro-batch, the write client is closed and hence triggers closure of timeline service. But subsequent micro-batches do succeed though.

Currently, we explicitly override the marker type to be DIRECT in Spark structured streaming.  We need to revisit the behavior of timeline service before we can enable timeline-server-based markers for Spark structured streaming.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)