[jira] [Created] (SPARK-28784) StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories

["Shruti Gumma (Jira)" <ji...@apache.org>]

Shruti Gumma created SPARK-28784:
------------------------------------

             Summary: StreamExecution and StreamingQueryManager should utilize CheckpointFileManager to interact with checkpoint directories
                 Key: SPARK-28784
                 URL: https://issues.apache.org/jira/browse/SPARK-28784
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Shruti Gumma


After PR [https://github.com/apache/spark/pull/21048], the CheckpointFileManager interface was created to handle all structured streaming checkpointing operations and helps users to choose how they wish to write checkpointing files atomically.
StreamExecution and StreamingQueryManager still uses some FileSystem operations without using the CheckpointFileManager.
For instance,
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L137]
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L392]

Instead, StreamExecution and StreamingQueryManager should use CheckpointFileManager for these operations.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org