You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/02/19 21:30:00 UTC
[jira] [Commented] (SPARK-20971) Purge the metadata log for
FileStreamSource
[ https://issues.apache.org/jira/browse/SPARK-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772351#comment-16772351 ]
Jungtaek Lim commented on SPARK-20971:
--------------------------------------
I guess this can be also handled in SPARK-20568 since SPARK-20568 touches same code path.
> Purge the metadata log for FileStreamSource
> -------------------------------------------
>
> Key: SPARK-20971
> URL: https://issues.apache.org/jira/browse/SPARK-20971
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.1.1
> Reporter: Shixiong Zhu
> Priority: Major
>
> Currently [FileStreamSource.commit|https://github.com/apache/spark/blob/16186cdcbce1a2ec8f839c550e6b571bf5dc2692/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala#L258] is empty. We can delete unused metadata logs in this method to reduce the size of log files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org