You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/01/25 06:43:00 UTC
[jira] [Assigned] (SPARK-38015) Mark legacy file naming functions as deprecated in FileCommitProtocol
[ https://issues.apache.org/jira/browse/SPARK-38015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-38015:
------------------------------------
Assignee: (was: Apache Spark)
> Mark legacy file naming functions as deprecated in FileCommitProtocol
> ---------------------------------------------------------------------
>
> Key: SPARK-38015
> URL: https://issues.apache.org/jira/browse/SPARK-38015
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.3.0
> Reporter: Cheng Su
> Priority: Minor
>
> [FileCommitProtocol|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala] is the class to commit Spark job output (staging file & directory renaming, etc). During Spark 3.2 development, we added new functions into this class to allow more flexible output file naming (the PR detail is [here|https://github.com/apache/spark/pull/33012]). We didn’t delete the existing file naming functions (newTaskTempFile(ext) & newTaskTempFileAbsPath(ext)), because we were aware of many other downstream projects or codebases already implemented their own custom implementation for FileCommitProtocol. Delete the existing functions would be a breaking change for them when upgrading Spark version, and we would like to avoid this unpleasant surprise for anyone if possible. But we also need to clean up legacy as we evolve our codebase.
> So for next step, I would like to propose:
> * Spark 3.3 (now): Add @deprecate annotation to legacy functions in FileCommitProtocol - [newTaskTempFile(ext)|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala#L98] & [newTaskTempFileAbsPath(ext)|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala#L135].
> * Next Spark major release (or whenever people feel comfortable): delete the legacy functions mentioned above from our codebase.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org