You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Yaniv Kunda <ya...@kundas.net> on 2025/06/25 17:08:58 UTC
Deprecations removal schedule
Hi all,
During local builds, I've noticed there are >800 warnings about deprecated
classes/methods.
Is there a planned schedule for the removal of some of them?
At least since Java 9, the Java `@Deprecation` annotation has a
`forRemoval` attribute, which is not used yet - but since Spark 4.x
requires Java 17, maybe it's a good time to start using it.
It's still beneficial to define some sort of an external removal
schedule/policy, whether time-based or version-based - to encourage users
to move away from the deprecated code.
(p.s., the comparable scala annotation doesn't have that attribute).
I've processed the build output (with a bit of manual filling-ins) and
created a table with most of the deprecation warnings, sorted by #
occurrences:
+-------+----------------------------------------------------+--------------------------------------+-------+-----+---------+
| Lang. | What | Where
| since | # | Source |
+-------+----------------------------------------------------+--------------------------------------+-------+-----+---------+
| Scala | class StreamingContext | package
streaming | 3.4.0 | 132 | |
| Scala | JavaStreamingContext |
org.apache.spark.streaming.api.java | 3.4.0 | 58 | |
| Scala | method Once | class
Trigger | 3.4.0 | 25 | |
| Scala | class UserDefinedAggregateFunction | package
expressions | 3.0.0 | 17 | |
| Scala | object StreamingContext | package
streaming | 3.4.0 | 15 | |
| Scala | class ChiSqSelector | package
feature | 3.1.1 | 14 | |
| Scala | class JavaStreamingContext | package java
| 3.4.0 | 14 | |
| Scala | class SparkListenerExecutorBlacklisted | package
scheduler | 3.1.0 | 13 | |
| Scala | method createExternalTable | class
SQLContext | 2.2.0 | 12 | |
| Scala | method load | class
SQLContext | 1.4.0 | 12 | |
| Scala | class SparkListenerExecutorBlacklistedForStage | package
scheduler | 3.1.0 | 11 | |
| Scala | class SparkListenerExecutorUnblacklisted | package
scheduler | 3.1.0 | 11 | |
| Scala | method schema | trait Table
| 3.4.0 | 11 | |
| Scala | class SparkListenerExecutorUnblacklistedForStage | package
scheduler | 3.1.0 | 10 | |
| Scala | class SparkListenerNodeBlacklistedForStage | package
scheduler | 3.1.0 | 10 | |
| Scala | method jsonRDD | class
SQLContext | 1.4.0 | 10 | |
| Scala | value blacklistedInStages | class
ExecutorSummary | 3.1.0 | 10 | |
| Scala | class SparkListenerNodeBlacklisted | package
scheduler | 3.1.0 | 9 | |
| Scala | value isBlacklisted | class
ExecutorSummary | 3.1.0 | 9 | |
| Scala | method applySchema | class
SQLContext | 1.3.0 | 8 | |
| Scala | class SparkListenerNodeUnblacklisted | package
scheduler | 3.1.0 | 7 | |
| Scala | method toDegrees | object
functions | 2.1.0 | 7 | |
| Scala | method toRadians | object
functions | 2.1.0 | 7 | |
| Scala | method jsonFile | class
SQLContext | 1.4.0 | 6 | |
| Scala | method jdbc | class
SQLContext | 1.4.0 | 6 | |
| Scala | method registerTempTable | class
Dataset | 2.0.0 | 6 | |
| Scala | method udf | object
functions | 3.0.0 | 6 | |
| Scala | method register | class
UDFRegistration | 3.0.0 | 6 | |
| Scala | value isBlacklistedForStage | class
ExecutorStageSummary | 3.1.0 | 5 | |
| Scala | method approxCountDistinct | object
functions | 2.1.0 | 5 | |
| Scala | method explode | class
Dataset | 2.0.0 | 4 | |
| Scala | method explode | class
Dataset | 3.5.0 | 4 | |
| Java | createTable(Identifier,Column[],Transform[],Map) | TableCatalog
| 4.1.0 | 4 | |
| Java | createTable(Identifier,StructType,Transform[],Map) | TableCatalog
| 3.4.0 | 4 | |
| Java | SparkListenerExecutorBlacklisted |
org.apache.spark.scheduler | 3.1.0 | 3 | |
| Java | SparkListenerExecutorBlacklistedForStage |
org.apache.spark.scheduler | 3.1.0 | 3 | |
| Java | SparkListenerExecutorUnblacklisted |
org.apache.spark.scheduler | 3.1.0 | 3 | |
| Java | SparkListenerNodeBlacklisted |
org.apache.spark.scheduler | 3.1.0 | 3 | |
| Java | SparkListenerNodeBlacklistedForStage |
org.apache.spark.scheduler | 3.1.0 | 3 | |
| Java | SparkListenerNodeUnblacklisted |
org.apache.spark.scheduler | 3.1.0 | 3 | |
| Scala | method !== | class Column
| 2.0.0 | 3 | |
| Scala | method createExternalTable | class
Catalog | 2.2.0 | 3 | |
| Scala | method newTaskTempFile | trait
FileCommitProtocol | 3.3.0 | 2 | |
| Scala | DEPRECATED_CHILD_CONNECTION_TIMEOUT |
SparkLauncher | 3.2.0 | 2 | |
| Java | DEPRECATED_CHILD_CONNECTION_TIMEOUT |
SparkLauncher | 3.2.0 | 2 | |
| Scala | method newTaskTempFileAbsPath | trait
FileCommitProtocol | 3.3.0 | 1 | |
| Scala | value holdingLocks | class
ThreadStackTrace | 4.0.0 | 1 | |
| Java | AppStatusSource.BLACKLISTED_EXECUTORS |
AppStatusSource | 3.1.0 | 1 | |
| Java | AppStatusSource.UNBLACKLISTED_EXECUTORS |
AppStatusSource | 3.1.0 | 1 | |
| Scala | method classifyException | class
JdbcDialect | 4.0.0 | 1 | |
| Scala | string sql | message
SqlCommand | | 5 | proto |
| Scala | trait AggregationBuffer | class
GenericUDAFEvaluator | | 16 | hive |
| Scala | method initialize | class
AbstractSerDe | | 1 | hive |
| Scala | method poll(long) | trait
Consumer | | 5 | kafka |
| Scala | class DefaultPartitioner | package
internals | | 3 | kafka |
| Scala | method getAllStatistics | class
FileSystem | | 3 | hadoop |
| Java | ParquetFileReader(Path,ParquetReadOptions) |
ParquetFileReader | | 3 | parquet |
| Java | ParquetFileReader(Path,ParquetReadOptions,List) |
ParquetFileReader | | 3 | parquet |
| Java | ParquetFileReader(Configuration,List,List,boolean) |
ParquetFileReader | | 3 | parquet |
| Java | readFooter(Configuration,Path) |
ParquetFileReader | | 3 | parquet |
| Java | BytesInput toByteArray() | BytesInput
| | 1 | parquet |
| Java | AvroParquetWriter builder(Path) |
AvroParquetWriter | | 1 | parquet |
| Scala | method readAllFootersInParallel | class
ParquetFileReader | | 1 | parquet |
| Java | RandomStringUtils random(int) |
RandomStringUtils | | 3 | c-lang3 |
| Java | RandomStringUtils randomAlphabetic(int) |
RandomStringUtils | | 3 | c-lang3 |
| Scala | method randomAlphanumeric | class
RandomStringUtils | | 1 | c-lang3 |
+-------+----------------------------------------------------+--------------------------------------+-------+-----+---------+
Most are spark's own deprecations - per your input, I can create issues and
PRs to handle specific areas.
Regarding the 3rd-party deprecations (at the end of the table) - most
(maybe all besides parquet) would be trivial to fix, for which I can create
separate tasks.
Re: Deprecations removal schedule
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you for the head-up with the summarization, Yaniv.
> Is there a planned schedule for the removal of some of them?
In general, we are very reluctant when it comes to the removals. Even for the deprecated ones.
https://spark.apache.org/versioning-policy.html
It should be considered case by case instead of marking the removal plans in a batch way. In other words, I'm not sure we want to remove those deprecated ones in the next Spark releases. At least, I believe those deprecated ones will be there for the lifetime of Spark 4.x for next 5+ years.
Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
Spark 3: 2020.06 (3.0.0) ~ 2026.04 (3.5.x)
Spark 4: 2025.02 (4.0.0) ~ 2031?
Maybe, we want to revisit this topic when we start to discuss Apache Spark 5 someday.
Best,
Dongjoon.
On 2025/06/25 17:08:58 Yaniv Kunda wrote:
> Hi all,
> During local builds, I've noticed there are >800 warnings about deprecated
> classes/methods.
> Is there a planned schedule for the removal of some of them?
>
> At least since Java 9, the Java `@Deprecation` annotation has a
> `forRemoval` attribute, which is not used yet - but since Spark 4.x
> requires Java 17, maybe it's a good time to start using it.
> It's still beneficial to define some sort of an external removal
> schedule/policy, whether time-based or version-based - to encourage users
> to move away from the deprecated code.
> (p.s., the comparable scala annotation doesn't have that attribute).
>
> I've processed the build output (with a bit of manual filling-ins) and
> created a table with most of the deprecation warnings, sorted by #
> occurrences:
> +-------+----------------------------------------------------+--------------------------------------+-------+-----+---------+
> | Lang. | What | Where
> | since | # | Source |
> +-------+----------------------------------------------------+--------------------------------------+-------+-----+---------+
> | Scala | class StreamingContext | package
> streaming | 3.4.0 | 132 | |
> | Scala | JavaStreamingContext |
> org.apache.spark.streaming.api.java | 3.4.0 | 58 | |
> | Scala | method Once | class
> Trigger | 3.4.0 | 25 | |
> | Scala | class UserDefinedAggregateFunction | package
> expressions | 3.0.0 | 17 | |
> | Scala | object StreamingContext | package
> streaming | 3.4.0 | 15 | |
> | Scala | class ChiSqSelector | package
> feature | 3.1.1 | 14 | |
> | Scala | class JavaStreamingContext | package java
> | 3.4.0 | 14 | |
> | Scala | class SparkListenerExecutorBlacklisted | package
> scheduler | 3.1.0 | 13 | |
> | Scala | method createExternalTable | class
> SQLContext | 2.2.0 | 12 | |
> | Scala | method load | class
> SQLContext | 1.4.0 | 12 | |
> | Scala | class SparkListenerExecutorBlacklistedForStage | package
> scheduler | 3.1.0 | 11 | |
> | Scala | class SparkListenerExecutorUnblacklisted | package
> scheduler | 3.1.0 | 11 | |
> | Scala | method schema | trait Table
> | 3.4.0 | 11 | |
> | Scala | class SparkListenerExecutorUnblacklistedForStage | package
> scheduler | 3.1.0 | 10 | |
> | Scala | class SparkListenerNodeBlacklistedForStage | package
> scheduler | 3.1.0 | 10 | |
> | Scala | method jsonRDD | class
> SQLContext | 1.4.0 | 10 | |
> | Scala | value blacklistedInStages | class
> ExecutorSummary | 3.1.0 | 10 | |
> | Scala | class SparkListenerNodeBlacklisted | package
> scheduler | 3.1.0 | 9 | |
> | Scala | value isBlacklisted | class
> ExecutorSummary | 3.1.0 | 9 | |
> | Scala | method applySchema | class
> SQLContext | 1.3.0 | 8 | |
> | Scala | class SparkListenerNodeUnblacklisted | package
> scheduler | 3.1.0 | 7 | |
> | Scala | method toDegrees | object
> functions | 2.1.0 | 7 | |
> | Scala | method toRadians | object
> functions | 2.1.0 | 7 | |
> | Scala | method jsonFile | class
> SQLContext | 1.4.0 | 6 | |
> | Scala | method jdbc | class
> SQLContext | 1.4.0 | 6 | |
> | Scala | method registerTempTable | class
> Dataset | 2.0.0 | 6 | |
> | Scala | method udf | object
> functions | 3.0.0 | 6 | |
> | Scala | method register | class
> UDFRegistration | 3.0.0 | 6 | |
> | Scala | value isBlacklistedForStage | class
> ExecutorStageSummary | 3.1.0 | 5 | |
> | Scala | method approxCountDistinct | object
> functions | 2.1.0 | 5 | |
> | Scala | method explode | class
> Dataset | 2.0.0 | 4 | |
> | Scala | method explode | class
> Dataset | 3.5.0 | 4 | |
> | Java | createTable(Identifier,Column[],Transform[],Map) | TableCatalog
> | 4.1.0 | 4 | |
> | Java | createTable(Identifier,StructType,Transform[],Map) | TableCatalog
> | 3.4.0 | 4 | |
> | Java | SparkListenerExecutorBlacklisted |
> org.apache.spark.scheduler | 3.1.0 | 3 | |
> | Java | SparkListenerExecutorBlacklistedForStage |
> org.apache.spark.scheduler | 3.1.0 | 3 | |
> | Java | SparkListenerExecutorUnblacklisted |
> org.apache.spark.scheduler | 3.1.0 | 3 | |
> | Java | SparkListenerNodeBlacklisted |
> org.apache.spark.scheduler | 3.1.0 | 3 | |
> | Java | SparkListenerNodeBlacklistedForStage |
> org.apache.spark.scheduler | 3.1.0 | 3 | |
> | Java | SparkListenerNodeUnblacklisted |
> org.apache.spark.scheduler | 3.1.0 | 3 | |
> | Scala | method !== | class Column
> | 2.0.0 | 3 | |
> | Scala | method createExternalTable | class
> Catalog | 2.2.0 | 3 | |
> | Scala | method newTaskTempFile | trait
> FileCommitProtocol | 3.3.0 | 2 | |
> | Scala | DEPRECATED_CHILD_CONNECTION_TIMEOUT |
> SparkLauncher | 3.2.0 | 2 | |
> | Java | DEPRECATED_CHILD_CONNECTION_TIMEOUT |
> SparkLauncher | 3.2.0 | 2 | |
> | Scala | method newTaskTempFileAbsPath | trait
> FileCommitProtocol | 3.3.0 | 1 | |
> | Scala | value holdingLocks | class
> ThreadStackTrace | 4.0.0 | 1 | |
> | Java | AppStatusSource.BLACKLISTED_EXECUTORS |
> AppStatusSource | 3.1.0 | 1 | |
> | Java | AppStatusSource.UNBLACKLISTED_EXECUTORS |
> AppStatusSource | 3.1.0 | 1 | |
> | Scala | method classifyException | class
> JdbcDialect | 4.0.0 | 1 | |
> | Scala | string sql | message
> SqlCommand | | 5 | proto |
> | Scala | trait AggregationBuffer | class
> GenericUDAFEvaluator | | 16 | hive |
> | Scala | method initialize | class
> AbstractSerDe | | 1 | hive |
> | Scala | method poll(long) | trait
> Consumer | | 5 | kafka |
> | Scala | class DefaultPartitioner | package
> internals | | 3 | kafka |
> | Scala | method getAllStatistics | class
> FileSystem | | 3 | hadoop |
> | Java | ParquetFileReader(Path,ParquetReadOptions) |
> ParquetFileReader | | 3 | parquet |
> | Java | ParquetFileReader(Path,ParquetReadOptions,List) |
> ParquetFileReader | | 3 | parquet |
> | Java | ParquetFileReader(Configuration,List,List,boolean) |
> ParquetFileReader | | 3 | parquet |
> | Java | readFooter(Configuration,Path) |
> ParquetFileReader | | 3 | parquet |
> | Java | BytesInput toByteArray() | BytesInput
> | | 1 | parquet |
> | Java | AvroParquetWriter builder(Path) |
> AvroParquetWriter | | 1 | parquet |
> | Scala | method readAllFootersInParallel | class
> ParquetFileReader | | 1 | parquet |
> | Java | RandomStringUtils random(int) |
> RandomStringUtils | | 3 | c-lang3 |
> | Java | RandomStringUtils randomAlphabetic(int) |
> RandomStringUtils | | 3 | c-lang3 |
> | Scala | method randomAlphanumeric | class
> RandomStringUtils | | 1 | c-lang3 |
> +-------+----------------------------------------------------+--------------------------------------+-------+-----+---------+
>
> Most are spark's own deprecations - per your input, I can create issues and
> PRs to handle specific areas.
> Regarding the 3rd-party deprecations (at the end of the table) - most
> (maybe all besides parquet) would be trivial to fix, for which I can create
> separate tasks.
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org