[jira] [Comment Edited] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

["Nadav Samet (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484941#comment-17484941 ] 

Nadav Samet edited comment on SPARK-38077 at 1/31/22, 9:14 PM:
---------------------------------------------------------------

I wasn't aware that the policy allows for it, but I'd like to express the impact on breaking binary-compatibility on _maintenance_ releases for library maintainers and ultimately end-users, in hope that it will be reconsidered.

A breakage like this one, doesn't allow a library author to ship a single artifact that works with both Spark 3.2.0 and Spark 3.2.1. Maintainers have to create a build for each version. Since sparksql-scalapb (which I maintain) calls {{Invoke}} directly (it does it through frameless), frameless will need to produce two versions of their library (for 3.2.0 and 3.2.1)

Today, we already have to cut a release for feature releases (3.0.x, 3.1.x and 3.2.x), see the [table here|#setting-up-your-project],] and the consequence of this policy is that we also have to cut a release for maintenance versions too, and that release would be incompatible with 3.2.0.


was (Author: thesamet):
I wasn't aware that the policy allows for it, but I'd like to express the impact on breaking binary-compatibility on _maintenance_ releases for library maintainers and ultimately end-users, in hope that it will be reconsidered.

A breakage like this one, doesn't allow a library author to ship a single artifact that works with both Spark 3.2.0 and Spark 3.2.1. Maintainers have to create a build for each version. Since my library does call Invoke directly (it does it through frameless), frameless will need to produce two versions of their library (for 3.2.0 and 3.2.1)

Today, we already have to cut a release for feature releases (3.0.x, 3.1.x and 3.2.x), see the [table here|[https://scalapb.github.io/docs/sparksql#setting-up-your-project],] and the consequence of this policy is that we also have to cut a release for maintenance versions too, and that release would be incompatible with 3.2.0.

> Spark 3.2.1 breaks binary compatibility with Spark 3.2.0
> --------------------------------------------------------
>
>                 Key: SPARK-38077
>                 URL: https://issues.apache.org/jira/browse/SPARK-38077
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 3.2.1
>            Reporter: Nadav Samet
>            Priority: Major
>
> [PR 35243|https://github.com/apache/spark/pull/35243] introduced a new parameter to class `Invoke` with a default value (`isDeterministic: Boolean = true`). Existing Spark libraries (such as [frameless|https://github.com/typelevel/frameless]) that invoke [Invoke|https://github.com/typelevel/frameless/blob/29961d549e332dddf5cd711ef699dde7460cc48a/dataset/src/main/scala/frameless/RecordEncoder.scala#L154] directly expect a method with 7 parameters, and the new version expects 8. If Frameless would recompile with Spark 3.2.1, the updated library will NOT be binary compatible with Spark 3.2.0. Adding default parameters to existing methods [should be avoided|https://github.com/jatcwang/binary-compatibility-guide#dont-adding-parameters-with-default-values-to-methods].
> One way forward would be to revert the change in the constructor and introduce a second constructor or a companion method that takes all the 8 parameters.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org