[jira] [Updated] (HUDI-3679) Spark Compatibility Model

["Raymond Xu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raymond Xu updated HUDI-3679:
-----------------------------
    Epic Status: Done  (was: To Do)

> Spark Compatibility Model
> -------------------------
>
>                 Key: HUDI-3679
>                 URL: https://issues.apache.org/jira/browse/HUDI-3679
>             Project: Apache Hudi
>          Issue Type: Epic
>          Components: dependencies
>            Reporter: Alexey Kudinkin
>            Assignee: Alexey Kudinkin
>            Priority: Blocker
>             Fix For: 0.11.0
>
>
> Currently we have inconsistent models of aliasing Spark version in the bundle name ("spark3", "spark3.1.x", "spark3.2.1", etc).
> To avoid confusion in the community, and make things consistent following compatibility pledge is proposed: 
> +Hudi bundles will stay compatible with _all patch versions_ of Spark w/in single minor version branch (ie Spark 3.1.x, 3.2.x, etc)+
>  
> Such compatibility model will be correspondingly reflected in the bundle names: corresponding spark minor version branch will be aliased in the bundle name to signal its compatibility with Spark versions w/in such branch ("spark3.1" for Spark 3.1.x, etc)
> To transition to the new model following steps are proposed:
>  # Add new artifacts carrying Spark minor versions in the bundle names (for ex, "spark3.2", "spark3.1", "spark2.4") in 0.11
>  # Deprecate single and triple digit bundles ("spark3", "spark2", "spark3.2.1") in favor of double-digit in 0.11
>  # Discontinue deprecated bundles (spark3, spark2, etc) from 0.12
>  
> As such following bundles will be released in 0.11
>  * "spark2.4" (Scala 2.11/2.12)
>  * "spark3.1"
>  * "spark3.2"
>  * "spark3" (this is the same as "spark3.2")
>  * "spark2" (Scala 2.11/2.12, this is the same as "spark2.4")



--
This message was sent by Atlassian Jira
(v8.20.7#820007)