[jira] [Created] (HBASE-30178) Add support to spark 4 in hbase-connectors

["Wellington Chevreuil (Jira)" <ji...@apache.org>]

Wellington Chevreuil created HBASE-30178:
--------------------------------------------

             Summary: Add support to spark 4 in hbase-connectors
                 Key: HBASE-30178
                 URL: https://issues.apache.org/jira/browse/HBASE-30178
             Project: HBase
          Issue Type: Umbrella
          Components: hbase-connectors
            Reporter: Wellington Chevreuil
            Assignee: Wellington Chevreuil


With spark 4.0.0 GA for around a year now, we should start working on making hbase-connectors integrate with spark 4.

Since we would still need to support spark 3 for a while and due to spark 4 requiring minimum jdk17 and scala 2.13, it would be easier to define a specific spark 4 module under the hbase-connectors root, so that spark.version, scala.version, compiler flags, and optional spark-4 only APIs never collide with spark 3 in a single parent pom. 

We should keep feature parity as much as possible. Datasources, context, rdd and connection cache related logic can be duplicated under this new hbase-spark4 module, in order to compile and package with jdk17 and scala 2.13. This module build should be isolated on a specific maven profile, so it wouldn't be build by default. 

Pushdown filter related classes and filter protocol would remain on a common module (no code duplication needed).

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)