[jira] [Created] (SPARK-31363) Improve DataSourceRegister interface

["Andrew Malone Melo (Jira)" <ji...@apache.org>]

Andrew Malone Melo created SPARK-31363:
------------------------------------------

             Summary: Improve DataSourceRegister interface
                 Key: SPARK-31363
                 URL: https://issues.apache.org/jira/browse/SPARK-31363
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.5, 3.0.0
            Reporter: Andrew Malone Melo


As the DSv2 API evolves, some breaking changes are occasionally made to the API. It's possible to split a plugin into a "common" part and multiple version-specific parts and this works good to have a single artifact for users. The one part that can't be currently worked around is the DataSourceRegister trait. This is an issue because users cargo-cult configuration values, and choosing the wrong plugin version gives a particularly baroque error message that bubbles up through ServiceLoader.

Currently, the class implementing DataSourceRegister must also be the class implementing the "toplevel" DataSourceV2 interface (and mixins), and these various interfaces occasionally change as the API evolves. As a practical matter, this means that there's no opportunity to decide at runtime which class to pass along to Spark. Attempting to add multiple DataSourceV2 implementations to services/META-INF causes an exception when the ServiceLoader tries to load the DataSourceRegister who implements the "different" DataSourceV2.

I would like to propose a new DataSourceRegister interface which adds a level of indirection between the what ServiceLoader and DataSourceV2 loads. E.g. (strawman)

{{interface DataSourceRegisterV2 {
  public String shortName();
  public Class<? implements DataSourceV2> getImplementation();
}}}

Then org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource would have its search algorithm extended to look for DataSourceRegisterV2 objects, and if one is located for the given shortName, return the class object from getImplementation(). At this point, the plugin could decide based on the current runtime environment which class to prevent to Spark. There wouldn't be any changes to plugins who don't implement this API.

If this is an acceptable idea, I can put together a PR for further comment.

Thanks
Andrew



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org