You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "dinesh (Jira)" <ji...@apache.org> on 2020/09/24 18:37:00 UTC
[jira] [Created] (SPARK-32988) Spark 2.3 and 2.4 backward
compatibility problems
dinesh created SPARK-32988:
------------------------------
Summary: Spark 2.3 and 2.4 backward compatibility problems
Key: SPARK-32988
URL: https://issues.apache.org/jira/browse/SPARK-32988
Project: Spark
Issue Type: Bug
Components: Java API
Affects Versions: 2.4.0, 2.3.2
Reporter: dinesh
With Spark 2.3
spark.sharedState.externalCatalog used to give an object of ExternalCatalog which was actually an instance of HiveExternalCatalog
With Spark 2.4
spark.sharedState.externalCatalog gives an object of ExternalCatalogWithListener which has HiveExternalCatalog as unwrapped delegate.
We use HiveExternalCatalog to get HiveClient to perform Hive Operations.
This change in method Signature is giving us backward compatibility problems such that code compiled with Spark 2.3 can not work with 2.4 due to the change. This results in Runtime exceptions including ClassCastException, NoSuchMethodError
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org