[jira] [Resolved] (LIVY-863) Missing JVM class imports for Spark3

["Gyorgy Gal (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LIVY-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gyorgy Gal resolved LIVY-863.
-----------------------------
    Resolution: Fixed

> Missing JVM class imports for Spark3
> ------------------------------------
>
>                 Key: LIVY-863
>                 URL: https://issues.apache.org/jira/browse/LIVY-863
>             Project: Livy
>          Issue Type: Bug
>          Components: REPL
>            Reporter: Moritz
>            Assignee: Nilesh Rathi
>            Priority: Major
>             Fix For: 0.9.0
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Livy creates a fake spark shell initiating its own java gateway. This gateway imports some java classes for the user which have diverged from the ones in Spark 3. Some imports are missing which lead to methods like `dataframe.explain()` to be broken:
> {code:java}
> result.explain(True)
> An error was encountered:
> 'JavaPackage' object is not callable
> Traceback (most recent call last):
>   File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 386, in explain
>     print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(), explain_mode))
> TypeError: 'JavaPackage' object is not callable
> {code}
> Spark is importing:
> [https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153]
> Livy is currently importing:
> [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)