[jira] [Commented] (LIVY-863) Missing JVM class imports for Spark3

["Moritz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LIVY-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366469#comment-17366469 ] 

Moritz commented on LIVY-863:
-----------------------------

I proposed a fix here, it's missing a test and potentially needs some code to account for different spark versions.

If you can help me with this and give some guidance, that would be great and I can add the additional things :)

 

https://github.com/apache/incubator-livy/pull/322

> Missing JVM class imports for Spark3
> ------------------------------------
>
>                 Key: LIVY-863
>                 URL: https://issues.apache.org/jira/browse/LIVY-863
>             Project: Livy
>          Issue Type: Bug
>          Components: REPL
>            Reporter: Moritz
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Livy creates a fake spark shell initiating its own java gateway. This gateway imports some java classes for the user which have diverged from the ones in Spark 3. Some imports are missing which lead to methods like `dataframe.explain()` to be broken:
> {code:java}
> result.explain(True)
> An error was encountered:
> 'JavaPackage' object is not callable
> Traceback (most recent call last):
>   File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 386, in explain
>     print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(), explain_mode))
> TypeError: 'JavaPackage' object is not callable
> {code}
> Spark is importing:
> [https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153]
> Livy is currently importing:
> [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)