[jira] [Created] (LIVY-795) Can't work with PySpark with Python >= 3.8 due to https://bugs.python.org/issue35894

["Shiquan Wang (Jira)" <ji...@apache.org>]

Shiquan Wang created LIVY-795:
---------------------------------

             Summary: Can't work with PySpark with Python >= 3.8 due to https://bugs.python.org/issue35894
                 Key: LIVY-795
                 URL: https://issues.apache.org/jira/browse/LIVY-795
             Project: Livy
          Issue Type: Bug
          Components: REPL
    Affects Versions: 0.7.0
         Environment: EMR 6.1.0
            Reporter: Shiquan Wang


 

When working with PySpark with Python >= 3.8, below error is raised:

 
{code:java}
TypeError: required field "type_ignores" missing from Module{code}
The error points to code in [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L223]

 

The error is caused by an API change in Python 3.8: [https://github.com/python/cpython/pull/11645/files] and was reported here [https://bugs.python.org/issue35894]

IPython has faced similar issue: [https://github.com/ipython/ipython/issues/11590]

IPython solved the issue in this PR: [https://github.com/ipython/ipython/pull/11593]  

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)