You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Larry McCay (Jira)" <ji...@apache.org> on 2022/12/27 05:59:00 UTC
[jira] [Resolved] (LIVY-795) Can't work with PySpark with Python >= 3.8 due to https://bugs.python.org/issue35894
[ https://issues.apache.org/jira/browse/LIVY-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Larry McCay resolved LIVY-795.
------------------------------
Resolution: Fixed
> Can't work with PySpark with Python >= 3.8 due to https://bugs.python.org/issue35894
> ------------------------------------------------------------------------------------
>
> Key: LIVY-795
> URL: https://issues.apache.org/jira/browse/LIVY-795
> Project: Livy
> Issue Type: Bug
> Components: REPL
> Affects Versions: 0.7.0
> Environment: EMR 6.1.0
> Reporter: Shiquan Wang
> Priority: Blocker
> Labels: easyfix
> Fix For: 0.8.0
>
> Attachments: image-2022-11-13-16-41-13-052.png
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
>
> When working with PySpark with Python >= 3.8, below error is raised:
> {code:java}
> TypeError: required field "type_ignores" missing from Module{code}
> The error points to code in [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L223]
>
> The error is caused by an API change in Python 3.8: [https://github.com/python/cpython/pull/11645/files] and was reported here [https://bugs.python.org/issue35894]
> IPython has faced similar issue: [https://github.com/ipython/ipython/issues/11590]
> IPython solved the issue in this PR: [https://github.com/ipython/ipython/pull/11593]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)