You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@livy.apache.org by GitBox <gi...@apache.org> on 2021/01/06 19:34:43 UTC
[GitHub] [incubator-livy] gabrielmagno opened a new pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
gabrielmagno opened a new pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314
## What changes were proposed in this pull request?
A mock for the `ast.Module` is created in case of running older Python versions (< 3.8). The mocked module will ignore the new `type_ignores` parameter, and pass only the first parameter to the old module.
As already mentioned by Shiquan Wang (the issue reporter), Python 3.8 introduced a change in the API (https://github.com/python/cpython/pull/11645/files), and IPython had a similar problem. The strategy used for this fix is similar to the one utilized by IPython in this PR: https://github.com/ipython/ipython/pull/11593
JIRA: https://issues.apache.org/jira/projects/LIVY/issues/LIVY-795
## How was this patch tested?
Tested locally, using Jupyter notebook to run pyspark commands.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] windmark edited a comment on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
windmark edited a comment on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942032154
Hi, bumping this issue @dacort @gabrielmagno since we are blocked by not being able to use 3.8 in our setup.
Looking at the failed master build, https://travis-ci.org/github/apache/incubator-livy/builds/718090124, from PR #302, do you have any plans to revert that change since it seems to block any further development? Looping in owners of that PR @mgaido91 @ajbozarth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] mgaido91 commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
mgaido91 commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942067151
I am more familiar with the thriftserver part, so I am not confident reviewing this PR, btw the CI is failing, so a first step would be having the CI working fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] gabrielmagno commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
gabrielmagno commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-756101719
The build for `Spark 3.0 Unit Tests` failed, but so does the master branch. Probably not related to the Python fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] windmark commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
windmark commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942032154
Hi, bumping this issue @dacort @gabrielmagno since we blocked by not being able to use 3.8 in our setup.
Looking at the failed master build, https://travis-ci.org/github/apache/incubator-livy/builds/718090124, from PR #302, do you have any plans to revert that change since it seems to block any further development? Looping in owners of that PR @mgaido91 @ajbozarth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] gabrielmagno commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
gabrielmagno commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-821155642
@siege089 would be glad to do so, but [latest master commit](https://github.com/apache/incubator-livy/commit/4d8a912699683b973eee76d4e91447d769a0cb0d) is from Aug 2020, and it was already incorporated in my branch when I made the pull request. It has a ["failed" build status](https://travis-ci.org/github/apache/incubator-livy/builds/718090124).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] mgaido91 commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
mgaido91 commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942762346
Then I would recommend fixing the tests on the master branch and then go ahead. It is the safest option IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] siege089 commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
siege089 commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-820862784
It appears master tests are passing, any chance this build could be rechecked and merged? Would love to be able to run multiple lines of python at once in 3.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] dacort commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
dacort commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942722327
I [tried to get CI working](https://github.com/apache/incubator-livy/pull/326#issuecomment-891374655), but didn't have much luck. :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] mgaido91 commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
mgaido91 commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942067151
I am more familiar with the thriftserver part, so I am not confident reviewing this PR, btw the CI is failing, so a first step would be having the CI working fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] gabrielmagno commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
gabrielmagno commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942758569
@mgaido91 just to point out that the [latest master commit](https://github.com/apache/incubator-livy/commit/4d8a912699683b973eee76d4e91447d769a0cb0d) has a ["failed" build status](https://travis-ci.org/github/apache/incubator-livy/builds/718090124).
This latest master commit is already incorporated in my branch, and I believe it is failing because of that, not exactly because of the Python 3.8 fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] gabrielmagno commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
gabrielmagno commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942766366
> Then I would recommend fixing the tests on the master branch and then go ahead. It is the safest option IMHO.
I agree with you, thank you for the feedback!
I would be glad to do it, but this is out of my expertise and knowledge of the project :-(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] dacort commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
dacort commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-891156411
@gabrielmagno Thanks for the reply! Let me see if I can clone the change and open a new PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] dacort edited a comment on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
dacort edited a comment on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-891156411
@gabrielmagno Thanks for the reply! ~~Let me see if I can clone the change and open a new PR.~~
edit: I see what you mean. I wonder if that was just a test blip? The change seems unrelated... :\
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] windmark edited a comment on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
windmark edited a comment on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942032154
Hi, bumping this issue @dacort @gabrielmagno since we are blocked by not being able to use 3.8 in our setup.
Looking at the failed master build, https://travis-ci.org/github/apache/incubator-livy/builds/718090124, from PR #302, do you have any plans to revert that change since it seems to block any further development? Looping in owners of that PR @mgaido91 @ajbozarth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] dacort commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
dacort commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-943411436
@mgaido91 It looks like https://github.com/apache/incubator-livy/pull/318 did the work to fix the tests on the master branch. Is there somebody that could take a look at that / merge it in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] dacort commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
dacort commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-888688271
*bump* Any chance we can revive this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] gabrielmagno commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
gabrielmagno commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-891095853
> @siege089 would be glad to do so, but [latest master commit](https://github.com/apache/incubator-livy/commit/4d8a912699683b973eee76d4e91447d769a0cb0d) is from Aug 2020, and it was already incorporated in my branch when I made the pull request. It has a ["failed" build status](https://travis-ci.org/github/apache/incubator-livy/builds/718090124).
@dacort Unfortunately I'm still limited to the master's fail commit from August 2020. If there is anything I could do to circumvent it in the meantime, I would be glad to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] derache123 commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
derache123 commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-1068526101
Any update on this PR? Is anyone still looking at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] windmark commented on pull request #314: [LIVY-795] Add support for PySpark with Python >= 3.8
Posted by GitBox <gi...@apache.org>.
windmark commented on pull request #314:
URL: https://github.com/apache/incubator-livy/pull/314#issuecomment-942032154
Hi, bumping this issue @dacort @gabrielmagno since we blocked by not being able to use 3.8 in our setup.
Looking at the failed master build, https://travis-ci.org/github/apache/incubator-livy/builds/718090124, from PR #302, do you have any plans to revert that change since it seems to block any further development? Looping in owners of that PR @mgaido91 @ajbozarth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org