You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by GitBox <gi...@apache.org> on 2022/03/21 07:15:07 UTC
[GitHub] [zeppelin] zjffdu opened a new pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
zjffdu opened a new pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323
### What is this PR for?
This PR removed module `spark-dependencies` and also update `interpreter.sh`
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-5680
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? No
* Does this needs documentation? No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078730387
@zjffdu Thank you for the clarification. LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078617276
@jongyoul After looking at the code, I realise that downloading `pyspark` and `sparkr` can not be removed, they are required for unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078787851
BTW, I've checked the CI but the error is ...
```
ImportError: cannot import name 'Markup' from 'jinja2' (/usr/share/miniconda/envs/python_3_with_R/lib/python3.7/site-packages/jinja2/__init__.py) expected:<SUCCESS> but was:<ERROR>
```
Is it good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078610324
Good point @jongyoul let me update that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul merged pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul merged pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078628304
@zjffdu Yes, correct. Agreed. Then, does the download work only for testing? I just wonder if it's possible to reduce our whole binary or not. If it can, we'd better consider it sincerely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078627564
I don't think so, we should enable user to run unit test locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078416337
@zjffdu agreed. In an early stage, it was needed for expanding our community by providing easiest way to use Zeppelin itself but thing's are changed. BTW, we also need to change our guide if we remove it, don't we?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1079624581
The error above might be related to #4331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1080250649
That's right @jongyoul Now, CI is passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078619948
> @jongyoul After looking at the code, I realise that downloading pyspark and sparkr can not be removed, they are required for unit test.
I have an idea about it. In my understanding, we are downloading Spark binary while running CI. Can we replace it? I'll also check it :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078712290
@jongyoul I mean the downloading in this maven pom.xml (https://github.com/apache/zeppelin/blob/master/spark/interpreter/pom.xml#L355), it is only for unit test. It won't increase the zeppelin binary distribution as long as we remove spark-dependencies module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1078589520
LGTM especially, it will help us to reduce whole build time as well.
By the way, if we forgive embedded mode, can we remove downloading python and R as well? We will replace them by setting paths based on SPARK_HOME. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] jongyoul commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
jongyoul commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1077553706
I'm just curious that this means we need to download spark manually if we would like to use Spark Interpreter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on pull request #4323: [ZEPPELIN-5680] Remove spark-dependencies module
Posted by GitBox <gi...@apache.org>.
zjffdu commented on pull request #4323:
URL: https://github.com/apache/zeppelin/pull/4323#issuecomment-1077685049
@jongyoul That's right, actually the embedded spark only works in limited functions, a lot of spark features are not available for this embedded spark. Instead, this embedded spark would often confuse users when they find that it behaves differently from official spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@zeppelin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org