You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@toree.apache.org by "ywskycn (via GitHub)" <gi...@apache.org> on 2023/06/23 04:59:45 UTC
[GitHub] [incubator-toree] ywskycn opened a new pull request, #200: [TOREE-536] AddJar command to support Google cloud storage
ywskycn opened a new pull request, #200:
URL: https://github.com/apache/incubator-toree/pull/200
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] ranadip commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "ranadip (via GitHub)" <gi...@apache.org>.
ranadip commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1708700526
I have been able to reproduce the same issue. It seems the jar in the example @ywskycn suggested works fine with AddJars but one created locally using sbt does not load even though it gets downloaded into the local Toree cache from GCS. No errors/warnings show up either on screen or in the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] kristopherkane commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "kristopherkane (via GitHub)" <gi...@apache.org>.
kristopherkane commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615185039
OK, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] kristopherkane commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "kristopherkane (via GitHub)" <gi...@apache.org>.
kristopherkane commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1674967740
@almaaouiwassim sorry, I did not. I've kind-of given up on Toree since we will also need kernel time Spark configuration changes and the level of effort to add those is too high, in addition to the driver not getting the AddJar classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] almaaouiwassim commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "almaaouiwassim (via GitHub)" <gi...@apache.org>.
almaaouiwassim commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1689672385
@ywskycn did you succeed to reproduce the issue please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] almaaouiwassim commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "almaaouiwassim (via GitHub)" <gi...@apache.org>.
almaaouiwassim commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1674934368
Hello, it seems I have the same issue as you @kristopherkane even with the local `file` prefix.
<img width="1026" alt="Screenshot 2023-08-11 at 10 38 15" src="https://github.com/apache/incubator-toree/assets/142032694/f935de67-ed79-4872-812a-a53f00f7670f">
I am using Dataproc 2.1, by defaut I have one driver and two executors and the yarn deploy mode is `client`. Seing the jar load messages duplicated twice, I am wondering if the classes are being loaded on the executors but not on the driver?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] ywskycn commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "ywskycn (via GitHub)" <gi...@apache.org>.
ywskycn commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615178416
> Those are my exact steps and toree jar locations.
>
> Did you by chance change any spark:spark* configuration at cluster creation?
no, I just follow all default configs, only specifying image version and Jupyter component.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] kristopherkane commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "kristopherkane (via GitHub)" <gi...@apache.org>.
kristopherkane commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615176679
Those are my exact steps and toree jar locations.
Did you by chance change any spark:spark* configuration at cluster creation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] kristopherkane commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "kristopherkane (via GitHub)" <gi...@apache.org>.
kristopherkane commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615177424
I also should say, my comment doesn't hold this PR up. That schema will be required regardless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1710434016
<img width="610" alt="image" src="https://github.com/apache/incubator-toree/assets/26535726/0823415f-c2c4-4000-af43-331a8848a41b">
Can not reproduce the issue with the latest master [commit](https://github.com/apache/incubator-toree/commit/937c1260bf7973ef20b0052989398ac9e789d160)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] kristopherkane commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "kristopherkane (via GitHub)" <gi...@apache.org>.
kristopherkane commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615151771
@ywskycn Interesting. I compiled this same change last week and did that to no avail on Dataproc 2.1. Is Dataproc also your runtime environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] lresende commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "lresende (via GitHub)" <gi...@apache.org>.
lresende commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1675884791
Could you clarify your additional requirements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] ywskycn commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "ywskycn (via GitHub)" <gi...@apache.org>.
ywskycn commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1681078136
@almaaouiwassim have you checked the Spark UI whether the jar was added?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] almaaouiwassim commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "almaaouiwassim (via GitHub)" <gi...@apache.org>.
almaaouiwassim commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1681957226
@ywskycn yes it has been added to the spark configuration
<img width="895" alt="Screenshot 2023-08-17 at 10 35 38" src="https://github.com/apache/incubator-toree/assets/142032694/8cef36be-dbdc-41c7-947e-5b140afbf191">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] almaaouiwassim commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "almaaouiwassim (via GitHub)" <gi...@apache.org>.
almaaouiwassim commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1680614559
Hey guys, I am back with few steps to reproduce the issue. The `AddJar` seems to work with some jars but no all of them, I couldn't get what makes my jar not well loaded by the kernel.
Here are the steps to reproduce the issue
- create a 2.1.21-debian11 dataproc cluster (probably the jar won't work with other env, but this is the one I am testing with)
- download the jar to the local filesystem using terminal on the notebook ```gsutil cp gs://untienots_public/myproject-assembly.jar .```
- load the jar in the notebook then do an import
You can check the screenshots to see the results and to see that my jar actually contains the class that couldn't be imported
<img width="1024" alt="Screenshot 2023-08-16 at 14 10 45" src="https://github.com/apache/incubator-toree/assets/142032694/70955798-32e2-4955-bb5a-966fb1cb4aae">
<img width="780" alt="Screenshot 2023-08-16 at 14 11 04" src="https://github.com/apache/incubator-toree/assets/142032694/490d4d92-4321-4983-ae81-aa4add0f2958">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] lresende merged pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "lresende (via GitHub)" <gi...@apache.org>.
lresende merged PR #200:
URL: https://github.com/apache/incubator-toree/pull/200
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [TOREE-536] AddJar command to support Google cloud storage [incubator-toree]
Posted by "almaaouiwassim (via GitHub)" <gi...@apache.org>.
almaaouiwassim commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1780832022
Hello guys, sorry for the late response, I had time recently to try to understand what exactly doesn't work with Apache Toree. It seems that any jar with a class on the root classpath of the jar (not under a package) is not well added to the kernel using AddJar.
I put a sample project to reproduce it in https://github.com/almaaouiwassim/my_project. just run `sbt assembly` to generate the jar.
I have also put the jar on GCS https://storage.cloud.google.com/untienots_public/my_project.jar?authuser=0
Please tell me if you need further information.
FI, even if you don't put classes at the root classpath, some dependencies can still do it and break your jar, in our case it was `spark-xml` who has a class `module-info.class` at the root. I guess many dependencies would do the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] almaaouiwassim commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "almaaouiwassim (via GitHub)" <gi...@apache.org>.
almaaouiwassim commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1711825865
@pan3793 can you please try with this jar `gs://untienots_public/myproject-assembly.jar` doing a
- `gstuil cp`
- then, loading it from local filesystem.
- then, `import com.untienots.MyObject`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] pan3793 commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1712028366
@almaaouiwassim I'm not familiar with the GCP platform, I suppose the jar size is small, would you mind sending it to me via email?
> I suspect it's linked to how the jar is actually built. If so I can provide my `build.sbt` file
I believe it would be helpful if you could share how to produce your problematic(maybe) jar from scratch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] kristopherkane commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "kristopherkane (via GitHub)" <gi...@apache.org>.
kristopherkane commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615072619
I have done this previously and tested it. Although it does work in Dataproc to make Toree's internal download function to 'cache JARs', The jar classes are not useable in the notebook itself. I assume not all permutations of Spark paths (runtime, repl, etc) are updated.
You can test this in a the kernel by simply importing a class from a JAR that was called to download with this magic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] ywskycn commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "ywskycn (via GitHub)" <gi...@apache.org>.
ywskycn commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615171000
@kristopherkane yes. What I have done:
- Apply this change to tags/v0.5.0-incubating-rc5 and `make dist`
- Use the newly built `dist/toree/lib/toree-assembly-0.5.0-incubating.jar` to replace `/opt/conda/miniconda3/lib/python3.10/site-packages/toree/lib/toree-assembly-0.5.0-incubating.jar` and `/opt/conda/miniconda3/share/jupyter/kernels/apache_toree_scala/lib/toree-assembly-0.5.0-incubating.jar` in a Dataproc cluster (image 2.1-debian11 with Jupyter component enabled).
- Test %AddJar and import command in a new notebook file, and it works well there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] ywskycn commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "ywskycn (via GitHub)" <gi...@apache.org>.
ywskycn commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1615119684
> I have done this previously and tested it. Although it does work in Dataproc to make Toree's internal download function to 'cache JARs', The jar classes are not useable in the notebook itself. I assume not all permutations of Spark paths (runtime, repl, etc) are updated.
>
> You can test this in a the kernel by simply importing a class from a JAR that was called to download with this magic.
@kristopherkane could you tell more about "not usable" here? I tested by `AddJar` and then `import` classes from that jar in the notebook, and it works there. Also I verified in the Spark application UI, where the jar has been added to the classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-toree] kristopherkane commented on pull request #200: [TOREE-536] AddJar command to support Google cloud storage
Posted by "kristopherkane (via GitHub)" <gi...@apache.org>.
kristopherkane commented on PR #200:
URL: https://github.com/apache/incubator-toree/pull/200#issuecomment-1679467026
@lresende The ability to set Spark configuration from the kernel itself and before the SparkSession is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@toree.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org