You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by tolomaus <ni...@gmail.com> on 2016/09/03 09:35:33 UTC
Dependency jars not refreshed after interpreter restart in Zeppelin
0.6.1
Hi,
I just upgraded from zeppelin 0.5.6/spark 1.6.2 to zeppelin 0.6.1/spark
2.0.0 and after moving my application's jars %deps to the spark interpreter
UI dependencies section I noticed that updated jars are not taken into
account anymore. Instead Zeppelin continues to load the original versions of
my jars.
I see that zeppelin initially moves the jars to
[zeppelin]/local-repo/2BXAQ6T44 and sends this location to spark. But even
when I manually remove them there it still doesn't trigger a refresh.
My temporary workaround is to have my CI system put the jars also to
[zeppelin]/local-repo/2BXAQ6T44.
I'm using a dedicated spark install in local mode.
Regards,
Niek.
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Dependency-jars-not-refreshed-after-interpreter-restart-in-Zeppelin-0-6-1-tp4035.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
Re: Dependency jars not refreshed after interpreter restart in
Zeppelin 0.6.1
Posted by tolomaus <ni...@gmail.com>.
Thanks Mina, will wait for 0.6.2!
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Dependency-jars-not-refreshed-after-interpreter-restart-in-Zeppelin-0-6-1-tp4035p4213.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
Re: Dependency jars not refreshed after interpreter restart in
Zeppelin 0.6.1
Posted by Mina Lee <mi...@apache.org>.
Hey Niek,
I could reproduce the issue with your description.
I created [1] jira ticket and [2] pull request to fix it. This patch is
going to be included in Zeppelin 0.6.2.
Thanks for reporting the issue.
[1] https://issues.apache.org/jira/browse/ZEPPELIN-1482
[2] https://github.com/apache/zeppelin/pull/1453
On Wed, Sep 21, 2016 at 5:11 PM tolomaus <ni...@gmail.com>
wrote:
> Hi Mina,
>
> Thanks a lot for digging into it.
>
> See here my work flow:
>
> I have two jar files: one with my code (A) and one with all dependencies
> (B). The one with the code is rebuilt regularly.
>
> 1. build jar file A v1
> 2. start zeppelin
> 3. remove dependencies for A and B in the spark interpreter, save, and add
> them again (this seems to reset zeppelin to the correct state again)
> 4. run the notebook => result shows code for A v1 is used => OK
> 5. build jar file A v2 and replace A v1 in the location specified in the
> interpreter dependencies for A
> 6. restart the interpreter
> 7. run the notebook => result shows code for A v1 is still used => NOK
>
> workaround:
> 8. restart interpreter
> 9. copy jar file A v2 to localRepo (replacing A v1)
> 10. run the notebook => result shows code for A v2 is now used => OK
>
> I kept a copy of the log file, happy to share it with you outside the
> mailing list.
>
> Kind regards,
>
> Niek.
>
>
>
>
> --
> View this message in context:
> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Dependency-jars-not-refreshed-after-interpreter-restart-in-Zeppelin-0-6-1-tp4035p4188.html
> Sent from the Apache Zeppelin Users (incubating) mailing list mailing list
> archive at Nabble.com.
>
Re: Dependency jars not refreshed after interpreter restart in
Zeppelin 0.6.1
Posted by tolomaus <ni...@gmail.com>.
Hi Mina,
Thanks a lot for digging into it.
See here my work flow:
I have two jar files: one with my code (A) and one with all dependencies
(B). The one with the code is rebuilt regularly.
1. build jar file A v1
2. start zeppelin
3. remove dependencies for A and B in the spark interpreter, save, and add
them again (this seems to reset zeppelin to the correct state again)
4. run the notebook => result shows code for A v1 is used => OK
5. build jar file A v2 and replace A v1 in the location specified in the
interpreter dependencies for A
6. restart the interpreter
7. run the notebook => result shows code for A v1 is still used => NOK
workaround:
8. restart interpreter
9. copy jar file A v2 to localRepo (replacing A v1)
10. run the notebook => result shows code for A v2 is now used => OK
I kept a copy of the log file, happy to share it with you outside the
mailing list.
Kind regards,
Niek.
--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Dependency-jars-not-refreshed-after-interpreter-restart-in-Zeppelin-0-6-1-tp4035p4188.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.
Re: Dependency jars not refreshed after interpreter restart in
Zeppelin 0.6.1
Posted by Mina Lee <mi...@apache.org>.
Hi Niek,
I tried to dig into the problem here, but I couldn't reproduce with
following step:
1. Add dependency '/my/path/a.jar' in GUI
2. Update '/my/path/a.jar' contents
3. go to interpreter page and click edit -> save
It would be helpful if you can point out what I missed to reproduce the
issue.
FYI, Spark interpreter process sets jars under local-repo/${interpreter_id}
to its classpath on start up so if you make changes under this directory
you will need to restart spark interpreter.
On Sat, Sep 3, 2016 at 11:35 AM tolomaus <ni...@gmail.com>
wrote:
> Hi,
>
> I just upgraded from zeppelin 0.5.6/spark 1.6.2 to zeppelin 0.6.1/spark
> 2.0.0 and after moving my application's jars %deps to the spark interpreter
> UI dependencies section I noticed that updated jars are not taken into
> account anymore. Instead Zeppelin continues to load the original versions
> of
> my jars.
>
> I see that zeppelin initially moves the jars to
> [zeppelin]/local-repo/2BXAQ6T44 and sends this location to spark. But even
> when I manually remove them there it still doesn't trigger a refresh.
>
> My temporary workaround is to have my CI system put the jars also to
> [zeppelin]/local-repo/2BXAQ6T44.
>
> I'm using a dedicated spark install in local mode.
>
> Regards,
>
> Niek.
>
>
>
>
> --
> View this message in context:
> http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Dependency-jars-not-refreshed-after-interpreter-restart-in-Zeppelin-0-6-1-tp4035.html
> Sent from the Apache Zeppelin Users (incubating) mailing list mailing list
> archive at Nabble.com.
>