%dep deprecated or not?

[Aish Fenton <af...@netflix.com>]

Hi all,
Can someone clear up if %dep is going to be deprecated? From the docs and
warning messages it sounds like that's still the plan. But from an in
person conversation I've had with Moon, it sounded like maybe not (although
this was a while back)?

If it's depreciated, that'll leave a big gap for us. Our typical workflow
involves us exchanging notebooks (export/import of the json) fairly
regularly. And every notebook has different library dependencies. Being
able to specify these inline, in the notebook, is really useful.

I remember the concern with %dep was with adding jars dynamically? If this
is still true, then maybe a compromise would be to keep a way of
documenting the jar dependencies per notebook, but when the user loads that
notebook prompt them with "Jar X is missing, click yes to add to your Spark
interpreter dependencies". So have it function more like an automatic way
to fill in the GUI dependencies settings, but at least it keeps the
self-documenting functionality too.

Aish

Re: %dep deprecated or not?

[Aish Fenton <af...@netflix.com>]

Thanks Moon!


On Fri, Nov 4, 2016 at 1:24 PM moon soo Lee <mo...@apache.org> wrote:

> Hi Aish,
>
> Deprecating %dep was plan before. %dep loads library into thread context
> classloader after JVM process creation. However, in few places, Spark tries
> to find class from System classloader, not from classloader of thread
> context. That was reason we introduced new dependency loading mechanism
> (through interpreter GUI) which set library into the JVM class path on
> process creation, and deprecating %dep.
>
> But since then, we've got multiple feedbacks from user that they likes
> %dep approaches for some reasons.
>
> - %dep is self-documenting library requirement in the notebook.
> - %dep allows per Note (and possible per User) library loading.
>
> And i agree. So, It make sense to keep %dep unless we have other
> alternatives.
>
> Thanks,
> moon
>
> On Fri, Nov 4, 2016 at 12:53 PM Aish Fenton <af...@netflix.com> wrote:
>
> Hi all,
> Can someone clear up if %dep is going to be deprecated? From the docs and
> warning messages it sounds like that's still the plan. But from an in
> person conversation I've had with Moon, it sounded like maybe not (although
> this was a while back)?
>
> If it's depreciated, that'll leave a big gap for us. Our typical workflow
> involves us exchanging notebooks (export/import of the json) fairly
> regularly. And every notebook has different library dependencies. Being
> able to specify these inline, in the notebook, is really useful.
>
> I remember the concern with %dep was with adding jars dynamically? If this
> is still true, then maybe a compromise would be to keep a way of
> documenting the jar dependencies per notebook, but when the user loads that
> notebook prompt them with "Jar X is missing, click yes to add to your Spark
> interpreter dependencies". So have it function more like an automatic way
> to fill in the GUI dependencies settings, but at least it keeps the
> self-documenting functionality too.
>
> Aish
>
>

Re: %dep deprecated or not?

[moon soo Lee <mo...@apache.org>]

Hi Aish,

Deprecating %dep was plan before. %dep loads library into thread context
classloader after JVM process creation. However, in few places, Spark tries
to find class from System classloader, not from classloader of thread
context. That was reason we introduced new dependency loading mechanism
(through interpreter GUI) which set library into the JVM class path on
process creation, and deprecating %dep.

But since then, we've got multiple feedbacks from user that they likes %dep
approaches for some reasons.

- %dep is self-documenting library requirement in the notebook.
- %dep allows per Note (and possible per User) library loading.

And i agree. So, It make sense to keep %dep unless we have other
alternatives.

Thanks,
moon

On Fri, Nov 4, 2016 at 12:53 PM Aish Fenton <af...@netflix.com> wrote:

> Hi all,
> Can someone clear up if %dep is going to be deprecated? From the docs and
> warning messages it sounds like that's still the plan. But from an in
> person conversation I've had with Moon, it sounded like maybe not (although
> this was a while back)?
>
> If it's depreciated, that'll leave a big gap for us. Our typical workflow
> involves us exchanging notebooks (export/import of the json) fairly
> regularly. And every notebook has different library dependencies. Being
> able to specify these inline, in the notebook, is really useful.
>
> I remember the concern with %dep was with adding jars dynamically? If this
> is still true, then maybe a compromise would be to keep a way of
> documenting the jar dependencies per notebook, but when the user loads that
> notebook prompt them with "Jar X is missing, click yes to add to your Spark
> interpreter dependencies". So have it function more like an automatic way
> to fill in the GUI dependencies settings, but at least it keeps the
> self-documenting functionality too.
>
> Aish
>