Getting all dependencies

[Pat Ferrel <pa...@occamsmachete.com>]

We have the job jar in hadoop, which includes all dependencies.

The same construct works for Spark if the *job.jar files get included in the classpath and they currently are.

I created a job.jar for spark but Dmitriy wants to remove it. That would be ok because the dependency got move into math-scala but the issue has only moved.

I don’t see how we can leave it to the user to configure their environment to include the right version of the right classes. As it stands now I can’t run Mahout without that math-scala job.jar. I realize there are other ways to solve this but don’t know of a better way.

If I don’t hear back with some other option my next commit will put a job.jar in math-scala and take it out of spark. Dmitriy would like to remove it from both places but without some other solution I don’t see how this can work—obviously we can call it something like managedlibs.jar but I have at least one dependency that does not get included due to other mahout job jars.

Re: Getting all dependencies

[Andrew Musselman <an...@gmail.com>]

I don't understand the whole topic but my preference is to have things
working for people who may not be comfortable fiddling with classpaths.

Is this an issue of what we ship in jars, or what jars get built when
people build from source, both, etc.?

On Sun, Dec 21, 2014 at 4:34 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:

> We have the job jar in hadoop, which includes all dependencies.
>
> The same construct works for Spark if the *job.jar files get included in
> the classpath and they currently are.
>
> I created a job.jar for spark but Dmitriy wants to remove it. That would
> be ok because the dependency got move into math-scala but the issue has
> only moved.
>
> I don’t see how we can leave it to the user to configure their environment
> to include the right version of the right classes. As it stands now I can’t
> run Mahout without that math-scala job.jar. I realize there are other ways
> to solve this but don’t know of a better way.
>
> If I don’t hear back with some other option my next commit will put a
> job.jar in math-scala and take it out of spark. Dmitriy would like to
> remove it from both places but without some other solution I don’t see how
> this can work—obviously we can call it something like managedlibs.jar but I
> have at least one dependency that does not get included due to other mahout
> job jars.
>
>