You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dmitriy Lyubimov <dl...@gmail.com> on 2013/07/31 04:40:22 UTC

fixed hbase version in SparkBuild (spark-0.8)

Hello,

after couple of days(!) of trying to understand where i get the
"NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
hbase.

While it is assumed that hadoop version is specified, hbase version is
fixed. This seem to create problem if hbase is used with a particular
version of CDH hadoop client in the backend. (there's a known compatibility
bug).

wouldn't it make sense in this case to allow to declare hbase version as
well, perhaps even tie it to the CDH version?

At the very least i think it deserves a specific mention in the header
section to provide opportunity to override, just like hadoop version does?

Thanks.
-D

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Konstantin Boudnik <co...@apache.org>.
Well, dependency clashes are a hell to trace. Maven has this dependency:tree
way of presenting all the project deps. in a graph form, but still it is a
pain in the neck.

I am about to roll out a fix for maven assembly to exclude examples from the
assembly.

As for the 3rd party repo (just my two cents): not sure about sbt, but in case
of Maven it can easily be handled by local users' settings.xml profiles and
repos, rather than be hardcoded in the project. Otherwise, the build
system might end up with more and more 3rd party distros' repositories being
included, which leads to mess, confusion, etc.

Cos

On Wed, Jul 31, 2013 at 11:38AM, Dmitriy Lyubimov wrote:
> On Wed, Jul 31, 2013 at 11:24 AM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > Dmitriy,
> >
> > lib_managed/ is controlled by sbt build - Maven uses ~/.m2/repository
> > directly. Hence, earlier Matei point stands on the need to fix it in sbt
> > build.
> >
> > I am not sure why would Cloudera solve the issue in the Spark build? :)
> >
> 
> I wasn't saying Сloudera  should. I just meant to say i saw other people
> stranded out there on that very issue right now while searching for the
> cause. This incompatibility bug is not at all obvious to track.
> 
> Ok i guess it should be ok for sbt build to declare HBASE_VERSION and use
> it.
> 
> I also discovered that cdh hbase counterpart does not seem to be published
> to any repository spark uses (unlike hadoop client jars). So to resolve, i
> also had to add a direct cloudera resolver. it is probably more than you
> want to do for spark, but any person using hbase and cdh would probably run
> into this surprisingly hard to track issue.
> 
> 
> >
> > Cos
> >
> > On Tue, Jul 30, 2013 at 11:13PM, Dmitriy Lyubimov wrote:
> > > FWIW it does get landed in lib_managed...
> > >
> > > i beleive there's a post in CDH group with the same very problem i had ,
> > > still unresolved/unadvised by Cloudera.
> > >
> > >
> > > On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > >
> > > > Ok, that makes sense, although I thought examples might be a good part
> > of
> > > > the
> > > > documentation for the beginners' references, etc. But I have problems
> > > > modifying
> > > > the assembly accordingly. Will do it the first thing in the morning.
> > > >
> > > > Thanks for the input.
> > > >   Cos
> > > >
> > > > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > > > > Basically the way to think of the assembly is that it should have
> > > > libraries
> > > > > that users' client programs need to run. These are core, repl
> > (needed if
> > > > > they use the shell), and likely bagel and streaming and mllib, though
> > > > > originally we'd opted to leave those out. We are still deciding on
> > that
> > > > --
> > > > > could go either way.
> > > > >
> > > > > Matei
> > > > >
> > > > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Yeah, that is true. But the assembly shouldn't include the examples
> > > > project at all IMO -- if it does now, we should remove it.
> > > > > >
> > > > > > Matei
> > > > > >
> > > > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org>
> > > > wrote:
> > > > > >
> > > > > >> Matei,
> > > > > >>
> > > > > >> Hbase dependencies aren't actually included into the Maven
> > assembly
> > > > as of this
> > > > > >> moment, because scope of hbase dependency in examples' module is
> > > > "compile"; but
> > > > > >> the assembly is only includes those with "runtime". Hence it is
> > > > automatically
> > > > > >> excluded.
> > > > > >>
> > > > > >> I believe, hbase is needed for examples during the execution time,
> > > > and if so -
> > > > > >> it would have to be fixed in the module. This will lead to need to
> > > > exclude it
> > > > > >> from the assembly, in turn.
> > > > > >>
> > > > > >> And of course... :)
> > > > > >>
> > > > > >>   s/putt/pull/
> > > > > >>
> > > > > >> Cos
> > > > > >>
> > > > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > > > > >>> Yeah, and maybe we will want to change to Maven as the
> > recommended
> > > > tool for
> > > > > >>> assembly building. I want to look into this more for the 0.8
> > release.
> > > > > >>>
> > > > > >>> Matei
> > > > > >>>
> > > > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org>
> > > > wrote:
> > > > > >>>
> > > > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > > > > >>>>> Let's at the very least make it configurable, but an even
> > better
> > > > thing will
> > > > > >>>>> be to make sbt assembly not include it. I think the only thing
> > > > that depends
> > > > > >>>>> on HBase is the examples project, but unfortunately SBT puts
> > all
> > > > its JARs in
> > > > > >>>>> the lib_managed folder and just stupidly creates an assembly by
> > > > grouping
> > > > > >>>>> those. The Maven build, for example, should not do that.
> > > > > >>>>
> > > > > >>>> It is very easy to exclude dependencies in Maven assembly, like
> > it
> > > > is done for
> > > > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> > > > Dmitriy, thank
> > > > > >>>> you!
> > > > > >>>>
> > > > > >>>> Cos
> > > > > >>>>
> > > > > >>>>> Matei
> > > > > >>>>>
> > > > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <
> > dlieu.7@gmail.com>
> > > > wrote:
> > > > > >>>>>
> > > > > >>>>>> Hello,
> > > > > >>>>>>
> > > > > >>>>>> after couple of days(!) of trying to understand where i get
> > the
> > > > > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8
> > now
> > > > includes
> > > > > >>>>>> hbase.
> > > > > >>>>>>
> > > > > >>>>>> While it is assumed that hadoop version is specified, hbase
> > > > version is
> > > > > >>>>>> fixed. This seem to create problem if hbase is used with a
> > > > particular
> > > > > >>>>>> version of CDH hadoop client in the backend. (there's a known
> > > > compatibility
> > > > > >>>>>> bug).
> > > > > >>>>>>
> > > > > >>>>>> wouldn't it make sense in this case to allow to declare hbase
> > > > version as
> > > > > >>>>>> well, perhaps even tie it to the CDH version?
> > > > > >>>>>>
> > > > > >>>>>> At the very least i think it deserves a specific mention in
> > the
> > > > header
> > > > > >>>>>> section to provide opportunity to override, just like hadoop
> > > > version does?
> > > > > >>>>>>
> > > > > >>>>>> Thanks.
> > > > > >>>>>> -D
> > > > > >>>>>
> > > > > >>>
> > > > > >
> > > > >
> > > >
> >

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Wed, Jul 31, 2013 at 11:24 AM, Konstantin Boudnik <co...@apache.org> wrote:

> Dmitriy,
>
> lib_managed/ is controlled by sbt build - Maven uses ~/.m2/repository
> directly. Hence, earlier Matei point stands on the need to fix it in sbt
> build.
>
> I am not sure why would Cloudera solve the issue in the Spark build? :)
>

I wasn't saying Сloudera  should. I just meant to say i saw other people
stranded out there on that very issue right now while searching for the
cause. This incompatibility bug is not at all obvious to track.

Ok i guess it should be ok for sbt build to declare HBASE_VERSION and use
it.

I also discovered that cdh hbase counterpart does not seem to be published
to any repository spark uses (unlike hadoop client jars). So to resolve, i
also had to add a direct cloudera resolver. it is probably more than you
want to do for spark, but any person using hbase and cdh would probably run
into this surprisingly hard to track issue.


>
> Cos
>
> On Tue, Jul 30, 2013 at 11:13PM, Dmitriy Lyubimov wrote:
> > FWIW it does get landed in lib_managed...
> >
> > i beleive there's a post in CDH group with the same very problem i had ,
> > still unresolved/unadvised by Cloudera.
> >
> >
> > On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> > > Ok, that makes sense, although I thought examples might be a good part
> of
> > > the
> > > documentation for the beginners' references, etc. But I have problems
> > > modifying
> > > the assembly accordingly. Will do it the first thing in the morning.
> > >
> > > Thanks for the input.
> > >   Cos
> > >
> > > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > > > Basically the way to think of the assembly is that it should have
> > > libraries
> > > > that users' client programs need to run. These are core, repl
> (needed if
> > > > they use the shell), and likely bagel and streaming and mllib, though
> > > > originally we'd opted to leave those out. We are still deciding on
> that
> > > --
> > > > could go either way.
> > > >
> > > > Matei
> > > >
> > > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com>
> > > wrote:
> > > >
> > > > > Yeah, that is true. But the assembly shouldn't include the examples
> > > project at all IMO -- if it does now, we should remove it.
> > > > >
> > > > > Matei
> > > > >
> > > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org>
> > > wrote:
> > > > >
> > > > >> Matei,
> > > > >>
> > > > >> Hbase dependencies aren't actually included into the Maven
> assembly
> > > as of this
> > > > >> moment, because scope of hbase dependency in examples' module is
> > > "compile"; but
> > > > >> the assembly is only includes those with "runtime". Hence it is
> > > automatically
> > > > >> excluded.
> > > > >>
> > > > >> I believe, hbase is needed for examples during the execution time,
> > > and if so -
> > > > >> it would have to be fixed in the module. This will lead to need to
> > > exclude it
> > > > >> from the assembly, in turn.
> > > > >>
> > > > >> And of course... :)
> > > > >>
> > > > >>   s/putt/pull/
> > > > >>
> > > > >> Cos
> > > > >>
> > > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > > > >>> Yeah, and maybe we will want to change to Maven as the
> recommended
> > > tool for
> > > > >>> assembly building. I want to look into this more for the 0.8
> release.
> > > > >>>
> > > > >>> Matei
> > > > >>>
> > > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org>
> > > wrote:
> > > > >>>
> > > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > > > >>>>> Let's at the very least make it configurable, but an even
> better
> > > thing will
> > > > >>>>> be to make sbt assembly not include it. I think the only thing
> > > that depends
> > > > >>>>> on HBase is the examples project, but unfortunately SBT puts
> all
> > > its JARs in
> > > > >>>>> the lib_managed folder and just stupidly creates an assembly by
> > > grouping
> > > > >>>>> those. The Maven build, for example, should not do that.
> > > > >>>>
> > > > >>>> It is very easy to exclude dependencies in Maven assembly, like
> it
> > > is done for
> > > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> > > Dmitriy, thank
> > > > >>>> you!
> > > > >>>>
> > > > >>>> Cos
> > > > >>>>
> > > > >>>>> Matei
> > > > >>>>>
> > > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <
> dlieu.7@gmail.com>
> > > wrote:
> > > > >>>>>
> > > > >>>>>> Hello,
> > > > >>>>>>
> > > > >>>>>> after couple of days(!) of trying to understand where i get
> the
> > > > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8
> now
> > > includes
> > > > >>>>>> hbase.
> > > > >>>>>>
> > > > >>>>>> While it is assumed that hadoop version is specified, hbase
> > > version is
> > > > >>>>>> fixed. This seem to create problem if hbase is used with a
> > > particular
> > > > >>>>>> version of CDH hadoop client in the backend. (there's a known
> > > compatibility
> > > > >>>>>> bug).
> > > > >>>>>>
> > > > >>>>>> wouldn't it make sense in this case to allow to declare hbase
> > > version as
> > > > >>>>>> well, perhaps even tie it to the CDH version?
> > > > >>>>>>
> > > > >>>>>> At the very least i think it deserves a specific mention in
> the
> > > header
> > > > >>>>>> section to provide opportunity to override, just like hadoop
> > > version does?
> > > > >>>>>>
> > > > >>>>>> Thanks.
> > > > >>>>>> -D
> > > > >>>>>
> > > > >>>
> > > > >
> > > >
> > >
>

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Konstantin Boudnik <co...@apache.org>.
Dmitriy,

lib_managed/ is controlled by sbt build - Maven uses ~/.m2/repository
directly. Hence, earlier Matei point stands on the need to fix it in sbt
build.

I am not sure why would Cloudera solve the issue in the Spark build? :)

Cos

On Tue, Jul 30, 2013 at 11:13PM, Dmitriy Lyubimov wrote:
> FWIW it does get landed in lib_managed...
> 
> i beleive there's a post in CDH group with the same very problem i had ,
> still unresolved/unadvised by Cloudera.
> 
> 
> On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > Ok, that makes sense, although I thought examples might be a good part of
> > the
> > documentation for the beginners' references, etc. But I have problems
> > modifying
> > the assembly accordingly. Will do it the first thing in the morning.
> >
> > Thanks for the input.
> >   Cos
> >
> > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > > Basically the way to think of the assembly is that it should have
> > libraries
> > > that users' client programs need to run. These are core, repl (needed if
> > > they use the shell), and likely bagel and streaming and mllib, though
> > > originally we'd opted to leave those out. We are still deciding on that
> > --
> > > could go either way.
> > >
> > > Matei
> > >
> > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com>
> > wrote:
> > >
> > > > Yeah, that is true. But the assembly shouldn't include the examples
> > project at all IMO -- if it does now, we should remove it.
> > > >
> > > > Matei
> > > >
> > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >
> > > >> Matei,
> > > >>
> > > >> Hbase dependencies aren't actually included into the Maven assembly
> > as of this
> > > >> moment, because scope of hbase dependency in examples' module is
> > "compile"; but
> > > >> the assembly is only includes those with "runtime". Hence it is
> > automatically
> > > >> excluded.
> > > >>
> > > >> I believe, hbase is needed for examples during the execution time,
> > and if so -
> > > >> it would have to be fixed in the module. This will lead to need to
> > exclude it
> > > >> from the assembly, in turn.
> > > >>
> > > >> And of course... :)
> > > >>
> > > >>   s/putt/pull/
> > > >>
> > > >> Cos
> > > >>
> > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > > >>> Yeah, and maybe we will want to change to Maven as the recommended
> > tool for
> > > >>> assembly building. I want to look into this more for the 0.8 release.
> > > >>>
> > > >>> Matei
> > > >>>
> > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >>>
> > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > > >>>>> Let's at the very least make it configurable, but an even better
> > thing will
> > > >>>>> be to make sbt assembly not include it. I think the only thing
> > that depends
> > > >>>>> on HBase is the examples project, but unfortunately SBT puts all
> > its JARs in
> > > >>>>> the lib_managed folder and just stupidly creates an assembly by
> > grouping
> > > >>>>> those. The Maven build, for example, should not do that.
> > > >>>>
> > > >>>> It is very easy to exclude dependencies in Maven assembly, like it
> > is done for
> > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> > Dmitriy, thank
> > > >>>> you!
> > > >>>>
> > > >>>> Cos
> > > >>>>
> > > >>>>> Matei
> > > >>>>>
> > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> > > >>>>>
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> after couple of days(!) of trying to understand where i get the
> > > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now
> > includes
> > > >>>>>> hbase.
> > > >>>>>>
> > > >>>>>> While it is assumed that hadoop version is specified, hbase
> > version is
> > > >>>>>> fixed. This seem to create problem if hbase is used with a
> > particular
> > > >>>>>> version of CDH hadoop client in the backend. (there's a known
> > compatibility
> > > >>>>>> bug).
> > > >>>>>>
> > > >>>>>> wouldn't it make sense in this case to allow to declare hbase
> > version as
> > > >>>>>> well, perhaps even tie it to the CDH version?
> > > >>>>>>
> > > >>>>>> At the very least i think it deserves a specific mention in the
> > header
> > > >>>>>> section to provide opportunity to override, just like hadoop
> > version does?
> > > >>>>>>
> > > >>>>>> Thanks.
> > > >>>>>> -D
> > > >>>>>
> > > >>>
> > > >
> > >
> >

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Dmitriy Lyubimov <dl...@gmail.com>.
FWIW it does get landed in lib_managed...

i beleive there's a post in CDH group with the same very problem i had ,
still unresolved/unadvised by Cloudera.


On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <co...@apache.org> wrote:

> Ok, that makes sense, although I thought examples might be a good part of
> the
> documentation for the beginners' references, etc. But I have problems
> modifying
> the assembly accordingly. Will do it the first thing in the morning.
>
> Thanks for the input.
>   Cos
>
> On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > Basically the way to think of the assembly is that it should have
> libraries
> > that users' client programs need to run. These are core, repl (needed if
> > they use the shell), and likely bagel and streaming and mllib, though
> > originally we'd opted to leave those out. We are still deciding on that
> --
> > could go either way.
> >
> > Matei
> >
> > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
> >
> > > Yeah, that is true. But the assembly shouldn't include the examples
> project at all IMO -- if it does now, we should remove it.
> > >
> > > Matei
> > >
> > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >
> > >> Matei,
> > >>
> > >> Hbase dependencies aren't actually included into the Maven assembly
> as of this
> > >> moment, because scope of hbase dependency in examples' module is
> "compile"; but
> > >> the assembly is only includes those with "runtime". Hence it is
> automatically
> > >> excluded.
> > >>
> > >> I believe, hbase is needed for examples during the execution time,
> and if so -
> > >> it would have to be fixed in the module. This will lead to need to
> exclude it
> > >> from the assembly, in turn.
> > >>
> > >> And of course... :)
> > >>
> > >>   s/putt/pull/
> > >>
> > >> Cos
> > >>
> > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > >>> Yeah, and maybe we will want to change to Maven as the recommended
> tool for
> > >>> assembly building. I want to look into this more for the 0.8 release.
> > >>>
> > >>> Matei
> > >>>
> > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >>>
> > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > >>>>> Let's at the very least make it configurable, but an even better
> thing will
> > >>>>> be to make sbt assembly not include it. I think the only thing
> that depends
> > >>>>> on HBase is the examples project, but unfortunately SBT puts all
> its JARs in
> > >>>>> the lib_managed folder and just stupidly creates an assembly by
> grouping
> > >>>>> those. The Maven build, for example, should not do that.
> > >>>>
> > >>>> It is very easy to exclude dependencies in Maven assembly, like it
> is done for
> > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> Dmitriy, thank
> > >>>> you!
> > >>>>
> > >>>> Cos
> > >>>>
> > >>>>> Matei
> > >>>>>
> > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> > >>>>>
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> after couple of days(!) of trying to understand where i get the
> > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now
> includes
> > >>>>>> hbase.
> > >>>>>>
> > >>>>>> While it is assumed that hadoop version is specified, hbase
> version is
> > >>>>>> fixed. This seem to create problem if hbase is used with a
> particular
> > >>>>>> version of CDH hadoop client in the backend. (there's a known
> compatibility
> > >>>>>> bug).
> > >>>>>>
> > >>>>>> wouldn't it make sense in this case to allow to declare hbase
> version as
> > >>>>>> well, perhaps even tie it to the CDH version?
> > >>>>>>
> > >>>>>> At the very least i think it deserves a specific mention in the
> header
> > >>>>>> section to provide opportunity to override, just like hadoop
> version does?
> > >>>>>>
> > >>>>>> Thanks.
> > >>>>>> -D
> > >>>>>
> > >>>
> > >
> >
>

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Konstantin Boudnik <co...@apache.org>.
Ok, that makes sense, although I thought examples might be a good part of the
documentation for the beginners' references, etc. But I have problems modifying
the assembly accordingly. Will do it the first thing in the morning.

Thanks for the input.
  Cos

On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> Basically the way to think of the assembly is that it should have libraries
> that users' client programs need to run. These are core, repl (needed if
> they use the shell), and likely bagel and streaming and mllib, though
> originally we'd opted to leave those out. We are still deciding on that --
> could go either way.
> 
> Matei
> 
> On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com> wrote:
> 
> > Yeah, that is true. But the assembly shouldn't include the examples project at all IMO -- if it does now, we should remove it.
> > 
> > Matei
> > 
> > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > 
> >> Matei,
> >> 
> >> Hbase dependencies aren't actually included into the Maven assembly as of this
> >> moment, because scope of hbase dependency in examples' module is "compile"; but
> >> the assembly is only includes those with "runtime". Hence it is automatically
> >> excluded.
> >> 
> >> I believe, hbase is needed for examples during the execution time, and if so -
> >> it would have to be fixed in the module. This will lead to need to exclude it
> >> from the assembly, in turn.
> >> 
> >> And of course... :)
> >> 
> >>   s/putt/pull/
> >> 
> >> Cos
> >> 
> >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> >>> Yeah, and maybe we will want to change to Maven as the recommended tool for
> >>> assembly building. I want to look into this more for the 0.8 release.
> >>> 
> >>> Matei
> >>> 
> >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >>> 
> >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> >>>>> Let's at the very least make it configurable, but an even better thing will
> >>>>> be to make sbt assembly not include it. I think the only thing that depends
> >>>>> on HBase is the examples project, but unfortunately SBT puts all its JARs in
> >>>>> the lib_managed folder and just stupidly creates an assembly by grouping
> >>>>> those. The Maven build, for example, should not do that.
> >>>> 
> >>>> It is very easy to exclude dependencies in Maven assembly, like it is done for
> >>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
> >>>> you!
> >>>> 
> >>>> Cos
> >>>> 
> >>>>> Matei
> >>>>> 
> >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> >>>>> 
> >>>>>> Hello,
> >>>>>> 
> >>>>>> after couple of days(!) of trying to understand where i get the
> >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
> >>>>>> hbase.
> >>>>>> 
> >>>>>> While it is assumed that hadoop version is specified, hbase version is
> >>>>>> fixed. This seem to create problem if hbase is used with a particular
> >>>>>> version of CDH hadoop client in the backend. (there's a known compatibility
> >>>>>> bug).
> >>>>>> 
> >>>>>> wouldn't it make sense in this case to allow to declare hbase version as
> >>>>>> well, perhaps even tie it to the CDH version?
> >>>>>> 
> >>>>>> At the very least i think it deserves a specific mention in the header
> >>>>>> section to provide opportunity to override, just like hadoop version does?
> >>>>>> 
> >>>>>> Thanks.
> >>>>>> -D
> >>>>> 
> >>> 
> > 
> 

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Konstantin Boudnik <co...@apache.org>.
And I have just submitted
  https://github.com/mesos/spark/pull/763

that depends on yours for mllib support.
  Cos

On Wed, Jul 31, 2013 at 02:27PM, Shivaram Venkataraman wrote:
> Actually I sent a pull request for this yesterday
> https://github.com/mesos/spark/pull/753
> 
> Shivaram
> 
> 
> On Wed, Jul 31, 2013 at 2:22 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > Interestingly, mllib isn't even getting built by maven... I will address
> > it as
> > a separate patch then.
> >
> > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > > Basically the way to think of the assembly is that it should have
> > libraries
> > > that users' client programs need to run. These are core, repl (needed if
> > > they use the shell), and likely bagel and streaming and mllib, though
> > > originally we'd opted to leave those out. We are still deciding on that
> > --
> > > could go either way.
> > >
> > > Matei
> > >
> > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com>
> > wrote:
> > >
> > > > Yeah, that is true. But the assembly shouldn't include the examples
> > project at all IMO -- if it does now, we should remove it.
> > > >
> > > > Matei
> > > >
> > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >
> > > >> Matei,
> > > >>
> > > >> Hbase dependencies aren't actually included into the Maven assembly
> > as of this
> > > >> moment, because scope of hbase dependency in examples' module is
> > "compile"; but
> > > >> the assembly is only includes those with "runtime". Hence it is
> > automatically
> > > >> excluded.
> > > >>
> > > >> I believe, hbase is needed for examples during the execution time,
> > and if so -
> > > >> it would have to be fixed in the module. This will lead to need to
> > exclude it
> > > >> from the assembly, in turn.
> > > >>
> > > >> And of course... :)
> > > >>
> > > >>   s/putt/pull/
> > > >>
> > > >> Cos
> > > >>
> > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > > >>> Yeah, and maybe we will want to change to Maven as the recommended
> > tool for
> > > >>> assembly building. I want to look into this more for the 0.8 release.
> > > >>>
> > > >>> Matei
> > > >>>
> > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >>>
> > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > > >>>>> Let's at the very least make it configurable, but an even better
> > thing will
> > > >>>>> be to make sbt assembly not include it. I think the only thing
> > that depends
> > > >>>>> on HBase is the examples project, but unfortunately SBT puts all
> > its JARs in
> > > >>>>> the lib_managed folder and just stupidly creates an assembly by
> > grouping
> > > >>>>> those. The Maven build, for example, should not do that.
> > > >>>>
> > > >>>> It is very easy to exclude dependencies in Maven assembly, like it
> > is done for
> > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> > Dmitriy, thank
> > > >>>> you!
> > > >>>>
> > > >>>> Cos
> > > >>>>
> > > >>>>> Matei
> > > >>>>>
> > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> > > >>>>>
> > > >>>>>> Hello,
> > > >>>>>>
> > > >>>>>> after couple of days(!) of trying to understand where i get the
> > > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now
> > includes
> > > >>>>>> hbase.
> > > >>>>>>
> > > >>>>>> While it is assumed that hadoop version is specified, hbase
> > version is
> > > >>>>>> fixed. This seem to create problem if hbase is used with a
> > particular
> > > >>>>>> version of CDH hadoop client in the backend. (there's a known
> > compatibility
> > > >>>>>> bug).
> > > >>>>>>
> > > >>>>>> wouldn't it make sense in this case to allow to declare hbase
> > version as
> > > >>>>>> well, perhaps even tie it to the CDH version?
> > > >>>>>>
> > > >>>>>> At the very least i think it deserves a specific mention in the
> > header
> > > >>>>>> section to provide opportunity to override, just like hadoop
> > version does?
> > > >>>>>>
> > > >>>>>> Thanks.
> > > >>>>>> -D
> > > >>>>>
> > > >>>
> > > >
> > >
> >

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
Actually I sent a pull request for this yesterday
https://github.com/mesos/spark/pull/753

Shivaram


On Wed, Jul 31, 2013 at 2:22 PM, Konstantin Boudnik <co...@apache.org> wrote:

> Interestingly, mllib isn't even getting built by maven... I will address
> it as
> a separate patch then.
>
> On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> > Basically the way to think of the assembly is that it should have
> libraries
> > that users' client programs need to run. These are core, repl (needed if
> > they use the shell), and likely bagel and streaming and mllib, though
> > originally we'd opted to leave those out. We are still deciding on that
> --
> > could go either way.
> >
> > Matei
> >
> > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
> >
> > > Yeah, that is true. But the assembly shouldn't include the examples
> project at all IMO -- if it does now, we should remove it.
> > >
> > > Matei
> > >
> > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >
> > >> Matei,
> > >>
> > >> Hbase dependencies aren't actually included into the Maven assembly
> as of this
> > >> moment, because scope of hbase dependency in examples' module is
> "compile"; but
> > >> the assembly is only includes those with "runtime". Hence it is
> automatically
> > >> excluded.
> > >>
> > >> I believe, hbase is needed for examples during the execution time,
> and if so -
> > >> it would have to be fixed in the module. This will lead to need to
> exclude it
> > >> from the assembly, in turn.
> > >>
> > >> And of course... :)
> > >>
> > >>   s/putt/pull/
> > >>
> > >> Cos
> > >>
> > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> > >>> Yeah, and maybe we will want to change to Maven as the recommended
> tool for
> > >>> assembly building. I want to look into this more for the 0.8 release.
> > >>>
> > >>> Matei
> > >>>
> > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >>>
> > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> > >>>>> Let's at the very least make it configurable, but an even better
> thing will
> > >>>>> be to make sbt assembly not include it. I think the only thing
> that depends
> > >>>>> on HBase is the examples project, but unfortunately SBT puts all
> its JARs in
> > >>>>> the lib_managed folder and just stupidly creates an assembly by
> grouping
> > >>>>> those. The Maven build, for example, should not do that.
> > >>>>
> > >>>> It is very easy to exclude dependencies in Maven assembly, like it
> is done for
> > >>>> Hadoop. Lemme send out a putt request - a good finding indeed,
> Dmitriy, thank
> > >>>> you!
> > >>>>
> > >>>> Cos
> > >>>>
> > >>>>> Matei
> > >>>>>
> > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> > >>>>>
> > >>>>>> Hello,
> > >>>>>>
> > >>>>>> after couple of days(!) of trying to understand where i get the
> > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now
> includes
> > >>>>>> hbase.
> > >>>>>>
> > >>>>>> While it is assumed that hadoop version is specified, hbase
> version is
> > >>>>>> fixed. This seem to create problem if hbase is used with a
> particular
> > >>>>>> version of CDH hadoop client in the backend. (there's a known
> compatibility
> > >>>>>> bug).
> > >>>>>>
> > >>>>>> wouldn't it make sense in this case to allow to declare hbase
> version as
> > >>>>>> well, perhaps even tie it to the CDH version?
> > >>>>>>
> > >>>>>> At the very least i think it deserves a specific mention in the
> header
> > >>>>>> section to provide opportunity to override, just like hadoop
> version does?
> > >>>>>>
> > >>>>>> Thanks.
> > >>>>>> -D
> > >>>>>
> > >>>
> > >
> >
>

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Konstantin Boudnik <co...@apache.org>.
Interestingly, mllib isn't even getting built by maven... I will address it as
a separate patch then.

On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote:
> Basically the way to think of the assembly is that it should have libraries
> that users' client programs need to run. These are core, repl (needed if
> they use the shell), and likely bagel and streaming and mllib, though
> originally we'd opted to leave those out. We are still deciding on that --
> could go either way.
> 
> Matei
> 
> On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com> wrote:
> 
> > Yeah, that is true. But the assembly shouldn't include the examples project at all IMO -- if it does now, we should remove it.
> > 
> > Matei
> > 
> > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > 
> >> Matei,
> >> 
> >> Hbase dependencies aren't actually included into the Maven assembly as of this
> >> moment, because scope of hbase dependency in examples' module is "compile"; but
> >> the assembly is only includes those with "runtime". Hence it is automatically
> >> excluded.
> >> 
> >> I believe, hbase is needed for examples during the execution time, and if so -
> >> it would have to be fixed in the module. This will lead to need to exclude it
> >> from the assembly, in turn.
> >> 
> >> And of course... :)
> >> 
> >>   s/putt/pull/
> >> 
> >> Cos
> >> 
> >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> >>> Yeah, and maybe we will want to change to Maven as the recommended tool for
> >>> assembly building. I want to look into this more for the 0.8 release.
> >>> 
> >>> Matei
> >>> 
> >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >>> 
> >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> >>>>> Let's at the very least make it configurable, but an even better thing will
> >>>>> be to make sbt assembly not include it. I think the only thing that depends
> >>>>> on HBase is the examples project, but unfortunately SBT puts all its JARs in
> >>>>> the lib_managed folder and just stupidly creates an assembly by grouping
> >>>>> those. The Maven build, for example, should not do that.
> >>>> 
> >>>> It is very easy to exclude dependencies in Maven assembly, like it is done for
> >>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
> >>>> you!
> >>>> 
> >>>> Cos
> >>>> 
> >>>>> Matei
> >>>>> 
> >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> >>>>> 
> >>>>>> Hello,
> >>>>>> 
> >>>>>> after couple of days(!) of trying to understand where i get the
> >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
> >>>>>> hbase.
> >>>>>> 
> >>>>>> While it is assumed that hadoop version is specified, hbase version is
> >>>>>> fixed. This seem to create problem if hbase is used with a particular
> >>>>>> version of CDH hadoop client in the backend. (there's a known compatibility
> >>>>>> bug).
> >>>>>> 
> >>>>>> wouldn't it make sense in this case to allow to declare hbase version as
> >>>>>> well, perhaps even tie it to the CDH version?
> >>>>>> 
> >>>>>> At the very least i think it deserves a specific mention in the header
> >>>>>> section to provide opportunity to override, just like hadoop version does?
> >>>>>> 
> >>>>>> Thanks.
> >>>>>> -D
> >>>>> 
> >>> 
> > 
> 

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Matei Zaharia <ma...@gmail.com>.
Basically the way to think of the assembly is that it should have libraries that users' client programs need to run. These are core, repl (needed if they use the shell), and likely bagel and streaming and mllib, though originally we'd opted to leave those out. We are still deciding on that -- could go either way.

Matei

On Jul 30, 2013, at 9:56 PM, Matei Zaharia <ma...@gmail.com> wrote:

> Yeah, that is true. But the assembly shouldn't include the examples project at all IMO -- if it does now, we should remove it.
> 
> Matei
> 
> On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
>> Matei,
>> 
>> Hbase dependencies aren't actually included into the Maven assembly as of this
>> moment, because scope of hbase dependency in examples' module is "compile"; but
>> the assembly is only includes those with "runtime". Hence it is automatically
>> excluded.
>> 
>> I believe, hbase is needed for examples during the execution time, and if so -
>> it would have to be fixed in the module. This will lead to need to exclude it
>> from the assembly, in turn.
>> 
>> And of course... :)
>> 
>>   s/putt/pull/
>> 
>> Cos
>> 
>> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
>>> Yeah, and maybe we will want to change to Maven as the recommended tool for
>>> assembly building. I want to look into this more for the 0.8 release.
>>> 
>>> Matei
>>> 
>>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
>>> 
>>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
>>>>> Let's at the very least make it configurable, but an even better thing will
>>>>> be to make sbt assembly not include it. I think the only thing that depends
>>>>> on HBase is the examples project, but unfortunately SBT puts all its JARs in
>>>>> the lib_managed folder and just stupidly creates an assembly by grouping
>>>>> those. The Maven build, for example, should not do that.
>>>> 
>>>> It is very easy to exclude dependencies in Maven assembly, like it is done for
>>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
>>>> you!
>>>> 
>>>> Cos
>>>> 
>>>>> Matei
>>>>> 
>>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> after couple of days(!) of trying to understand where i get the
>>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
>>>>>> hbase.
>>>>>> 
>>>>>> While it is assumed that hadoop version is specified, hbase version is
>>>>>> fixed. This seem to create problem if hbase is used with a particular
>>>>>> version of CDH hadoop client in the backend. (there's a known compatibility
>>>>>> bug).
>>>>>> 
>>>>>> wouldn't it make sense in this case to allow to declare hbase version as
>>>>>> well, perhaps even tie it to the CDH version?
>>>>>> 
>>>>>> At the very least i think it deserves a specific mention in the header
>>>>>> section to provide opportunity to override, just like hadoop version does?
>>>>>> 
>>>>>> Thanks.
>>>>>> -D
>>>>> 
>>> 
> 


Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Matei Zaharia <ma...@gmail.com>.
Yeah, that is true. But the assembly shouldn't include the examples project at all IMO -- if it does now, we should remove it.

Matei

On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <co...@apache.org> wrote:

> Matei,
> 
> Hbase dependencies aren't actually included into the Maven assembly as of this
> moment, because scope of hbase dependency in examples' module is "compile"; but
> the assembly is only includes those with "runtime". Hence it is automatically
> excluded.
> 
> I believe, hbase is needed for examples during the execution time, and if so -
> it would have to be fixed in the module. This will lead to need to exclude it
> from the assembly, in turn.
> 
> And of course... :)
> 
>    s/putt/pull/
> 
> Cos
> 
> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
>> Yeah, and maybe we will want to change to Maven as the recommended tool for
>> assembly building. I want to look into this more for the 0.8 release.
>> 
>> Matei
>> 
>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
>> 
>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
>>>> Let's at the very least make it configurable, but an even better thing will
>>>> be to make sbt assembly not include it. I think the only thing that depends
>>>> on HBase is the examples project, but unfortunately SBT puts all its JARs in
>>>> the lib_managed folder and just stupidly creates an assembly by grouping
>>>> those. The Maven build, for example, should not do that.
>>> 
>>> It is very easy to exclude dependencies in Maven assembly, like it is done for
>>> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
>>> you!
>>> 
>>> Cos
>>> 
>>>> Matei
>>>> 
>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> after couple of days(!) of trying to understand where i get the
>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
>>>>> hbase.
>>>>> 
>>>>> While it is assumed that hadoop version is specified, hbase version is
>>>>> fixed. This seem to create problem if hbase is used with a particular
>>>>> version of CDH hadoop client in the backend. (there's a known compatibility
>>>>> bug).
>>>>> 
>>>>> wouldn't it make sense in this case to allow to declare hbase version as
>>>>> well, perhaps even tie it to the CDH version?
>>>>> 
>>>>> At the very least i think it deserves a specific mention in the header
>>>>> section to provide opportunity to override, just like hadoop version does?
>>>>> 
>>>>> Thanks.
>>>>> -D
>>>> 
>> 


Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Konstantin Boudnik <co...@apache.org>.
Matei,

Hbase dependencies aren't actually included into the Maven assembly as of this
moment, because scope of hbase dependency in examples' module is "compile"; but
the assembly is only includes those with "runtime". Hence it is automatically
excluded.

I believe, hbase is needed for examples during the execution time, and if so -
it would have to be fixed in the module. This will lead to need to exclude it
from the assembly, in turn.

And of course... :)

    s/putt/pull/

Cos

On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote:
> Yeah, and maybe we will want to change to Maven as the recommended tool for
> assembly building. I want to look into this more for the 0.8 release.
> 
> Matei
> 
> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> >> Let's at the very least make it configurable, but an even better thing will
> >> be to make sbt assembly not include it. I think the only thing that depends
> >> on HBase is the examples project, but unfortunately SBT puts all its JARs in
> >> the lib_managed folder and just stupidly creates an assembly by grouping
> >> those. The Maven build, for example, should not do that.
> > 
> > It is very easy to exclude dependencies in Maven assembly, like it is done for
> > Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
> > you!
> > 
> > Cos
> > 
> >> Matei
> >> 
> >> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> >> 
> >>> Hello,
> >>> 
> >>> after couple of days(!) of trying to understand where i get the
> >>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
> >>> hbase.
> >>> 
> >>> While it is assumed that hadoop version is specified, hbase version is
> >>> fixed. This seem to create problem if hbase is used with a particular
> >>> version of CDH hadoop client in the backend. (there's a known compatibility
> >>> bug).
> >>> 
> >>> wouldn't it make sense in this case to allow to declare hbase version as
> >>> well, perhaps even tie it to the CDH version?
> >>> 
> >>> At the very least i think it deserves a specific mention in the header
> >>> section to provide opportunity to override, just like hadoop version does?
> >>> 
> >>> Thanks.
> >>> -D
> >> 
> 

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Matei Zaharia <ma...@gmail.com>.
Yeah, and maybe we will want to change to Maven as the recommended tool for assembly building. I want to look into this more for the 0.8 release.

Matei

On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <co...@apache.org> wrote:

> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
>> Let's at the very least make it configurable, but an even better thing will
>> be to make sbt assembly not include it. I think the only thing that depends
>> on HBase is the examples project, but unfortunately SBT puts all its JARs in
>> the lib_managed folder and just stupidly creates an assembly by grouping
>> those. The Maven build, for example, should not do that.
> 
> It is very easy to exclude dependencies in Maven assembly, like it is done for
> Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
> you!
> 
> Cos
> 
>> Matei
>> 
>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>> 
>>> Hello,
>>> 
>>> after couple of days(!) of trying to understand where i get the
>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
>>> hbase.
>>> 
>>> While it is assumed that hadoop version is specified, hbase version is
>>> fixed. This seem to create problem if hbase is used with a particular
>>> version of CDH hadoop client in the backend. (there's a known compatibility
>>> bug).
>>> 
>>> wouldn't it make sense in this case to allow to declare hbase version as
>>> well, perhaps even tie it to the CDH version?
>>> 
>>> At the very least i think it deserves a specific mention in the header
>>> section to provide opportunity to override, just like hadoop version does?
>>> 
>>> Thanks.
>>> -D
>> 


Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Konstantin Boudnik <co...@apache.org>.
On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote:
> Let's at the very least make it configurable, but an even better thing will
> be to make sbt assembly not include it. I think the only thing that depends
> on HBase is the examples project, but unfortunately SBT puts all its JARs in
> the lib_managed folder and just stupidly creates an assembly by grouping
> those. The Maven build, for example, should not do that.

It is very easy to exclude dependencies in Maven assembly, like it is done for
Hadoop. Lemme send out a putt request - a good finding indeed, Dmitriy, thank
you!

Cos

> Matei
> 
> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> 
> > Hello,
> > 
> > after couple of days(!) of trying to understand where i get the
> > "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
> > hbase.
> > 
> > While it is assumed that hadoop version is specified, hbase version is
> > fixed. This seem to create problem if hbase is used with a particular
> > version of CDH hadoop client in the backend. (there's a known compatibility
> > bug).
> > 
> > wouldn't it make sense in this case to allow to declare hbase version as
> > well, perhaps even tie it to the CDH version?
> > 
> > At the very least i think it deserves a specific mention in the header
> > section to provide opportunity to override, just like hadoop version does?
> > 
> > Thanks.
> > -D
> 

Re: fixed hbase version in SparkBuild (spark-0.8)

Posted by Matei Zaharia <ma...@gmail.com>.
Let's at the very least make it configurable, but an even better thing will be to make sbt assembly not include it. I think the only thing that depends on HBase is the examples project, but unfortunately SBT puts all its JARs in the lib_managed folder and just stupidly creates an assembly by grouping those. The Maven build, for example, should not do that.

Matei

On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:

> Hello,
> 
> after couple of days(!) of trying to understand where i get the
> "NoSuchMethod" error, i traced it down to the fact that 0.8 now includes
> hbase.
> 
> While it is assumed that hadoop version is specified, hbase version is
> fixed. This seem to create problem if hbase is used with a particular
> version of CDH hadoop client in the backend. (there's a known compatibility
> bug).
> 
> wouldn't it make sense in this case to allow to declare hbase version as
> well, perhaps even tie it to the CDH version?
> 
> At the very least i think it deserves a specific mention in the header
> section to provide opportunity to override, just like hadoop version does?
> 
> Thanks.
> -D