You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Enis Söztutar <en...@apache.org> on 2014/07/01 00:08:52 UTC
Re: hbase-testing-utils [Was: HBase 0.98.x dependency problems]
Cos,
I think that module was introduced to circumvent maven not doing transient
dependency resolution for test jars, something along those lines. It's only
job is to list dependencies to test jars from a main (as opposed to test)
module.
Enis
On Fri, Jun 27, 2014 at 12:51 PM, Konstantin Boudnik <co...@apache.org> wrote:
> I am looking into this hbase-testing-util module and don't see any use of
> it,
> at least not within the hbase itself. The module doesn't have any source
> code
> and doesn't provide any functionality for the Hbase itself.
>
> The only relevant ticket I found is HBASE-9699, which only specifies one
> use
> of this module, namely
> https://github.com/elliottneilclark/hbase-downstreamer,
> that seems to be a "Fake downstream project used figuring what is required
> when depending on hbase client and minicluster". The explcit use of test
> artifacts in non-test scopes are a big no-no in my opinion. I don't think
> we
> need to repeat the same mistake that some of the Hadoop's modules are
> making.
>
> I'd like someone with better understanding of the subject to shed the
> light on
> the adoption of this module, and possible modifications of it? If the whole
> purpose of it is to provide a set of jar files than perhaps a simple
> assembly
> will suffice?
>
> Thanks in advance,
> Cos
>
> On Fri, Jun 27, 2014 at 12:27PM, Konstantin Boudnik wrote:
> > Thanks a bunch Ted - it really opened my eye on that's the issue (you
> can tell
> > I am still a noob).
> >
> > Cos
> >
> > On Fri, Jun 27, 2014 at 05:04AM, Ted Yu wrote:
> > > Thanks for finding this issue, Cos.
> > >
> > > I opened HBASE-11422 and attached a patch there - there is no
> dependency on
> > > 2.2.0 if -Dhadoop-two.version=2.3.0 is specified.
> > >
> > > Cheers
> > >
> > >
> > > On Thu, Jun 26, 2014 at 9:43 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >
> > > > Weird... running
> > > > mvn install dependency:tree -Dhadoop-two.version=2.3.0
> > > > keeps pulling in 2.2.0 for test artifacts. An easy way to reproduce
> it is
> > > > to
> > > > wipe out clean hadoop 2.2.0 artifacts from ~/.m2 as well as 0.98.2*
> (if
> > > > installed) and then run the command above. Something like this can
> be seen:
> > > >
> > > > [INFO] +- org.apache.hbase:hbase-shell:jar:0.98.2:compile
> > > > [INFO] | +- org.apache.hbase:hbase-prefix-tree:jar:0.98.2:runtime
> > > > [INFO] | +- com.yammer.metrics:metrics-core:jar:2.1.2:compile
> > > > [INFO] | +- org.jruby:jruby-complete:jar:1.6.8:compile
> > > > [INFO] | +- org.apache.hadoop:hadoop-client:jar:2.3.0:compile
> > > > [INFO] | | +-
> > > > org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0:compile
> > > > [INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile
> > > > [INFO] | | \-
> > > > org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0:compile
> > > > [INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.3.0:compile
> > > > [INFO] | | \- commons-daemon:commons-daemon:jar:1.0.13:compile
> > > > [INFO] | \-
> org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.2.0:compile
> > > >
> > > > I have fixed a problem with site target, but still see this...
> > > >
> > > > Any ideas are very appreciated! Thanks.
> > > > Cos
> > > >
> > > >
> > > > On Tue, Jun 17, 2014 at 08:33PM, Konstantin Boudnik wrote:
> > > > > Thank you very much Ted - that helped beautifully! I think the
> build
> > > > system
> > > > > becomes a product on its own :)
> > > > >
> > > > > Regards,
> > > > > Cos
> > > > >
> > > > > On Tue, Jun 17, 2014 at 12:40PM, Ted Yu wrote:
> > > > > > Have you tried the following command ?
> > > > > >
> > > > > > mvn dependency:tree -Dhadoop-two.version=2.3.0
> > > > > >
> > > > > > The output of the above has 2.3.0 as the dependency.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 17, 2014 at 11:40 AM, Konstantin Boudnik <
> cos@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > Guys,
> > > > > > >
> > > > > > > I have noticed an interesting problem with HBase 0.98 line. I
> have
> > > > ended
> > > > > > > up with
> > > > > > > a crappy Hadoop 2.2.0 artifacts in my local M2 cache - don't
> ask me
> > > > how
> > > > > > > that
> > > > > > > happen ;( - which causes compilation problems in HBase. And
> that
> > > > brought
> > > > > > > this
> > > > > > > whole issue into the light. Here's the essence of it:
> > > > > > >
> > > > > > > while running for hbase-server module
> > > > > > > % mvn dependency:tree -Dhadoop.version=2.3.0
> > > > > > > I am getting a reference to hadoop 2.2.0
> > > > > > >
> > > > > > > [INFO] +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
> > > > > > > ..
> > > > > > > [INFO] +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
> > > > > > > [INFO] +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
> > > > > > >
> > > > > > > This only happens for 0.98. On master the reference goes to
> 2.4.0
> > > > > > > The problem here is that -Dhadoop.version is being seemingly
> > > > ignored, which
> > > > > > > might lead to binaries with dependencies inconsistent with
> what it
> > > > was
> > > > > > > built
> > > > > > > against. I am not sure if this had came up before, but
> certainly
> > > > would
> > > > > > > appreciate the community's input, if any.
> > > > > > >
> > > > > > > Thanks a lot!
> > > > > > > --
> > > > > > > Take care,
> > > > > > > Cos
> > > > > > >
> > > > > > >
> > > >
>
Re: hbase-testing-utils [Was: HBase 0.98.x dependency problems]
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks Enis. I will look into this to possibly fix the fixed deps resolution
issue. In the mean while, I will complete the patch for 0.98 and master for
everything but this module. Hopefully, it would be taken into 0.98.4. Will try
to post it tomorrow.
Thanks guys,
Cos
On Mon, Jun 30, 2014 at 03:08PM, Enis Söztutar wrote:
> Cos,
>
> I think that module was introduced to circumvent maven not doing transient
> dependency resolution for test jars, something along those lines. It's only
> job is to list dependencies to test jars from a main (as opposed to test)
> module.
>
> Enis
>
>
> On Fri, Jun 27, 2014 at 12:51 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > I am looking into this hbase-testing-util module and don't see any use of
> > it,
> > at least not within the hbase itself. The module doesn't have any source
> > code
> > and doesn't provide any functionality for the Hbase itself.
> >
> > The only relevant ticket I found is HBASE-9699, which only specifies one
> > use
> > of this module, namely
> > https://github.com/elliottneilclark/hbase-downstreamer,
> > that seems to be a "Fake downstream project used figuring what is required
> > when depending on hbase client and minicluster". The explcit use of test
> > artifacts in non-test scopes are a big no-no in my opinion. I don't think
> > we
> > need to repeat the same mistake that some of the Hadoop's modules are
> > making.
> >
> > I'd like someone with better understanding of the subject to shed the
> > light on
> > the adoption of this module, and possible modifications of it? If the whole
> > purpose of it is to provide a set of jar files than perhaps a simple
> > assembly
> > will suffice?
> >
> > Thanks in advance,
> > Cos
> >
> > On Fri, Jun 27, 2014 at 12:27PM, Konstantin Boudnik wrote:
> > > Thanks a bunch Ted - it really opened my eye on that's the issue (you
> > can tell
> > > I am still a noob).
> > >
> > > Cos
> > >
> > > On Fri, Jun 27, 2014 at 05:04AM, Ted Yu wrote:
> > > > Thanks for finding this issue, Cos.
> > > >
> > > > I opened HBASE-11422 and attached a patch there - there is no
> > dependency on
> > > > 2.2.0 if -Dhadoop-two.version=2.3.0 is specified.
> > > >
> > > > Cheers
> > > >
> > > >
> > > > On Thu, Jun 26, 2014 at 9:43 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >
> > > > > Weird... running
> > > > > mvn install dependency:tree -Dhadoop-two.version=2.3.0
> > > > > keeps pulling in 2.2.0 for test artifacts. An easy way to reproduce
> > it is
> > > > > to
> > > > > wipe out clean hadoop 2.2.0 artifacts from ~/.m2 as well as 0.98.2*
> > (if
> > > > > installed) and then run the command above. Something like this can
> > be seen:
> > > > >
> > > > > [INFO] +- org.apache.hbase:hbase-shell:jar:0.98.2:compile
> > > > > [INFO] | +- org.apache.hbase:hbase-prefix-tree:jar:0.98.2:runtime
> > > > > [INFO] | +- com.yammer.metrics:metrics-core:jar:2.1.2:compile
> > > > > [INFO] | +- org.jruby:jruby-complete:jar:1.6.8:compile
> > > > > [INFO] | +- org.apache.hadoop:hadoop-client:jar:2.3.0:compile
> > > > > [INFO] | | +-
> > > > > org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.3.0:compile
> > > > > [INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.3.0:compile
> > > > > [INFO] | | \-
> > > > > org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.3.0:compile
> > > > > [INFO] | +- org.apache.hadoop:hadoop-hdfs:jar:2.3.0:compile
> > > > > [INFO] | | \- commons-daemon:commons-daemon:jar:1.0.13:compile
> > > > > [INFO] | \-
> > org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.2.0:compile
> > > > >
> > > > > I have fixed a problem with site target, but still see this...
> > > > >
> > > > > Any ideas are very appreciated! Thanks.
> > > > > Cos
> > > > >
> > > > >
> > > > > On Tue, Jun 17, 2014 at 08:33PM, Konstantin Boudnik wrote:
> > > > > > Thank you very much Ted - that helped beautifully! I think the
> > build
> > > > > system
> > > > > > becomes a product on its own :)
> > > > > >
> > > > > > Regards,
> > > > > > Cos
> > > > > >
> > > > > > On Tue, Jun 17, 2014 at 12:40PM, Ted Yu wrote:
> > > > > > > Have you tried the following command ?
> > > > > > >
> > > > > > > mvn dependency:tree -Dhadoop-two.version=2.3.0
> > > > > > >
> > > > > > > The output of the above has 2.3.0 as the dependency.
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 17, 2014 at 11:40 AM, Konstantin Boudnik <
> > cos@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Guys,
> > > > > > > >
> > > > > > > > I have noticed an interesting problem with HBase 0.98 line. I
> > have
> > > > > ended
> > > > > > > > up with
> > > > > > > > a crappy Hadoop 2.2.0 artifacts in my local M2 cache - don't
> > ask me
> > > > > how
> > > > > > > > that
> > > > > > > > happen ;( - which causes compilation problems in HBase. And
> > that
> > > > > brought
> > > > > > > > this
> > > > > > > > whole issue into the light. Here's the essence of it:
> > > > > > > >
> > > > > > > > while running for hbase-server module
> > > > > > > > % mvn dependency:tree -Dhadoop.version=2.3.0
> > > > > > > > I am getting a reference to hadoop 2.2.0
> > > > > > > >
> > > > > > > > [INFO] +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
> > > > > > > > ..
> > > > > > > > [INFO] +- org.apache.hadoop:hadoop-auth:jar:2.2.0:compile
> > > > > > > > [INFO] +- org.apache.hadoop:hadoop-client:jar:2.2.0:compile
> > > > > > > >
> > > > > > > > This only happens for 0.98. On master the reference goes to
> > 2.4.0
> > > > > > > > The problem here is that -Dhadoop.version is being seemingly
> > > > > ignored, which
> > > > > > > > might lead to binaries with dependencies inconsistent with
> > what it
> > > > > was
> > > > > > > > built
> > > > > > > > against. I am not sure if this had came up before, but
> > certainly
> > > > > would
> > > > > > > > appreciate the community's input, if any.
> > > > > > > >
> > > > > > > > Thanks a lot!
> > > > > > > > --
> > > > > > > > Take care,
> > > > > > > > Cos
> > > > > > > >
> > > > > > > >
> > > > >
> >