You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by Viraj Jasani <vj...@apache.org> on 2021/03/02 18:00:38 UTC
Re: [DISCUSS] HBase 2.2.5 public maven artifacts are incompatible with Hadoop 3
Hi,
> However, up to now, we could use these artifacts with Hadoop 3.0 and they
> worked. (Or we just didn't hit the incompatibilities in our test suite.)
I have a question reg Phoenix 5.x supporting Hadoop 3 only.
I believe this means Phoenix 5.x cannot work with HBase 2.x that
is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2
jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars).
Is this correct?
If this is correct, shall we not support Hadoop 2 because HBase 2
still has default Hadoop profile: 2.0? Moreover, what is the root
cause behind Phoenix 5.x only supporting Hadoop 3.x?
As per the compatibility guidelines [1], HBase 2.3 is fully compatible
with Hadoop 2.10.x.
Apologies for these questions might be repeated ones but
somehow I still could not trace back the root cause.
1. https://hbase.apache.org/book.html#basic.prerequisites
On 2020/07/07 08:39:23, Istvan Toth <st...@apache.org> wrote:
> Hi!
>
> (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for this, but I
> am repeating it here for greater visibility)
>
> The HBase artifacts downloadable from maven central are built with Hadoop
> 2.x
>
> However, up to now, we could use these artifacts with Hadoop 3.0 and they
> worked. (Or we just didn't hit the incompatibilities in our test suite.)
>
> This seems to have changed with 2.2.5, as the public maven artifact doesn't
> work with Hadoop 3.0.3 or 3.1.2 .
>
> This is a known issue in HBase, and documented, but this means that
>
> Any client JAR we'd build with Hbase 2.2.5 would have the same problem.
> We cannot run our tests with Hbase 2.2.5
> HBase's suggested solution is to rebuild HBase with the Hadoop version used
> in the cluster, and use that. This, however, doesn't fit into our test or
> distribution process.
>
> I'm looking for your input on how to fix this.
>
> The options I could think of are:
>
> - Ask HBase to fix it for us (it doesn't seem to be a priority)
> - HBase could publish artifacts for Hadoop3
> - Or could try to stick the common subset of Hadoop APIs
> - Re-publish the official HBase releases built with Hadoop 3 under the
> org.apache.phoenix groupId
> - Hack up our build system to rebuild HBase from source, and use that for
> Phoenix
>
> None of these are particularly compelling, but I expect that this will have
> to be solved somehow if we want to keep up with future Hadoop releases.
>
> Looking forward to hearing your ideas and opinion on this
>
> Istvan
>
Re: [DISCUSS] HBase 2.2.5 public maven artifacts are incompatible
with Hadoop 3
Posted by Viraj Jasani <vj...@apache.org>.
Thanks for chiming in Geoffrey, Istvan and Duo.
> I'm not sure the usage of phoenix, if it is necessary, I think we could
try
> to publish extra artifacts which are compiled against hadoop 3.x.
Duo, recently published Phoenix 5.1.0 can run only with HBase 2.x compiled
with Hadoop 3.0 profile. While releasing Phoenix 5, all HBase supported
version tarballs are released based on each HBase profile supported by
Phoenix (e.g
https://dist.apache.org/repos/dist/release/phoenix/phoenix-5.1.0/)
What I am trying to convince is the fact that Phoenix 5 should support
Hadoop 2 compiled HBase (default) as well because not all in-place rolling
upgrades would like to deal with upgrading Hadoop while upgrading HBase
from HBase 1 to HBase 2 (and hence, Phoenix 4 to Phoenix 5). Geoffrey has
correctly mentioned all the points in details, Thanks Geoffrey.
Also, as for publishing HBase 2 extra artifacts compiled against Hadoop 3,
I think it's a good idea but we might not have to do it (unless
explicitly mentioned by our users, has anyone requested the same in the
past?) as we at least have Hadoop profiles available for both Hadoop 2 and
3, hence users do have a way to generate artifacts themselves if they are
not fine with default Hadoop 2.
For Phoenix too, having Hadoop profiles for using Hadoop 2 or 3 with any of
supported HBase versions would be a great option, and IMHO Phoenix also
might not need to release artifacts for both Hadoop profiles because users
have a way to get any combination they would like to have as long as we
have profiles available. If we decide to release all combinations, it might
look something
like phoenix-hbase-hadoop-2.1-5.1.0-3.1.4,
phoenix-hbase-hadoop-2.1-5.1.0-2.10.0
and so on. Anyways, this decision could be taken up on a separate thread :)
On Fri, Mar 5, 2021 at 7:07 PM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
> The reason why we only publish artifacts compiled with hadoop 2 for HBase
> is that, you can use these artifacts to run HBase against a 3.x HDFS
> cluster, no problem, you do not need to replace the hadoop jars in HBase to
> hadoop 3, as we will not make use any new features at client side...
>
> I'm not sure the usage of phoenix, if it is necessary, I think we could try
> to publish extra artifacts which are compiled against hadoop 3.x.
>
> Thanks.
>
> Istvan Toth <st...@apache.org> 于2021年3月5日周五 上午1:40写道：
>
> > Hi!
> >
> > Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track this.
> >
> > Istvan
> >
> > On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <gj...@apache.org>
> > wrote:
> >
> > > If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it
> > would
> > > be a huge help if we could do that. A couple different reasons:
> > >
> > > 1. Ease of development. The workaround described in BUILDING.md adds
> > extra
> > > friction to getting an environment setup, and it has to be redone for
> > each
> > > patch release of HBase an engineer needs to work with. It's confusing
> for
> > > both new contributors and experienced ones who don't usually work with
> > 5.x
> > > or haven't lately.
> > >
> > > 2. Ease of upgrade. At my dayjob, we try to avoid upgrading
> combinations
> > of
> > > Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing
> > and
> > > debugging simpler. HBase 2.x and Phoenix 5.x have to be done in
> lockstep,
> > > and there's no avoiding that. But since HBase 1.x doesn't support
> Hadoop
> > 3,
> > > and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an
> > > upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop,
> > > HBase and Phoenix to new major versions simultaneously. It would be
> > really
> > > nice to do HBase / Phoenix first, and then Hadoop later.
> > >
> > > Geoffrey
> > >
> > >
> > >
> > > On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <stoty@cloudera.com.invalid
> >
> > > wrote:
> > >
> > > > Hi!
> > > >
> > > > For the first part of your question: Yes, you are correct.
> > > >
> > > > I think that there is no inherent technical issue that would make it
> > > > impossible to support Hbase 2 on Hadoop 2 for Phoenix.
> > > >
> > > > While this happened before my time, I guess that supporting Hadoop 3
> > only
> > > > was a decision that partly stemmed from the then current Phoenix
> maven
> > > > setup,
> > > > which wasn't equipped to support multiple profiles (see the multiple
> > > > branches for the HBase 1.x versions), and partly because this was the
> > > > configuration that
> > > > the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested
> in.
> > > >
> > > > I don't know how much work and additional complexity it would take to
> > add
> > > > Hadoop2 support for master.
> > > > My guess is that it would need some major changes on the maven side
> for
> > > > building, packaging, testing, and docs,
> > > > but probably little in the way of Java code changes.
> > > >
> > > > It would also make providing binaries for Phoenix, connectors and PQS
> > > even
> > > > more hairy than now.
> > > >
> > > > If anyone has more historical context, or a better idea of what it
> > would
> > > > take to do this, please let us know!
> > > >
> > > > Istvan
> > > >
> > > >
> > > >
> > > > On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <vj...@apache.org>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > > However, up to now, we could use these artifacts with Hadoop 3.0
> > and
> > > > they
> > > > > > worked. (Or we just didn't hit the incompatibilities in our test
> > > > suite.)
> > > > >
> > > > > I have a question reg Phoenix 5.x supporting Hadoop 3 only.
> > > > > I believe this means Phoenix 5.x cannot work with HBase 2.x that
> > > > > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2
> > > > > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars).
> > > > > Is this correct?
> > > > >
> > > > > If this is correct, shall we not support Hadoop 2 because HBase 2
> > > > > still has default Hadoop profile: 2.0? Moreover, what is the root
> > > > > cause behind Phoenix 5.x only supporting Hadoop 3.x?
> > > > > As per the compatibility guidelines [1], HBase 2.3 is fully
> > compatible
> > > > > with Hadoop 2.10.x.
> > > > >
> > > > > Apologies for these questions might be repeated ones but
> > > > > somehow I still could not trace back the root cause.
> > > > >
> > > > >
> > > > > 1. https://hbase.apache.org/book.html#basic.prerequisites
> > > > >
> > > > >
> > > > > On 2020/07/07 08:39:23, Istvan Toth <st...@apache.org> wrote:
> > > > > > Hi!
> > > > > >
> > > > > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for
> > > this,
> > > > > but I
> > > > > > am repeating it here for greater visibility)
> > > > > >
> > > > > > The HBase artifacts downloadable from maven central are built
> with
> > > > Hadoop
> > > > > > 2.x
> > > > > >
> > > > > > However, up to now, we could use these artifacts with Hadoop 3.0
> > and
> > > > they
> > > > > > worked. (Or we just didn't hit the incompatibilities in our test
> > > > suite.)
> > > > > >
> > > > > > This seems to have changed with 2.2.5, as the public maven
> artifact
> > > > > doesn't
> > > > > > work with Hadoop 3.0.3 or 3.1.2 .
> > > > > >
> > > > > > This is a known issue in HBase, and documented, but this means
> that
> > > > > >
> > > > > > Any client JAR we'd build with Hbase 2.2.5 would have the same
> > > problem.
> > > > > > We cannot run our tests with Hbase 2.2.5
> > > > > > HBase's suggested solution is to rebuild HBase with the Hadoop
> > > version
> > > > > used
> > > > > > in the cluster, and use that. This, however, doesn't fit into our
> > > test
> > > > or
> > > > > > distribution process.
> > > > > >
> > > > > > I'm looking for your input on how to fix this.
> > > > > >
> > > > > > The options I could think of are:
> > > > > >
> > > > > > - Ask HBase to fix it for us (it doesn't seem to be a priority)
> > > > > > - HBase could publish artifacts for Hadoop3
> > > > > > - Or could try to stick the common subset of Hadoop APIs
> > > > > > - Re-publish the official HBase releases built with Hadoop 3
> under
> > > the
> > > > > > org.apache.phoenix groupId
> > > > > > - Hack up our build system to rebuild HBase from source, and use
> > that
> > > > for
> > > > > > Phoenix
> > > > > >
> > > > > > None of these are particularly compelling, but I expect that this
> > > will
> > > > > have
> > > > > > to be solved somehow if we want to keep up with future Hadoop
> > > releases.
> > > > > >
> > > > > > Looking forward to hearing your ideas and opinion on this
> > > > > >
> > > > > > Istvan
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > *István Tóth* | Staff Software Engineer
> > > > stoty@cloudera.com <https://www.cloudera.com>
> > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > <https://www.cloudera.com/>
> > > > ------------------------------
> > > >
> > >
> >
>
Re: [DISCUSS] HBase 2.2.5 public maven artifacts are incompatible
with Hadoop 3
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
The reason why we only publish artifacts compiled with hadoop 2 for HBase
is that, you can use these artifacts to run HBase against a 3.x HDFS
cluster, no problem, you do not need to replace the hadoop jars in HBase to
hadoop 3, as we will not make use any new features at client side...
I'm not sure the usage of phoenix, if it is necessary, I think we could try
to publish extra artifacts which are compiled against hadoop 3.x.
Thanks.
Istvan Toth <st...@apache.org> 于2021年3月5日周五 上午1:40写道：
> Hi!
>
> Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track this.
>
> Istvan
>
> On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <gj...@apache.org>
> wrote:
>
> > If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it
> would
> > be a huge help if we could do that. A couple different reasons:
> >
> > 1. Ease of development. The workaround described in BUILDING.md adds
> extra
> > friction to getting an environment setup, and it has to be redone for
> each
> > patch release of HBase an engineer needs to work with. It's confusing for
> > both new contributors and experienced ones who don't usually work with
> 5.x
> > or haven't lately.
> >
> > 2. Ease of upgrade. At my dayjob, we try to avoid upgrading combinations
> of
> > Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing
> and
> > debugging simpler. HBase 2.x and Phoenix 5.x have to be done in lockstep,
> > and there's no avoiding that. But since HBase 1.x doesn't support Hadoop
> 3,
> > and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an
> > upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop,
> > HBase and Phoenix to new major versions simultaneously. It would be
> really
> > nice to do HBase / Phoenix first, and then Hadoop later.
> >
> > Geoffrey
> >
> >
> >
> > On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <st...@cloudera.com.invalid>
> > wrote:
> >
> > > Hi!
> > >
> > > For the first part of your question: Yes, you are correct.
> > >
> > > I think that there is no inherent technical issue that would make it
> > > impossible to support Hbase 2 on Hadoop 2 for Phoenix.
> > >
> > > While this happened before my time, I guess that supporting Hadoop 3
> only
> > > was a decision that partly stemmed from the then current Phoenix maven
> > > setup,
> > > which wasn't equipped to support multiple profiles (see the multiple
> > > branches for the HBase 1.x versions), and partly because this was the
> > > configuration that
> > > the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in.
> > >
> > > I don't know how much work and additional complexity it would take to
> add
> > > Hadoop2 support for master.
> > > My guess is that it would need some major changes on the maven side for
> > > building, packaging, testing, and docs,
> > > but probably little in the way of Java code changes.
> > >
> > > It would also make providing binaries for Phoenix, connectors and PQS
> > even
> > > more hairy than now.
> > >
> > > If anyone has more historical context, or a better idea of what it
> would
> > > take to do this, please let us know!
> > >
> > > Istvan
> > >
> > >
> > >
> > > On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <vj...@apache.org>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > > However, up to now, we could use these artifacts with Hadoop 3.0
> and
> > > they
> > > > > worked. (Or we just didn't hit the incompatibilities in our test
> > > suite.)
> > > >
> > > > I have a question reg Phoenix 5.x supporting Hadoop 3 only.
> > > > I believe this means Phoenix 5.x cannot work with HBase 2.x that
> > > > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2
> > > > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars).
> > > > Is this correct?
> > > >
> > > > If this is correct, shall we not support Hadoop 2 because HBase 2
> > > > still has default Hadoop profile: 2.0? Moreover, what is the root
> > > > cause behind Phoenix 5.x only supporting Hadoop 3.x?
> > > > As per the compatibility guidelines [1], HBase 2.3 is fully
> compatible
> > > > with Hadoop 2.10.x.
> > > >
> > > > Apologies for these questions might be repeated ones but
> > > > somehow I still could not trace back the root cause.
> > > >
> > > >
> > > > 1. https://hbase.apache.org/book.html#basic.prerequisites
> > > >
> > > >
> > > > On 2020/07/07 08:39:23, Istvan Toth <st...@apache.org> wrote:
> > > > > Hi!
> > > > >
> > > > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for
> > this,
> > > > but I
> > > > > am repeating it here for greater visibility)
> > > > >
> > > > > The HBase artifacts downloadable from maven central are built with
> > > Hadoop
> > > > > 2.x
> > > > >
> > > > > However, up to now, we could use these artifacts with Hadoop 3.0
> and
> > > they
> > > > > worked. (Or we just didn't hit the incompatibilities in our test
> > > suite.)
> > > > >
> > > > > This seems to have changed with 2.2.5, as the public maven artifact
> > > > doesn't
> > > > > work with Hadoop 3.0.3 or 3.1.2 .
> > > > >
> > > > > This is a known issue in HBase, and documented, but this means that
> > > > >
> > > > > Any client JAR we'd build with Hbase 2.2.5 would have the same
> > problem.
> > > > > We cannot run our tests with Hbase 2.2.5
> > > > > HBase's suggested solution is to rebuild HBase with the Hadoop
> > version
> > > > used
> > > > > in the cluster, and use that. This, however, doesn't fit into our
> > test
> > > or
> > > > > distribution process.
> > > > >
> > > > > I'm looking for your input on how to fix this.
> > > > >
> > > > > The options I could think of are:
> > > > >
> > > > > - Ask HBase to fix it for us (it doesn't seem to be a priority)
> > > > > - HBase could publish artifacts for Hadoop3
> > > > > - Or could try to stick the common subset of Hadoop APIs
> > > > > - Re-publish the official HBase releases built with Hadoop 3 under
> > the
> > > > > org.apache.phoenix groupId
> > > > > - Hack up our build system to rebuild HBase from source, and use
> that
> > > for
> > > > > Phoenix
> > > > >
> > > > > None of these are particularly compelling, but I expect that this
> > will
> > > > have
> > > > > to be solved somehow if we want to keep up with future Hadoop
> > releases.
> > > > >
> > > > > Looking forward to hearing your ideas and opinion on this
> > > > >
> > > > > Istvan
> > > > >
> > > >
> > >
> > >
> > > --
> > > *István Tóth* | Staff Software Engineer
> > > stoty@cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > <https://www.cloudera.com/>
> > > ------------------------------
> > >
> >
>
Re: [DISCUSS] HBase 2.2.5 public maven artifacts are incompatible
with Hadoop 3
Posted by Istvan Toth <st...@apache.org>.
Hi!
Opened https://issues.apache.org/jira/browse/PHOENIX-6404 to track this.
Istvan
On Wed, Mar 3, 2021 at 11:34 PM Geoffrey Jacoby <gj...@apache.org> wrote:
> If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it would
> be a huge help if we could do that. A couple different reasons:
>
> 1. Ease of development. The workaround described in BUILDING.md adds extra
> friction to getting an environment setup, and it has to be redone for each
> patch release of HBase an engineer needs to work with. It's confusing for
> both new contributors and experienced ones who don't usually work with 5.x
> or haven't lately.
>
> 2. Ease of upgrade. At my dayjob, we try to avoid upgrading combinations of
> Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing and
> debugging simpler. HBase 2.x and Phoenix 5.x have to be done in lockstep,
> and there's no avoiding that. But since HBase 1.x doesn't support Hadoop 3,
> and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an
> upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop,
> HBase and Phoenix to new major versions simultaneously. It would be really
> nice to do HBase / Phoenix first, and then Hadoop later.
>
> Geoffrey
>
>
>
> On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <st...@cloudera.com.invalid>
> wrote:
>
> > Hi!
> >
> > For the first part of your question: Yes, you are correct.
> >
> > I think that there is no inherent technical issue that would make it
> > impossible to support Hbase 2 on Hadoop 2 for Phoenix.
> >
> > While this happened before my time, I guess that supporting Hadoop 3 only
> > was a decision that partly stemmed from the then current Phoenix maven
> > setup,
> > which wasn't equipped to support multiple profiles (see the multiple
> > branches for the HBase 1.x versions), and partly because this was the
> > configuration that
> > the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in.
> >
> > I don't know how much work and additional complexity it would take to add
> > Hadoop2 support for master.
> > My guess is that it would need some major changes on the maven side for
> > building, packaging, testing, and docs,
> > but probably little in the way of Java code changes.
> >
> > It would also make providing binaries for Phoenix, connectors and PQS
> even
> > more hairy than now.
> >
> > If anyone has more historical context, or a better idea of what it would
> > take to do this, please let us know!
> >
> > Istvan
> >
> >
> >
> > On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <vj...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > > However, up to now, we could use these artifacts with Hadoop 3.0 and
> > they
> > > > worked. (Or we just didn't hit the incompatibilities in our test
> > suite.)
> > >
> > > I have a question reg Phoenix 5.x supporting Hadoop 3 only.
> > > I believe this means Phoenix 5.x cannot work with HBase 2.x that
> > > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2
> > > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars).
> > > Is this correct?
> > >
> > > If this is correct, shall we not support Hadoop 2 because HBase 2
> > > still has default Hadoop profile: 2.0? Moreover, what is the root
> > > cause behind Phoenix 5.x only supporting Hadoop 3.x?
> > > As per the compatibility guidelines [1], HBase 2.3 is fully compatible
> > > with Hadoop 2.10.x.
> > >
> > > Apologies for these questions might be repeated ones but
> > > somehow I still could not trace back the root cause.
> > >
> > >
> > > 1. https://hbase.apache.org/book.html#basic.prerequisites
> > >
> > >
> > > On 2020/07/07 08:39:23, Istvan Toth <st...@apache.org> wrote:
> > > > Hi!
> > > >
> > > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for
> this,
> > > but I
> > > > am repeating it here for greater visibility)
> > > >
> > > > The HBase artifacts downloadable from maven central are built with
> > Hadoop
> > > > 2.x
> > > >
> > > > However, up to now, we could use these artifacts with Hadoop 3.0 and
> > they
> > > > worked. (Or we just didn't hit the incompatibilities in our test
> > suite.)
> > > >
> > > > This seems to have changed with 2.2.5, as the public maven artifact
> > > doesn't
> > > > work with Hadoop 3.0.3 or 3.1.2 .
> > > >
> > > > This is a known issue in HBase, and documented, but this means that
> > > >
> > > > Any client JAR we'd build with Hbase 2.2.5 would have the same
> problem.
> > > > We cannot run our tests with Hbase 2.2.5
> > > > HBase's suggested solution is to rebuild HBase with the Hadoop
> version
> > > used
> > > > in the cluster, and use that. This, however, doesn't fit into our
> test
> > or
> > > > distribution process.
> > > >
> > > > I'm looking for your input on how to fix this.
> > > >
> > > > The options I could think of are:
> > > >
> > > > - Ask HBase to fix it for us (it doesn't seem to be a priority)
> > > > - HBase could publish artifacts for Hadoop3
> > > > - Or could try to stick the common subset of Hadoop APIs
> > > > - Re-publish the official HBase releases built with Hadoop 3 under
> the
> > > > org.apache.phoenix groupId
> > > > - Hack up our build system to rebuild HBase from source, and use that
> > for
> > > > Phoenix
> > > >
> > > > None of these are particularly compelling, but I expect that this
> will
> > > have
> > > > to be solved somehow if we want to keep up with future Hadoop
> releases.
> > > >
> > > > Looking forward to hearing your ideas and opinion on this
> > > >
> > > > Istvan
> > > >
> > >
> >
> >
> > --
> > *István Tóth* | Staff Software Engineer
> > stoty@cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > <https://www.cloudera.com/>
> > ------------------------------
> >
>
Re: [DISCUSS] HBase 2.2.5 public maven artifacts are incompatible
with Hadoop 3
Posted by Geoffrey Jacoby <gj...@apache.org>.
If the effort to support Hadoop 2 in Phoenix 5 isn't prohibitive, it would
be a huge help if we could do that. A couple different reasons:
1. Ease of development. The workaround described in BUILDING.md adds extra
friction to getting an environment setup, and it has to be redone for each
patch release of HBase an engineer needs to work with. It's confusing for
both new contributors and experienced ones who don't usually work with 5.x
or haven't lately.
2. Ease of upgrade. At my dayjob, we try to avoid upgrading combinations of
Hadoop, HBase, and/or Phoenix at the same time to make upgrade testing and
debugging simpler. HBase 2.x and Phoenix 5.x have to be done in lockstep,
and there's no avoiding that. But since HBase 1.x doesn't support Hadoop 3,
and Phoenix 5 doesn't support HBase 1.x or Hadoop 2, that means that an
upgrade from any Phoenix 4 to any Phoenix 5 requires upgrading Hadoop,
HBase and Phoenix to new major versions simultaneously. It would be really
nice to do HBase / Phoenix first, and then Hadoop later.
Geoffrey
On Wed, Mar 3, 2021 at 12:55 AM Istvan Toth <st...@cloudera.com.invalid>
wrote:
> Hi!
>
> For the first part of your question: Yes, you are correct.
>
> I think that there is no inherent technical issue that would make it
> impossible to support Hbase 2 on Hadoop 2 for Phoenix.
>
> While this happened before my time, I guess that supporting Hadoop 3 only
> was a decision that partly stemmed from the then current Phoenix maven
> setup,
> which wasn't equipped to support multiple profiles (see the multiple
> branches for the HBase 1.x versions), and partly because this was the
> configuration that
> the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in.
>
> I don't know how much work and additional complexity it would take to add
> Hadoop2 support for master.
> My guess is that it would need some major changes on the maven side for
> building, packaging, testing, and docs,
> but probably little in the way of Java code changes.
>
> It would also make providing binaries for Phoenix, connectors and PQS even
> more hairy than now.
>
> If anyone has more historical context, or a better idea of what it would
> take to do this, please let us know!
>
> Istvan
>
>
>
> On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <vj...@apache.org> wrote:
>
> > Hi,
> >
> > > However, up to now, we could use these artifacts with Hadoop 3.0 and
> they
> > > worked. (Or we just didn't hit the incompatibilities in our test
> suite.)
> >
> > I have a question reg Phoenix 5.x supporting Hadoop 3 only.
> > I believe this means Phoenix 5.x cannot work with HBase 2.x that
> > is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2
> > jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars).
> > Is this correct?
> >
> > If this is correct, shall we not support Hadoop 2 because HBase 2
> > still has default Hadoop profile: 2.0? Moreover, what is the root
> > cause behind Phoenix 5.x only supporting Hadoop 3.x?
> > As per the compatibility guidelines [1], HBase 2.3 is fully compatible
> > with Hadoop 2.10.x.
> >
> > Apologies for these questions might be repeated ones but
> > somehow I still could not trace back the root cause.
> >
> >
> > 1. https://hbase.apache.org/book.html#basic.prerequisites
> >
> >
> > On 2020/07/07 08:39:23, Istvan Toth <st...@apache.org> wrote:
> > > Hi!
> > >
> > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for this,
> > but I
> > > am repeating it here for greater visibility)
> > >
> > > The HBase artifacts downloadable from maven central are built with
> Hadoop
> > > 2.x
> > >
> > > However, up to now, we could use these artifacts with Hadoop 3.0 and
> they
> > > worked. (Or we just didn't hit the incompatibilities in our test
> suite.)
> > >
> > > This seems to have changed with 2.2.5, as the public maven artifact
> > doesn't
> > > work with Hadoop 3.0.3 or 3.1.2 .
> > >
> > > This is a known issue in HBase, and documented, but this means that
> > >
> > > Any client JAR we'd build with Hbase 2.2.5 would have the same problem.
> > > We cannot run our tests with Hbase 2.2.5
> > > HBase's suggested solution is to rebuild HBase with the Hadoop version
> > used
> > > in the cluster, and use that. This, however, doesn't fit into our test
> or
> > > distribution process.
> > >
> > > I'm looking for your input on how to fix this.
> > >
> > > The options I could think of are:
> > >
> > > - Ask HBase to fix it for us (it doesn't seem to be a priority)
> > > - HBase could publish artifacts for Hadoop3
> > > - Or could try to stick the common subset of Hadoop APIs
> > > - Re-publish the official HBase releases built with Hadoop 3 under the
> > > org.apache.phoenix groupId
> > > - Hack up our build system to rebuild HBase from source, and use that
> for
> > > Phoenix
> > >
> > > None of these are particularly compelling, but I expect that this will
> > have
> > > to be solved somehow if we want to keep up with future Hadoop releases.
> > >
> > > Looking forward to hearing your ideas and opinion on this
> > >
> > > Istvan
> > >
> >
>
>
> --
> *István Tóth* | Staff Software Engineer
> stoty@cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> <https://www.cloudera.com/>
> ------------------------------
>
Re: [DISCUSS] HBase 2.2.5 public maven artifacts are incompatible
with Hadoop 3
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Hi!
For the first part of your question: Yes, you are correct.
I think that there is no inherent technical issue that would make it
impossible to support Hbase 2 on Hadoop 2 for Phoenix.
While this happened before my time, I guess that supporting Hadoop 3 only
was a decision that partly stemmed from the then current Phoenix maven
setup,
which wasn't equipped to support multiple profiles (see the multiple
branches for the HBase 1.x versions), and partly because this was the
configuration that
the implementers (Ankit, Josh, Rajeshbabu, Sergey) were interested in.
I don't know how much work and additional complexity it would take to add
Hadoop2 support for master.
My guess is that it would need some major changes on the maven side for
building, packaging, testing, and docs,
but probably little in the way of Java code changes.
It would also make providing binaries for Phoenix, connectors and PQS even
more hairy than now.
If anyone has more historical context, or a better idea of what it would
take to do this, please let us know!
Istvan
On Tue, Mar 2, 2021 at 7:00 PM Viraj Jasani <vj...@apache.org> wrote:
> Hi,
>
> > However, up to now, we could use these artifacts with Hadoop 3.0 and they
> > worked. (Or we just didn't hit the incompatibilities in our test suite.)
>
> I have a question reg Phoenix 5.x supporting Hadoop 3 only.
> I believe this means Phoenix 5.x cannot work with HBase 2.x that
> is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2
> jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars).
> Is this correct?
>
> If this is correct, shall we not support Hadoop 2 because HBase 2
> still has default Hadoop profile: 2.0? Moreover, what is the root
> cause behind Phoenix 5.x only supporting Hadoop 3.x?
> As per the compatibility guidelines [1], HBase 2.3 is fully compatible
> with Hadoop 2.10.x.
>
> Apologies for these questions might be repeated ones but
> somehow I still could not trace back the root cause.
>
>
> 1. https://hbase.apache.org/book.html#basic.prerequisites
>
>
> On 2020/07/07 08:39:23, Istvan Toth <st...@apache.org> wrote:
> > Hi!
> >
> > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for this,
> but I
> > am repeating it here for greater visibility)
> >
> > The HBase artifacts downloadable from maven central are built with Hadoop
> > 2.x
> >
> > However, up to now, we could use these artifacts with Hadoop 3.0 and they
> > worked. (Or we just didn't hit the incompatibilities in our test suite.)
> >
> > This seems to have changed with 2.2.5, as the public maven artifact
> doesn't
> > work with Hadoop 3.0.3 or 3.1.2 .
> >
> > This is a known issue in HBase, and documented, but this means that
> >
> > Any client JAR we'd build with Hbase 2.2.5 would have the same problem.
> > We cannot run our tests with Hbase 2.2.5
> > HBase's suggested solution is to rebuild HBase with the Hadoop version
> used
> > in the cluster, and use that. This, however, doesn't fit into our test or
> > distribution process.
> >
> > I'm looking for your input on how to fix this.
> >
> > The options I could think of are:
> >
> > - Ask HBase to fix it for us (it doesn't seem to be a priority)
> > - HBase could publish artifacts for Hadoop3
> > - Or could try to stick the common subset of Hadoop APIs
> > - Re-publish the official HBase releases built with Hadoop 3 under the
> > org.apache.phoenix groupId
> > - Hack up our build system to rebuild HBase from source, and use that for
> > Phoenix
> >
> > None of these are particularly compelling, but I expect that this will
> have
> > to be solved somehow if we want to keep up with future Hadoop releases.
> >
> > Looking forward to hearing your ideas and opinion on this
> >
> > Istvan
> >
>
--
*István Tóth* | Staff Software Engineer
stoty@cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
<https://www.cloudera.com/>
------------------------------