You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by Curtis Howard <cu...@gmail.com> on 2018/11/01 18:30:35 UTC
Re: Phoenix 5.0 + Cloudera CDH 6.0 Integration
Hi Josh, Pedro,
I've made a first attempt at this (CDH 6 parcel) on top of the latest
version of the master branch:
https://github.com/curtishoward/phoenix/tree/PHOENIX-4956_on_master_nov1_59a7dd1
Summary of changes:
--> Dependency changes and other adjustments, based on similar previous
PHOENIX-4372 work that I found:
https://github.com/apache/phoenix/commit/024f0f22a5929da6f095dc0025b8e899e2f0c47b
--> revert PHOENIX-4423 (CDH 6 uses Hive 2.1.1)
The branch builds and tests successfully - after creating and installing
the parcel on a CDH6.0.0 cluster (HBase 2.0.0), I'm seeing a combination of
the following errors, after attempting to start phoenix-sqlline.py (as the
user 'systest'):
Error: ERROR 2006 (INT08): Incompatible jars detected between client and
server. Ensure that phoenix-[version]-server.jar is put on the classpath of
HBase in every region server: Failed after attempts=11, exceptions:
Thu Nov 01 14:08:04 EDT 2018,
RpcRetryingCaller{globalStartTime=1541095683837, pause=100,
maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
SYSTEM.CATALOG,,1536076818211.9389e912759fb1abcb4b32a5ddd2b779. is not
online on curtis-c6-2.domain.com,16020,1541087017170
...
From the region server though (curtis-c6-2.domain.com), I can see that the
sqlline client connection attempt made it through?
2:08:03.865 PM INFO Server Auth successful for systest (auth:SIMPLE)
2:08:03.865 PM INFO Server Connection from <curtis-c6-1 IP>
port: 50188 with version info: version: "2.0.0-cdh6.0.0" url:
"git://xxxxxx.domain.com/container.common/build/cdh/hbase/2.0.0-cdh6.0.0/source"
revision: "" user: "jenkins" date: "Fri Aug 17 15:50:53 PDT 2018"
src_checksum: "58a2f45827fa231e6d4ef1d11350b5ac" version_major: 2
version_minor: 0
Is it possible I'm just missing some JARs on my classpath?
[systest@curtis-c6-1 ~]$ find /opt/cloudera/ -type f | grep phoenix | grep
jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-client.jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-server.jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-queryserver.jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-thin-client.jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-hive.jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/lib/phoenix-pig-5.1.0-cdh6.0.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/lib/phoenix-flume-5.1.0-cdh6.0.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/lib/phoenix-core-5.1.0-cdh6.0.0.jar
...
Thanks in advance for any advice
Curtis
On 2018/09/14 22:36:49, Josh Elser <e....@apache.org> wrote:
> If you do decide you want to try this, you may want to try using the >
> master branch (5.1.0-SNAPSHOT), and then help us drive to a 5.1.0
release.>
>
> I just pushed a change to the shading for the phoenix-server.jar earlier >
> this week which should remove some headache due to conflicting versions >
> of dependencies on the RegionServer classpath.>
>
> On 9/14/18 2:17 AM, Pedro Boado wrote:>
> > Hi Curtis,>
> > >
> > As far as I am aware nobody is working on it.>
> > >
> > The approach I followed is, starting from master:>
> > - Change parent pom to use cloudera dependencies instead of Apache's
for>
> > compilation>
> > - Make it compile (relatively easy for 4.x)>
> > - Make it pass tests (not that easy)>
> > - Once it passed, wrote the parcel module.>
> > >
> > There aren't a lot of code differences between cdh and base branches so
I'd>
> > try to follow the same approach - basing it on changes in the current>
> > differences-.>
> > >
> > I really think this effort is needed to try to bring both branches
again as>
> > close to each other as possible. Maybe some code differences are no
longer>
> > needed because CDH is already closer to the Apache version of Hadoop
3.0>
> > and Hbase 2.0 than it was when I did the migration.>
> > >
> > Hope it helps,>
> > Pedro.>
> > >
> > On 14 Sep 2018 04:57, "Curtis Howard" <cu...@gmail.com> wrote:>
> > >
> > Hi all,>
> > >
> > Is there anyone working towards Phoenix 5.0 / Cloudera (CDH) 6.0>
> > integration at this point? I could not find any related JIRA for this>
> > after a quick search, and wanted to check here first before adding one.>
> > >
> > If I were to attempt this myself, is there a suggested approach? I can
see>
> > from previous 4.x-cdh5.* branches supporting these releases that the>
> > changes for PHOENIX-4372 (
https://issues.apache.org/jira/browse/PHOENIX-4372>
> > )>
> > move the builds to CDH dependencies - for example:>
> >
https://github.com/apache/phoenix/commit/024f0f22a5929da6f095dc0025b8e899e2f0c47b
>
> > >
> > Would following the pattern of that commit (or attempting a
cherry-pick)>
> > onto the the v5.0.0-HBase-2.0 tagged release (>
> > https://github.com/apache/phoenix/tree/v5.0.0-HBase-2.0) be a
reasonable>
> > starting point?>
> > >
> > Thanks in advance>
> > >
> > Curtis>
> > >
>
Re: Phoenix 5.0 + Cloudera CDH 6.0 Integration
Posted by Pedro Boado <pe...@gmail.com>.
Hi Curtis,
Sorry for the late reply. I've been building it locally from your branch. I
believe long IT have not been run ( mvn clean verify ) . I believe more
work is needed. But it is a good start. We better continue conversation in
PHOENIX-4956.
Cheers,
Pedro.
On Thu, 1 Nov 2018 at 18:30, Curtis Howard <cu...@gmail.com>
wrote:
> Hi Josh, Pedro,
>
> I've made a first attempt at this (CDH 6 parcel) on top of the latest
> version of the master branch:
>
> https://github.com/curtishoward/phoenix/tree/PHOENIX-4956_on_master_nov1_59a7dd1
>
> Summary of changes:
> --> Dependency changes and other adjustments, based on similar previous
> PHOENIX-4372 work that I found:
>
> https://github.com/apache/phoenix/commit/024f0f22a5929da6f095dc0025b8e899e2f0c47b
> --> revert PHOENIX-4423 (CDH 6 uses Hive 2.1.1)
>
> The branch builds and tests successfully - after creating and installing
> the parcel on a CDH6.0.0 cluster (HBase 2.0.0), I'm seeing a combination of
> the following errors, after attempting to start phoenix-sqlline.py (as the
> user 'systest'):
>
> Error: ERROR 2006 (INT08): Incompatible jars detected between client and
> server. Ensure that phoenix-[version]-server.jar is put on the classpath of
> HBase in every region server: Failed after attempts=11, exceptions:
> Thu Nov 01 14:08:04 EDT 2018,
> RpcRetryingCaller{globalStartTime=1541095683837, pause=100,
> maxAttempts=11}, org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException:
> SYSTEM.CATALOG,,1536076818211.9389e912759fb1abcb4b32a5ddd2b779. is not
> online on curtis-c6-2.domain.com,16020,1541087017170
> ...
>
> From the region server though (curtis-c6-2.domain.com), I can see that the
> sqlline client connection attempt made it through?
>
> 2:08:03.865 PM INFO Server Auth successful for systest
> (auth:SIMPLE)
>
> 2:08:03.865 PM INFO Server Connection from <curtis-c6-1 IP>
> port: 50188 with version info: version: "2.0.0-cdh6.0.0" url:
> "git://
> xxxxxx.domain.com/container.common/build/cdh/hbase/2.0.0-cdh6.0.0/source"
> revision: "" user: "jenkins" date: "Fri Aug 17 15:50:53 PDT 2018"
> src_checksum: "58a2f45827fa231e6d4ef1d11350b5ac" version_major: 2
> version_minor: 0
>
>
>
> Is it possible I'm just missing some JARs on my classpath?
>
> [systest@curtis-c6-1 ~]$ find /opt/cloudera/ -type f | grep phoenix | grep
> jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-client.jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-server.jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-queryserver.jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-thin-client.jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/phoenix-5.1.0-cdh6.0.0-hive.jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/lib/phoenix-pig-5.1.0-cdh6.0.0.jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/lib/phoenix-flume-5.1.0-cdh6.0.0.jar
>
> /opt/cloudera/parcels/APACHE_PHOENIX-5.1.0-cdh6.0.0.p0.3/lib/phoenix/lib/phoenix-core-5.1.0-cdh6.0.0.jar
> ...
>
> Thanks in advance for any advice
> Curtis
>
>
> On 2018/09/14 22:36:49, Josh Elser <e....@apache.org> wrote:
> > If you do decide you want to try this, you may want to try using the >
> > master branch (5.1.0-SNAPSHOT), and then help us drive to a 5.1.0
> release.>
> >
> > I just pushed a change to the shading for the phoenix-server.jar earlier
> >
>
> > this week which should remove some headache due to conflicting versions >
> > of dependencies on the RegionServer classpath.>
> >
> > On 9/14/18 2:17 AM, Pedro Boado wrote:>
> > > Hi Curtis,>
> > > >
> > > As far as I am aware nobody is working on it.>
> > > >
> > > The approach I followed is, starting from master:>
> > > - Change parent pom to use cloudera dependencies instead of Apache's
> for>
> > > compilation>
> > > - Make it compile (relatively easy for 4.x)>
> > > - Make it pass tests (not that easy)>
> > > - Once it passed, wrote the parcel module.>
> > > >
> > > There aren't a lot of code differences between cdh and base branches so
> I'd>
> > > try to follow the same approach - basing it on changes in the current>
> > > differences-.>
> > > >
> > > I really think this effort is needed to try to bring both branches
> again as>
> > > close to each other as possible. Maybe some code differences are no
> longer>
> > > needed because CDH is already closer to the Apache version of Hadoop
> 3.0>
> > > and Hbase 2.0 than it was when I did the migration.>
> > > >
> > > Hope it helps,>
> > > Pedro.>
> > > >
> > > On 14 Sep 2018 04:57, "Curtis Howard" <cu...@gmail.com> wrote:>
> > > >
> > > Hi all,>
> > > >
> > > Is there anyone working towards Phoenix 5.0 / Cloudera (CDH) 6.0>
> > > integration at this point? I could not find any related JIRA for this>
> > > after a quick search, and wanted to check here first before adding
> one.>
>
> > > >
> > > If I were to attempt this myself, is there a suggested approach? I can
> see>
> > > from previous 4.x-cdh5.* branches supporting these releases that the>
> > > changes for PHOENIX-4372 (
> https://issues.apache.org/jira/browse/PHOENIX-4372>
> > > )>
> > > move the builds to CDH dependencies - for example:>
> > >
>
> https://github.com/apache/phoenix/commit/024f0f22a5929da6f095dc0025b8e899e2f0c47b
> >
> > > >
> > > Would following the pattern of that commit (or attempting a
> cherry-pick)>
> > > onto the the v5.0.0-HBase-2.0 tagged release (>
> > > https://github.com/apache/phoenix/tree/v5.0.0-HBase-2.0) be a
> reasonable>
> > > starting point?>
> > > >
> > > Thanks in advance>
> > > >
> > > Curtis>
> > > >
> >
>
--
Un saludo.
Pedro Boado.