You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Enrico Olivelli <eo...@gmail.com> on 2016/12/30 13:05:39 UTC
A Java Client to connect to both 0.94 and 1.2.4 HBase
Hi all,
The migration from 0.94 to 1.2.4 is really a pain and some time ago I
asked to this list if there was a way to connect to both a 0.94
cluster and a 1.2.4 cluster.
I have created a proof-of-concept project which is able to achieve my
goal but "shading" most of the 0.94 client.
This is the source code and the list of tricks to make it work:
https://github.com/eolivelli/multi-hbase
The above code simply performs a full scan from a 0.94 table and
performs puts to a 1.2.4 cluster, but I think that most of the basic
API of the 0.94 client will work and the full API of the 1.2.4 is
available.
I'm going to implement a more stable version of the client to be used
in production.
I will appreciate a lot If anyone has some time to take a look and
give me some feedback.
Cheers
Enrico Olivelli
Re: A Java Client to connect to both 0.94 and 1.2.4 HBase
Posted by Enrico Olivelli <eo...@gmail.com>.
Ted,
2016-12-30 17:39 GMT+01:00 Ted Yu <yu...@gmail.com>:
> For hbase094_hbase_default.xml, currently it is
> under hbase094/src/main/resources
> Looks like it belongs to the test directory.
>
Yes because for now I have used a JUnit test in order to run the project.
I will write done some command-line tool in order to run the full migration
If you or other users are interested in such a tool you can write down some
use-case and I will be happy to keep it in consideration.
>
> bq. use the 'Configuration#set" method to programmatically change the
> values
>
Maybe I will let the user pass its own hbase-site.xml file, but instead of
loading directly I will load into memory and then call
HBaseConfiguration.set(key,value) for each entry.
The way HBaseConfiguration looks ho hbase-site.xml is somehow tricky
because it happens in the constructor and so even with subclassing it is
not possible to disable the 'version check'
> For end user, it would be more convenient to pass hbase-site.xml
>
> Cheers
>
> On Fri, Dec 30, 2016 at 8:19 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > 2016-12-30 17:11 GMT+01:00 Ted Yu <yu...@gmail.com>:
> >
> > > For scanHBase094Table():
> > >
> > > baseDefaults.addResource("hbase094_hbase_default.xml");
> > >
> > > where hbase.rootdir points to file://
> > >
> >
> > > How is the user supposed to plug in hbase-site.xml for the 0.94
> cluster ?
> > >
> >
> > the resource gets loaded from the classpath by the Hadoop Configuration
> > API.
> >
> > This is just a code sample to prove that a JVM can connect to two
> > incompatible versions of HBase.
> >
> > you can plug in your configuration by loading a "Configuration" built by
> > reading your configuration file.
> > Personally I do not use the .xml files for configuration of clients but
> use
> > the 'Configuration#set" method to programmatically change the values
> >
> >
> > >
> > > For TableMigrationManager, I don't see where setBatchSize() is called.
> > Does
> > > this mean that batch size of 0 would be in effect ?
> > >
> >
> > It is a bug, so the API uses its default values. See the HBase API docs
> >
> >
> > >
> > > Do you use any API specific to 1.2.4 ? Otherwise I suggest renaming
> > > hbase124
> > > as hbase1 since 1.x releases are compatible.
> > >
> >
> > +1
> > The only API that I use are the same as for 0.94 and 1.2
> > (scan,get,put,delete.....)
> >
> >
> > > Thanks
> > >
> > >
> > >
> > > On Fri, Dec 30, 2016 at 6:33 AM, Enrico Olivelli <eo...@gmail.com>
> > > wrote:
> > >
> > > > Thanks you Duo Zhang for the interest.
> > > >
> > > > The example comes from a Java class which actually "migrates" data
> > > > from one table in a 0.94 cluster to a 1.2.4 cluster
> > > >
> > > > this is the file:
> > > > https://github.com/eolivelli/multi-hbase/blob/master/multi-
> > > > hbase-client/src/test/java/MigrateHBaseDataExample.java
> > > >
> > > > this can be the foundation for a future migration tool but at this
> > > > stage that project is only a proof-of-concept
> > > >
> > > >
> > > > 2016-12-30 14:22 GMT+01:00 张铎(Duo Zhang) <pa...@gmail.com>:
> > > > > Nice try!
> > > > >
> > > > > Is this a migration tool? Or a thin layer on top of the original
> > hbase
> > > > > client to make the user be able to access both 0.94 and 1.2.4 with
> > the
> > > > same
> > > > > API?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2016-12-30 21:05 GMT+08:00 Enrico Olivelli <eo...@gmail.com>:
> > > > >
> > > > >> Hi all,
> > > > >> The migration from 0.94 to 1.2.4 is really a pain and some time
> > ago I
> > > > >> asked to this list if there was a way to connect to both a 0.94
> > > > >> cluster and a 1.2.4 cluster.
> > > > >>
> > > > >> I have created a proof-of-concept project which is able to achieve
> > my
> > > > >> goal but "shading" most of the 0.94 client.
> > > > >>
> > > > >> This is the source code and the list of tricks to make it work:
> > > > >> https://github.com/eolivelli/multi-hbase
> > > > >>
> > > > >> The above code simply performs a full scan from a 0.94 table and
> > > > >> performs puts to a 1.2.4 cluster, but I think that most of the
> basic
> > > > >> API of the 0.94 client will work and the full API of the 1.2.4 is
> > > > >> available.
> > > > >>
> > > > >> I'm going to implement a more stable version of the client to be
> > used
> > > > >> in production.
> > > > >>
> > > > >> I will appreciate a lot If anyone has some time to take a look and
> > > > >> give me some feedback.
> > > > >>
> > > > >> Cheers
> > > > >> Enrico Olivelli
> > > > >>
> > > >
> > >
> >
>
Re: A Java Client to connect to both 0.94 and 1.2.4 HBase
Posted by Ted Yu <yu...@gmail.com>.
For hbase094_hbase_default.xml, currently it is
under hbase094/src/main/resources
Looks like it belongs to the test directory.
bq. use the 'Configuration#set" method to programmatically change the values
For end user, it would be more convenient to pass hbase-site.xml
Cheers
On Fri, Dec 30, 2016 at 8:19 AM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Hi Ted,
>
> 2016-12-30 17:11 GMT+01:00 Ted Yu <yu...@gmail.com>:
>
> > For scanHBase094Table():
> >
> > baseDefaults.addResource("hbase094_hbase_default.xml");
> >
> > where hbase.rootdir points to file://
> >
>
> > How is the user supposed to plug in hbase-site.xml for the 0.94 cluster ?
> >
>
> the resource gets loaded from the classpath by the Hadoop Configuration
> API.
>
> This is just a code sample to prove that a JVM can connect to two
> incompatible versions of HBase.
>
> you can plug in your configuration by loading a "Configuration" built by
> reading your configuration file.
> Personally I do not use the .xml files for configuration of clients but use
> the 'Configuration#set" method to programmatically change the values
>
>
> >
> > For TableMigrationManager, I don't see where setBatchSize() is called.
> Does
> > this mean that batch size of 0 would be in effect ?
> >
>
> It is a bug, so the API uses its default values. See the HBase API docs
>
>
> >
> > Do you use any API specific to 1.2.4 ? Otherwise I suggest renaming
> > hbase124
> > as hbase1 since 1.x releases are compatible.
> >
>
> +1
> The only API that I use are the same as for 0.94 and 1.2
> (scan,get,put,delete.....)
>
>
> > Thanks
> >
> >
> >
> > On Fri, Dec 30, 2016 at 6:33 AM, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Thanks you Duo Zhang for the interest.
> > >
> > > The example comes from a Java class which actually "migrates" data
> > > from one table in a 0.94 cluster to a 1.2.4 cluster
> > >
> > > this is the file:
> > > https://github.com/eolivelli/multi-hbase/blob/master/multi-
> > > hbase-client/src/test/java/MigrateHBaseDataExample.java
> > >
> > > this can be the foundation for a future migration tool but at this
> > > stage that project is only a proof-of-concept
> > >
> > >
> > > 2016-12-30 14:22 GMT+01:00 张铎(Duo Zhang) <pa...@gmail.com>:
> > > > Nice try!
> > > >
> > > > Is this a migration tool? Or a thin layer on top of the original
> hbase
> > > > client to make the user be able to access both 0.94 and 1.2.4 with
> the
> > > same
> > > > API?
> > > >
> > > > Thanks.
> > > >
> > > > 2016-12-30 21:05 GMT+08:00 Enrico Olivelli <eo...@gmail.com>:
> > > >
> > > >> Hi all,
> > > >> The migration from 0.94 to 1.2.4 is really a pain and some time
> ago I
> > > >> asked to this list if there was a way to connect to both a 0.94
> > > >> cluster and a 1.2.4 cluster.
> > > >>
> > > >> I have created a proof-of-concept project which is able to achieve
> my
> > > >> goal but "shading" most of the 0.94 client.
> > > >>
> > > >> This is the source code and the list of tricks to make it work:
> > > >> https://github.com/eolivelli/multi-hbase
> > > >>
> > > >> The above code simply performs a full scan from a 0.94 table and
> > > >> performs puts to a 1.2.4 cluster, but I think that most of the basic
> > > >> API of the 0.94 client will work and the full API of the 1.2.4 is
> > > >> available.
> > > >>
> > > >> I'm going to implement a more stable version of the client to be
> used
> > > >> in production.
> > > >>
> > > >> I will appreciate a lot If anyone has some time to take a look and
> > > >> give me some feedback.
> > > >>
> > > >> Cheers
> > > >> Enrico Olivelli
> > > >>
> > >
> >
>
Re: A Java Client to connect to both 0.94 and 1.2.4 HBase
Posted by Enrico Olivelli <eo...@gmail.com>.
Hi Ted,
2016-12-30 17:11 GMT+01:00 Ted Yu <yu...@gmail.com>:
> For scanHBase094Table():
>
> baseDefaults.addResource("hbase094_hbase_default.xml");
>
> where hbase.rootdir points to file://
>
> How is the user supposed to plug in hbase-site.xml for the 0.94 cluster ?
>
the resource gets loaded from the classpath by the Hadoop Configuration API.
This is just a code sample to prove that a JVM can connect to two
incompatible versions of HBase.
you can plug in your configuration by loading a "Configuration" built by
reading your configuration file.
Personally I do not use the .xml files for configuration of clients but use
the 'Configuration#set" method to programmatically change the values
>
> For TableMigrationManager, I don't see where setBatchSize() is called. Does
> this mean that batch size of 0 would be in effect ?
>
It is a bug, so the API uses its default values. See the HBase API docs
>
> Do you use any API specific to 1.2.4 ? Otherwise I suggest renaming
> hbase124
> as hbase1 since 1.x releases are compatible.
>
+1
The only API that I use are the same as for 0.94 and 1.2
(scan,get,put,delete.....)
> Thanks
>
>
>
> On Fri, Dec 30, 2016 at 6:33 AM, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Thanks you Duo Zhang for the interest.
> >
> > The example comes from a Java class which actually "migrates" data
> > from one table in a 0.94 cluster to a 1.2.4 cluster
> >
> > this is the file:
> > https://github.com/eolivelli/multi-hbase/blob/master/multi-
> > hbase-client/src/test/java/MigrateHBaseDataExample.java
> >
> > this can be the foundation for a future migration tool but at this
> > stage that project is only a proof-of-concept
> >
> >
> > 2016-12-30 14:22 GMT+01:00 张铎(Duo Zhang) <pa...@gmail.com>:
> > > Nice try!
> > >
> > > Is this a migration tool? Or a thin layer on top of the original hbase
> > > client to make the user be able to access both 0.94 and 1.2.4 with the
> > same
> > > API?
> > >
> > > Thanks.
> > >
> > > 2016-12-30 21:05 GMT+08:00 Enrico Olivelli <eo...@gmail.com>:
> > >
> > >> Hi all,
> > >> The migration from 0.94 to 1.2.4 is really a pain and some time ago I
> > >> asked to this list if there was a way to connect to both a 0.94
> > >> cluster and a 1.2.4 cluster.
> > >>
> > >> I have created a proof-of-concept project which is able to achieve my
> > >> goal but "shading" most of the 0.94 client.
> > >>
> > >> This is the source code and the list of tricks to make it work:
> > >> https://github.com/eolivelli/multi-hbase
> > >>
> > >> The above code simply performs a full scan from a 0.94 table and
> > >> performs puts to a 1.2.4 cluster, but I think that most of the basic
> > >> API of the 0.94 client will work and the full API of the 1.2.4 is
> > >> available.
> > >>
> > >> I'm going to implement a more stable version of the client to be used
> > >> in production.
> > >>
> > >> I will appreciate a lot If anyone has some time to take a look and
> > >> give me some feedback.
> > >>
> > >> Cheers
> > >> Enrico Olivelli
> > >>
> >
>
Re: A Java Client to connect to both 0.94 and 1.2.4 HBase
Posted by Ted Yu <yu...@gmail.com>.
For scanHBase094Table():
baseDefaults.addResource("hbase094_hbase_default.xml");
where hbase.rootdir points to file://
How is the user supposed to plug in hbase-site.xml for the 0.94 cluster ?
For TableMigrationManager, I don't see where setBatchSize() is called. Does
this mean that batch size of 0 would be in effect ?
Do you use any API specific to 1.2.4 ? Otherwise I suggest renaming hbase124
as hbase1 since 1.x releases are compatible.
Thanks
On Fri, Dec 30, 2016 at 6:33 AM, Enrico Olivelli <eo...@gmail.com>
wrote:
> Thanks you Duo Zhang for the interest.
>
> The example comes from a Java class which actually "migrates" data
> from one table in a 0.94 cluster to a 1.2.4 cluster
>
> this is the file:
> https://github.com/eolivelli/multi-hbase/blob/master/multi-
> hbase-client/src/test/java/MigrateHBaseDataExample.java
>
> this can be the foundation for a future migration tool but at this
> stage that project is only a proof-of-concept
>
>
> 2016-12-30 14:22 GMT+01:00 张铎(Duo Zhang) <pa...@gmail.com>:
> > Nice try!
> >
> > Is this a migration tool? Or a thin layer on top of the original hbase
> > client to make the user be able to access both 0.94 and 1.2.4 with the
> same
> > API?
> >
> > Thanks.
> >
> > 2016-12-30 21:05 GMT+08:00 Enrico Olivelli <eo...@gmail.com>:
> >
> >> Hi all,
> >> The migration from 0.94 to 1.2.4 is really a pain and some time ago I
> >> asked to this list if there was a way to connect to both a 0.94
> >> cluster and a 1.2.4 cluster.
> >>
> >> I have created a proof-of-concept project which is able to achieve my
> >> goal but "shading" most of the 0.94 client.
> >>
> >> This is the source code and the list of tricks to make it work:
> >> https://github.com/eolivelli/multi-hbase
> >>
> >> The above code simply performs a full scan from a 0.94 table and
> >> performs puts to a 1.2.4 cluster, but I think that most of the basic
> >> API of the 0.94 client will work and the full API of the 1.2.4 is
> >> available.
> >>
> >> I'm going to implement a more stable version of the client to be used
> >> in production.
> >>
> >> I will appreciate a lot If anyone has some time to take a look and
> >> give me some feedback.
> >>
> >> Cheers
> >> Enrico Olivelli
> >>
>
Re: A Java Client to connect to both 0.94 and 1.2.4 HBase
Posted by Enrico Olivelli <eo...@gmail.com>.
Thanks you Duo Zhang for the interest.
The example comes from a Java class which actually "migrates" data
from one table in a 0.94 cluster to a 1.2.4 cluster
this is the file:
https://github.com/eolivelli/multi-hbase/blob/master/multi-hbase-client/src/test/java/MigrateHBaseDataExample.java
this can be the foundation for a future migration tool but at this
stage that project is only a proof-of-concept
2016-12-30 14:22 GMT+01:00 张铎(Duo Zhang) <pa...@gmail.com>:
> Nice try!
>
> Is this a migration tool? Or a thin layer on top of the original hbase
> client to make the user be able to access both 0.94 and 1.2.4 with the same
> API?
>
> Thanks.
>
> 2016-12-30 21:05 GMT+08:00 Enrico Olivelli <eo...@gmail.com>:
>
>> Hi all,
>> The migration from 0.94 to 1.2.4 is really a pain and some time ago I
>> asked to this list if there was a way to connect to both a 0.94
>> cluster and a 1.2.4 cluster.
>>
>> I have created a proof-of-concept project which is able to achieve my
>> goal but "shading" most of the 0.94 client.
>>
>> This is the source code and the list of tricks to make it work:
>> https://github.com/eolivelli/multi-hbase
>>
>> The above code simply performs a full scan from a 0.94 table and
>> performs puts to a 1.2.4 cluster, but I think that most of the basic
>> API of the 0.94 client will work and the full API of the 1.2.4 is
>> available.
>>
>> I'm going to implement a more stable version of the client to be used
>> in production.
>>
>> I will appreciate a lot If anyone has some time to take a look and
>> give me some feedback.
>>
>> Cheers
>> Enrico Olivelli
>>
Re: A Java Client to connect to both 0.94 and 1.2.4 HBase
Posted by "张铎 (Duo Zhang)" <pa...@gmail.com>.
Nice try!
Is this a migration tool? Or a thin layer on top of the original hbase
client to make the user be able to access both 0.94 and 1.2.4 with the same
API?
Thanks.
2016-12-30 21:05 GMT+08:00 Enrico Olivelli <eo...@gmail.com>:
> Hi all,
> The migration from 0.94 to 1.2.4 is really a pain and some time ago I
> asked to this list if there was a way to connect to both a 0.94
> cluster and a 1.2.4 cluster.
>
> I have created a proof-of-concept project which is able to achieve my
> goal but "shading" most of the 0.94 client.
>
> This is the source code and the list of tricks to make it work:
> https://github.com/eolivelli/multi-hbase
>
> The above code simply performs a full scan from a 0.94 table and
> performs puts to a 1.2.4 cluster, but I think that most of the basic
> API of the 0.94 client will work and the full API of the 1.2.4 is
> available.
>
> I'm going to implement a more stable version of the client to be used
> in production.
>
> I will appreciate a lot If anyone has some time to take a look and
> give me some feedback.
>
> Cheers
> Enrico Olivelli
>