You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Buntu Dev <bu...@gmail.com> on 2015/08/15 01:56:06 UTC
Replicating MySQL table to HBase
I'm looking for ways to setup an incremental update task to replicate the
MySQL table to HBase. Sqoop seems like an option to import but doesn't seem
to handle row updates and deletes.
Hoping there might be other better ways to handle the
inserts/updates/deletes.
Thanks!
Re: Replicating MySQL table to HBase
Posted by Buntu Dev <bu...@gmail.com>.
Let me clarify that our analysis involves looking at the changes over a
period of time and HBase provides helps us with time based lookups and
state of the things at particular point of time.
On Fri, Aug 14, 2015 at 5:16 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> But why do you want that in HBase? Do you need random read and random write
> milliseconds access to those rows? Or you are going to run some analytics
> jobs which are going to do full scans anyway? If the later, why not just
> dumping that as files and doing the queries with a SQL engine?
>
> If you need HBase for user, then you might have to build a tool to migrate
> the data from MySQL to HBase because the MySQL schema will most probably
> not fit the HBase tables...
>
> 2015-08-14 20:12 GMT-04:00 Buntu Dev <bu...@gmail.com>:
>
> > We got a bunch of datasets in our CDH cluster and want to do some data
> > analysis on those which requires some meta data only available in MySQL.
> We
> > could do a one time Sqoop and then want to setup a job to capture the
> > changes and write to HBase. I'm looking for options to handle the MySQL
> > changes, thanks!
> >
> > On Fri, Aug 14, 2015 at 5:00 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > Hi,
> > >
> > > Before even going into that direction, why do you want to do that? It's
> > > most probably not a good idea.
> > >
> > > Is is for backup? For replication? etc.
> > >
> > > JM
> > >
> > > 2015-08-14 19:56 GMT-04:00 Buntu Dev <bu...@gmail.com>:
> > >
> > > > I'm looking for ways to setup an incremental update task to replicate
> > the
> > > > MySQL table to HBase. Sqoop seems like an option to import but
> doesn't
> > > seem
> > > > to handle row updates and deletes.
> > > >
> > > > Hoping there might be other better ways to handle the
> > > > inserts/updates/deletes.
> > > >
> > > > Thanks!
> > > >
> > >
> >
>
Re: Replicating MySQL table to HBase
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
But why do you want that in HBase? Do you need random read and random write
milliseconds access to those rows? Or you are going to run some analytics
jobs which are going to do full scans anyway? If the later, why not just
dumping that as files and doing the queries with a SQL engine?
If you need HBase for user, then you might have to build a tool to migrate
the data from MySQL to HBase because the MySQL schema will most probably
not fit the HBase tables...
2015-08-14 20:12 GMT-04:00 Buntu Dev <bu...@gmail.com>:
> We got a bunch of datasets in our CDH cluster and want to do some data
> analysis on those which requires some meta data only available in MySQL. We
> could do a one time Sqoop and then want to setup a job to capture the
> changes and write to HBase. I'm looking for options to handle the MySQL
> changes, thanks!
>
> On Fri, Aug 14, 2015 at 5:00 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Hi,
> >
> > Before even going into that direction, why do you want to do that? It's
> > most probably not a good idea.
> >
> > Is is for backup? For replication? etc.
> >
> > JM
> >
> > 2015-08-14 19:56 GMT-04:00 Buntu Dev <bu...@gmail.com>:
> >
> > > I'm looking for ways to setup an incremental update task to replicate
> the
> > > MySQL table to HBase. Sqoop seems like an option to import but doesn't
> > seem
> > > to handle row updates and deletes.
> > >
> > > Hoping there might be other better ways to handle the
> > > inserts/updates/deletes.
> > >
> > > Thanks!
> > >
> >
>
Re: Replicating MySQL table to HBase
Posted by Buntu Dev <bu...@gmail.com>.
We got a bunch of datasets in our CDH cluster and want to do some data
analysis on those which requires some meta data only available in MySQL. We
could do a one time Sqoop and then want to setup a job to capture the
changes and write to HBase. I'm looking for options to handle the MySQL
changes, thanks!
On Fri, Aug 14, 2015 at 5:00 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hi,
>
> Before even going into that direction, why do you want to do that? It's
> most probably not a good idea.
>
> Is is for backup? For replication? etc.
>
> JM
>
> 2015-08-14 19:56 GMT-04:00 Buntu Dev <bu...@gmail.com>:
>
> > I'm looking for ways to setup an incremental update task to replicate the
> > MySQL table to HBase. Sqoop seems like an option to import but doesn't
> seem
> > to handle row updates and deletes.
> >
> > Hoping there might be other better ways to handle the
> > inserts/updates/deletes.
> >
> > Thanks!
> >
>
Re: Replicating MySQL table to HBase
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi,
Before even going into that direction, why do you want to do that? It's
most probably not a good idea.
Is is for backup? For replication? etc.
JM
2015-08-14 19:56 GMT-04:00 Buntu Dev <bu...@gmail.com>:
> I'm looking for ways to setup an incremental update task to replicate the
> MySQL table to HBase. Sqoop seems like an option to import but doesn't seem
> to handle row updates and deletes.
>
> Hoping there might be other better ways to handle the
> inserts/updates/deletes.
>
> Thanks!
>