You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jeetendra Gangele <ga...@gmail.com> on 2015/07/27 09:39:16 UTC

Migration of data from POSTSQL

I have a production data in PSQL and i want ti migrant the data to Hbase.
Also if there are any changes in my PSQL data , I wanted to update the
Hbase.

Since I am migrating from production, i don't wanted to hit too many
request to my server, also this Hbase data should always be sync with PSQL
data.

Scoop won't work here any other alternatives?

Re: Migration of data from POSTSQL

Posted by Jeetendra Gangele <ga...@gmail.com>.
Thanks for reply

I have a use case where where I am consuming the Events from RabbitMQ using
spark streaming.This event has some fields on which I want to query the
PostgreSQL and bring the data and then do the join between event data and
PostgreSQl data and put the aggregated data into HDFS, so that I run run
analytics query over this data using SparkSQL.

my question is PostgreSQL data in production data so i don't want to hit so
many times.

at any given  1 seconds time I may have 3000 events,that means I need to
fire 3000 parallel query to my PostGreSQl and this data keeps on growing,
so my database will go down.

I can't migrate this PostgreSQL data since lots of system using it,but I
can take this data to some NOSQL like base and query the Hbase, but here
issue is How can I make sure that Hbase has upto date data?

Any anyone suggest me best approach/ method to handle this case?

On 27 July 2015 at 23:16, Esteban Gutierrez <es...@cloudera.com> wrote:

> Hi Jeetendra,
>
> Unless you are considering to use Phoenix (https://phoenix.apache.org) or
> Trafodion (https://wiki.trafodion.org/wiki/index.php/Main_Page) for data
> access to HBase there is no single path to migrate a production application
> from PostgreSQL to HBase, also you will have to validate all SQL statements
> you use or generate in your application since there might be differences
> across SQL implementations (the same you would have to do if migration to
> another RDBMS). Replicating the data from PostgreSQL to HBase shouldn't be
> that hard if your client implements hooks and you could write on both
> platforms at the time, if thats not possible you might need to plug into
> the PostgreSQL WAL infrastructure to replicate the edits in real time to
> HBase. (I think FB or others have done something similar from MySQL to
> HBase).
>
> cheers,
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Mon, Jul 27, 2015 at 8:57 AM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
> > Yes.
> >
> > On 27 July 2015 at 21:02, Ted Yu <yu...@gmail.com> wrote:
> >
> > > By PSQL did you mean PostgreSQL ?
> > >
> > > Cheers
> > >
> > > On Mon, Jul 27, 2015 at 12:39 AM, Jeetendra Gangele <
> > gangele397@gmail.com>
> > > wrote:
> > >
> > > > I have a production data in PSQL and i want ti migrant the data to
> > Hbase.
> > > > Also if there are any changes in my PSQL data , I wanted to update
> the
> > > > Hbase.
> > > >
> > > > Since I am migrating from production, i don't wanted to hit too many
> > > > request to my server, also this Hbase data should always be sync with
> > > PSQL
> > > > data.
> > > >
> > > > Scoop won't work here any other alternatives?
> > > >
> > >
> >
>

Re: Migration of data from POSTSQL

Posted by Esteban Gutierrez <es...@cloudera.com>.
Hi Jeetendra,

Unless you are considering to use Phoenix (https://phoenix.apache.org) or
Trafodion (https://wiki.trafodion.org/wiki/index.php/Main_Page) for data
access to HBase there is no single path to migrate a production application
from PostgreSQL to HBase, also you will have to validate all SQL statements
you use or generate in your application since there might be differences
across SQL implementations (the same you would have to do if migration to
another RDBMS). Replicating the data from PostgreSQL to HBase shouldn't be
that hard if your client implements hooks and you could write on both
platforms at the time, if thats not possible you might need to plug into
the PostgreSQL WAL infrastructure to replicate the edits in real time to
HBase. (I think FB or others have done something similar from MySQL to
HBase).

cheers,
esteban.

--
Cloudera, Inc.


On Mon, Jul 27, 2015 at 8:57 AM, Jeetendra Gangele <ga...@gmail.com>
wrote:

> Yes.
>
> On 27 July 2015 at 21:02, Ted Yu <yu...@gmail.com> wrote:
>
> > By PSQL did you mean PostgreSQL ?
> >
> > Cheers
> >
> > On Mon, Jul 27, 2015 at 12:39 AM, Jeetendra Gangele <
> gangele397@gmail.com>
> > wrote:
> >
> > > I have a production data in PSQL and i want ti migrant the data to
> Hbase.
> > > Also if there are any changes in my PSQL data , I wanted to update the
> > > Hbase.
> > >
> > > Since I am migrating from production, i don't wanted to hit too many
> > > request to my server, also this Hbase data should always be sync with
> > PSQL
> > > data.
> > >
> > > Scoop won't work here any other alternatives?
> > >
> >
>

Re: Migration of data from POSTSQL

Posted by Jeetendra Gangele <ga...@gmail.com>.
Yes.

On 27 July 2015 at 21:02, Ted Yu <yu...@gmail.com> wrote:

> By PSQL did you mean PostgreSQL ?
>
> Cheers
>
> On Mon, Jul 27, 2015 at 12:39 AM, Jeetendra Gangele <ga...@gmail.com>
> wrote:
>
> > I have a production data in PSQL and i want ti migrant the data to Hbase.
> > Also if there are any changes in my PSQL data , I wanted to update the
> > Hbase.
> >
> > Since I am migrating from production, i don't wanted to hit too many
> > request to my server, also this Hbase data should always be sync with
> PSQL
> > data.
> >
> > Scoop won't work here any other alternatives?
> >
>

Re: Migration of data from POSTSQL

Posted by Ted Yu <yu...@gmail.com>.
By PSQL did you mean PostgreSQL ?

Cheers

On Mon, Jul 27, 2015 at 12:39 AM, Jeetendra Gangele <ga...@gmail.com>
wrote:

> I have a production data in PSQL and i want ti migrant the data to Hbase.
> Also if there are any changes in my PSQL data , I wanted to update the
> Hbase.
>
> Since I am migrating from production, i don't wanted to hit too many
> request to my server, also this Hbase data should always be sync with PSQL
> data.
>
> Scoop won't work here any other alternatives?
>