You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Mehmet Simsek <nu...@gmail.com> on 2012/12/13 16:46:32 UTC
Bulk Loading from Oracle to Hbase
Hi,I want to load 5 billion rows from oracle table to hbase table. Which technique is the best for this bulk loading?
Thanks
M.S
Re: Bulk Loading from Oracle to Hbase
Posted by anil gupta <an...@gmail.com>.
Hi Mehmet,
My suggestion would be:
1. Use Sqoop to copy the data from Oracle into HDFS.
2. Use HBase Bulk Loader to load the data into HBase Table.
Sqoop implementation of loading data directly into HBase uses HBase Api
which is not as efficient as HBase Bulk Loader. Also, features provided by
Sqoop for loading data into HBase are limited.
HTH,
Anil Gupta
On Thu, Dec 13, 2012 at 8:48 AM, Amandeep Khurana <am...@gmail.com> wrote:
> Mehmet
>
> What's the problem you are getting while running the Sqoop job? Can
> you give details?
>
> -Amandeep
>
> On Thu, Dec 13, 2012 at 8:44 AM, Manoj Babu <ma...@gmail.com> wrote:
> > Mehmet,
> >
> > You can try to write a MapReduce using DBInputFormat and insert into
> HBase.
> >
> http://hadoop.apache.org/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html
> >
> >
> > Cheers!
> > Manoj.
> >
> >
> >
> > On Thu, Dec 13, 2012 at 9:27 PM, Mehmet Simsek <nurettinsimsek@gmail.com
> >wrote:
> >
> >> Thanks to answer Mohammad.I tried sqoop but there are biliions rows.
> >> Select query processing too long.namenode does not get respond from
> daanode
> >> in 600sec. And throws exception. I can increase this duration but select
> >> query running too long may be 6 hours. Is there a another good
> solution? Or
> >> another technique in sqoop?
> >>
> >> M.S
> >>
> >> On 13 Ara 2012, at 17:48, Mohammad Tariq <do...@gmail.com> wrote:
> >>
> >> > Hello Mehmet,
> >> >
> >> > Have a look at Acache Sqoop <http://sqoop.apache.org/>
> >> >
> >> > Regards,
> >> > Mohammad Tariq
> >> >
> >> >
> >> >
> >> > On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <
> nurettinsimsek@gmail.com
> >> >wrote:
> >> >
> >> >> Hi,I want to load 5 billion rows from oracle table to hbase table.
> Which
> >> >> technique is the best for this bulk loading?
> >> >>
> >> >> Thanks
> >> >>
> >> >> M.S
> >> >>
> >> >>
> >>
>
--
Thanks & Regards,
Anil Gupta
Re: Bulk Loading from Oracle to Hbase
Posted by Amandeep Khurana <am...@gmail.com>.
Mehmet
What's the problem you are getting while running the Sqoop job? Can
you give details?
-Amandeep
On Thu, Dec 13, 2012 at 8:44 AM, Manoj Babu <ma...@gmail.com> wrote:
> Mehmet,
>
> You can try to write a MapReduce using DBInputFormat and insert into HBase.
> http://hadoop.apache.org/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html
>
>
> Cheers!
> Manoj.
>
>
>
> On Thu, Dec 13, 2012 at 9:27 PM, Mehmet Simsek <nu...@gmail.com>wrote:
>
>> Thanks to answer Mohammad.I tried sqoop but there are biliions rows.
>> Select query processing too long.namenode does not get respond from daanode
>> in 600sec. And throws exception. I can increase this duration but select
>> query running too long may be 6 hours. Is there a another good solution? Or
>> another technique in sqoop?
>>
>> M.S
>>
>> On 13 Ara 2012, at 17:48, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> > Hello Mehmet,
>> >
>> > Have a look at Acache Sqoop <http://sqoop.apache.org/>
>> >
>> > Regards,
>> > Mohammad Tariq
>> >
>> >
>> >
>> > On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <nurettinsimsek@gmail.com
>> >wrote:
>> >
>> >> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
>> >> technique is the best for this bulk loading?
>> >>
>> >> Thanks
>> >>
>> >> M.S
>> >>
>> >>
>>
Re: Bulk Loading from Oracle to Hbase
Posted by Manoj Babu <ma...@gmail.com>.
Mehmet,
You can try to write a MapReduce using DBInputFormat and insert into HBase.
http://hadoop.apache.org/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/db/DBInputFormat.html
Cheers!
Manoj.
On Thu, Dec 13, 2012 at 9:27 PM, Mehmet Simsek <nu...@gmail.com>wrote:
> Thanks to answer Mohammad.I tried sqoop but there are biliions rows.
> Select query processing too long.namenode does not get respond from daanode
> in 600sec. And throws exception. I can increase this duration but select
> query running too long may be 6 hours. Is there a another good solution? Or
> another technique in sqoop?
>
> M.S
>
> On 13 Ara 2012, at 17:48, Mohammad Tariq <do...@gmail.com> wrote:
>
> > Hello Mehmet,
> >
> > Have a look at Acache Sqoop <http://sqoop.apache.org/>
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <nurettinsimsek@gmail.com
> >wrote:
> >
> >> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> >> technique is the best for this bulk loading?
> >>
> >> Thanks
> >>
> >> M.S
> >>
> >>
>
Re: Bulk Loading from Oracle to Hbase
Posted by Mehmet Simsek <nu...@gmail.com>.
Thanks to answer Mohammad.I tried sqoop but there are biliions rows. Select query processing too long.namenode does not get respond from daanode in 600sec. And throws exception. I can increase this duration but select query running too long may be 6 hours. Is there a another good solution? Or another technique in sqoop?
M.S
On 13 Ara 2012, at 17:48, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Mehmet,
>
> Have a look at Acache Sqoop <http://sqoop.apache.org/>
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <nu...@gmail.com>wrote:
>
>> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
>> technique is the best for this bulk loading?
>>
>> Thanks
>>
>> M.S
>>
>>
Re: Bulk Loading from Oracle to Hbase
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Mehmet,
Have a look at Acache Sqoop <http://sqoop.apache.org/>
Regards,
Mohammad Tariq
On Thu, Dec 13, 2012 at 9:16 PM, Mehmet Simsek <nu...@gmail.com>wrote:
> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> technique is the best for this bulk loading?
>
> Thanks
>
> M.S
>
>
Re: Bulk Loading from Oracle to Hbase
Posted by Azuryy Yu <az...@gmail.com>.
Then, I suggest:
1) Increase your session per user in RAC equal or greater than your map
tasks
2) I suppose you want 100 map tasks, then write a little java code to
generate 100 files, each file has only one line, you can write any in this
line.
3) Define a STEP for all maps. then each map task select STEP records from
DB.
4) Put all your files to the HDFS
5) Create Map(no reduce here), execute "select * from Oracle where rownum
>= lineno and rownum < maptask_id * STEP" in each map task.
On Fri, Dec 14, 2012 at 11:36 AM, Mehmet Simsek <nu...@gmail.com>wrote:
> Hello Azuryy,
>
> Oracle in RAC.because of too long running sql statement datanode cannot
> respond to namenode about statistics.because of that exception is thrown by
> namenode like as "cannot respond from data node in 600sec..." In sqoop tool.
>
> Is there an another solution about this problem?
>
> M.S.
>
> On 14 Ara 2012, at 05:11, Azuryy Yu <az...@gmail.com> wrote:
>
> > Hello Mehmet,
> >
> > what about your oracle? a single node or RAC ?
> >
> >
> > On Thu, Dec 13, 2012 at 11:46 PM, Mehmet Simsek <
> nurettinsimsek@gmail.com>wrote:
> >
> >> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> >> technique is the best for this bulk loading?
> >>
> >> Thanks
> >>
> >> M.S
> >>
> >>
>
Re: Bulk Loading from Oracle to Hbase
Posted by Mehmet Simsek <nu...@gmail.com>.
Hello Azuryy,
Oracle in RAC.because of too long running sql statement datanode cannot respond to namenode about statistics.because of that exception is thrown by namenode like as "cannot respond from data node in 600sec..." In sqoop tool.
Is there an another solution about this problem?
M.S.
On 14 Ara 2012, at 05:11, Azuryy Yu <az...@gmail.com> wrote:
> Hello Mehmet,
>
> what about your oracle? a single node or RAC ?
>
>
> On Thu, Dec 13, 2012 at 11:46 PM, Mehmet Simsek <nu...@gmail.com>wrote:
>
>> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
>> technique is the best for this bulk loading?
>>
>> Thanks
>>
>> M.S
>>
>>
Re: Bulk Loading from Oracle to Hbase
Posted by Manoj Babu <ma...@gmail.com>.
can have look on this feature of hdfs.push from oracle R Connector.
http://docs.oracle.com/cd/E27101_01/doc.10/e27365/orhc.htm#CFHHEFHA
Cheers!
Manoj.
On Fri, Dec 14, 2012 at 8:41 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hello Mehmet,
>
> what about your oracle? a single node or RAC ?
>
>
> On Thu, Dec 13, 2012 at 11:46 PM, Mehmet Simsek <nurettinsimsek@gmail.com
> >wrote:
>
> > Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> > technique is the best for this bulk loading?
> >
> > Thanks
> >
> > M.S
> >
> >
>
Re: Bulk Loading from Oracle to Hbase
Posted by Azuryy Yu <az...@gmail.com>.
Hello Mehmet,
what about your oracle? a single node or RAC ?
On Thu, Dec 13, 2012 at 11:46 PM, Mehmet Simsek <nu...@gmail.com>wrote:
> Hi,I want to load 5 billion rows from oracle table to hbase table. Which
> technique is the best for this bulk loading?
>
> Thanks
>
> M.S
>
>