You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sever Fundatureanu <fu...@gmail.com> on 2012/07/02 19:30:09 UTC
Re: HBase bulk load through co-processors
Can someone please help me with this?
Thanks,
Sever
On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
fundatureanu.sever@gmail.com> wrote:
> My keys are built of 4 8-byte Ids. I am currently doing the load with MR
> but I get a timeout when doing the loadIncrementalFiles call:
>
> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> unrecoverable error from region server
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=10, exceptions:
> Sun Jun 24 21:29:01 CEST 2012,
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9,
> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
> 10.141.0.79:60020
> failed on socket timeout exception: java.net.SocketTimeoutException: 60000
> millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[co
> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
> 10.141.0.79:60020]
>
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> unrecoverable error from region server
>
> Is there a way in which I can increase the timeout period?
>
> Thank you,
>
> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell <ap...@apache.org>wrote:
>
>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
>> <fu...@gmail.com> wrote:
>> > I have to bulkload 6 tables which contain the same information but with
>> a
>> > different order to cover all possible access patterns. Would it be a
>> good
>> > idea to do only one load and use co-processors to populate the other
>> > tables, instead of doing the traditional MR bulkload which would
>> require 6
>> > separate jobs?
>>
>> Without knowing more than you've said, it seems better to use MR to
>> build all input.
>>
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein (via Tom White)
>>
>
>
>
> --
> Sever Fundatureanu
>
> Vrije Universiteit Amsterdam
> E-mail: fundatureanu.sever@gmail.com
>
--
Sever Fundatureanu
Vrije Universiteit Amsterdam
E-mail: fundatureanu.sever@gmail.com
Re: HBase bulk load through co-processors
Posted by Andrew Purtell <an...@gmail.com>.
Sever,
You and Jesse Yates should talk. See http://jyates.github.com/2012/07/09/consistent-enough-secondary-indexes.html
- Andy
On Jul 14, 2012, at 5:24 AM, Sever Fundatureanu <fu...@gmail.com> wrote:
> My intention is to implement a Secondary Index as suggested here:
> http://wiki.apache.org/hadoop/Hbase/SecondaryIndexing.
> It is advised here to add secondary index edits to a "shared work queue".
> And that "The shared queue would be a thread or threadpool that picks up
> these secondary table edit jobs and applies them using a normal Put
> operation to the secondary table". Is this shared queue some kind of
> mechanism external to all Region servers? Or a queue shared only between
> the threads local to one RS?
>
> Thanks,
> Sever
>
> On Mon, Jul 9, 2012 at 10:16 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
>> On Sat, Jul 7, 2012 at 2:58 AM, Sever Fundatureanu <
>> fundatureanu.sever@gmail.com> wrote:
>>
>>> Also does anybody know what is the flow in the system when a coprocessor
>>> from one RS make a Put call with a row key which falls on another RS?
>> I.e.
>>> do the Region servers communicate directly between each other?
>>>
>>
>> In this case, the coprocessor in your RS is acting like any other HBase
>> client. Puts will write from the coproc to the target RS like a normal
>> write. That is, of course, assuming I understand your implementation.
>>
>> -n
>>
>> On Fri, Jul 6, 2012 at 10:16 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>>>
>>>> Sever,
>>>>
>>>> I presume you're loading your data via online Puts via the MR job (as
>>>> opposed to generating HFiles). What are you hoping to gain from a
>>>> coprocessor implementation vs the 6 MR jobs? Have you pre-split your
>>>> tables? Can the RegionServer(s) handle all the concurrent mappers?
>>>>
>>>> -n
>>>>
>>>> On Mon, Jul 2, 2012 at 11:58 AM, Sever Fundatureanu <
>>>> fundatureanu.sever@gmail.com> wrote:
>>>>
>>>>> I agree that increasing the timeout is not the best option, I will
>> work
>>>>> both on better balancing the load and maybe doing it in increments
>> like
>>>> you
>>>>> suggested. However for now I want a quick fix to the problem.
>>>>>
>>>>> Just to see if I understand this right: a zookeeper node redirects my
>>>>> client to a region server node and then my client talk directly to
>> this
>>>>> region server; now the timeout happens on the client while talking to
>>> the
>>>>> RS right? It expects some kind of confirmation and it times out.. if
>>> this
>>>>> is the case how can I increase this timeout? I only found in the
>>>>> documentation "zookeeper.session.timeout" which is the timeout
>> between
>>>>> zookeeper and HBase.
>>>>>
>>>>> Thanks,
>>>>> Sever
>>>>>
>>>>> On Mon, Jul 2, 2012 at 8:19 PM, Jean-Marc Spaggiari <
>>>>> jean-marc@spaggiari.org
>>>>>> wrote:
>>>>>
>>>>>> Hi Sever,
>>>>>>
>>>>>> It seems one of the nodes in your cluster is overwhelmed with the
>>> load
>>>>>> you are giving him.
>>>>>>
>>>>>> So IMO, you have two options here:
>>>>>> First, you can try to reduce the load. I mean, split the bulk in
>>>>>> multiple smaller bulks and load them one by one to give the time to
>>>>>> your cluster to dispatch it correctly.
>>>>>> Second, you can inscreade the timeone from 60s to 120s. But you
>> might
>>>>>> face the same issue with 120s so I really recommand the fist
>> option.
>>>>>>
>>>>>> JM
>>>>>>
>>>>>> 2012/7/2, Sever Fundatureanu <fu...@gmail.com>:
>>>>>>> Can someone please help me with this?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sever
>>>>>>>
>>>>>>> On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
>>>>>>> fundatureanu.sever@gmail.com> wrote:
>>>>>>>
>>>>>>>> My keys are built of 4 8-byte Ids. I am currently doing the
>> load
>>>> with
>>>>>> MR
>>>>>>>> but I get a timeout when doing the loadIncrementalFiles call:
>>>>>>>>
>>>>>>>> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles:
>>> Encountered
>>>>>>>> unrecoverable error from region server
>>>>>>>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
>>>> after
>>>>>>>> attempts=10, exceptions:
>>>>>>>> Sun Jun 24 21:29:01 CEST 2012,
>>>>>>>>
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9
>>> ,
>>>>>>>> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
>>>>>>>> 10.141.0.79:60020
>>>>>>>> failed on socket timeout exception:
>>> java.net.SocketTimeoutException:
>>>>>>>> 60000
>>>>>>>> millis timeout while waiting for channel to be ready for read.
>> ch
>>> :
>>>>>>>> java.nio.channels.SocketChannel[co
>>>>>>>> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
>>>>>>>> 10.141.0.79:60020]
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
>>>>>>>> at
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
>>>>>>>> at
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
>>>>>>>> at
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
>>>>>>>> at
>>>>>>>>
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>> at
>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>> at
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles:
>>> Encountered
>>>>>>>> unrecoverable error from region server
>>>>>>>>
>>>>>>>> Is there a way in which I can increase the timeout period?
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell
>>>>>>>> <ap...@apache.org>wrote:
>>>>>>>>
>>>>>>>>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
>>>>>>>>> <fu...@gmail.com> wrote:
>>>>>>>>>> I have to bulkload 6 tables which contain the same
>> information
>>>> but
>>>>>>>>>> with
>>>>>>>>> a
>>>>>>>>>> different order to cover all possible access patterns. Would
>> it
>>>> be
>>>>> a
>>>>>>>>> good
>>>>>>>>>> idea to do only one load and use co-processors to populate
>> the
>>>>> other
>>>>>>>>>> tables, instead of doing the traditional MR bulkload which
>>> would
>>>>>>>>> require 6
>>>>>>>>>> separate jobs?
>>>>>>>>>
>>>>>>>>> Without knowing more than you've said, it seems better to use
>> MR
>>> to
>>>>>>>>> build all input.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>> - Andy
>>>>>>>>>
>>>>>>>>> Problems worthy of attack prove their worth by hitting back. -
>>> Piet
>>>>>>>>> Hein (via Tom White)
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sever Fundatureanu
>>>>>>>>
>>>>>>>> Vrije Universiteit Amsterdam
>>>>>>>> E-mail: fundatureanu.sever@gmail.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sever Fundatureanu
>>>>>>>
>>>>>>> Vrije Universiteit Amsterdam
>>>>>>> E-mail: fundatureanu.sever@gmail.com
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sever Fundatureanu
>>>>>
>>>>> Vrije Universiteit Amsterdam
>>>>> E-mail: fundatureanu.sever@gmail.com
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Sever Fundatureanu
>>>
>>> Vrije Universiteit Amsterdam
>>> E-mail: fundatureanu.sever@gmail.com
>>>
>>
>
>
>
> --
> Sever Fundatureanu
>
> Vrije Universiteit Amsterdam
> E-mail: fundatureanu.sever@gmail.com
Re: HBase bulk load through co-processors
Posted by Sever Fundatureanu <fu...@gmail.com>.
My intention is to implement a Secondary Index as suggested here:
http://wiki.apache.org/hadoop/Hbase/SecondaryIndexing.
It is advised here to add secondary index edits to a "shared work queue".
And that "The shared queue would be a thread or threadpool that picks up
these secondary table edit jobs and applies them using a normal Put
operation to the secondary table". Is this shared queue some kind of
mechanism external to all Region servers? Or a queue shared only between
the threads local to one RS?
Thanks,
Sever
On Mon, Jul 9, 2012 at 10:16 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> On Sat, Jul 7, 2012 at 2:58 AM, Sever Fundatureanu <
> fundatureanu.sever@gmail.com> wrote:
>
> > Also does anybody know what is the flow in the system when a coprocessor
> > from one RS make a Put call with a row key which falls on another RS?
> I.e.
> > do the Region servers communicate directly between each other?
> >
>
> In this case, the coprocessor in your RS is acting like any other HBase
> client. Puts will write from the coproc to the target RS like a normal
> write. That is, of course, assuming I understand your implementation.
>
> -n
>
> On Fri, Jul 6, 2012 at 10:16 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> >
> > > Sever,
> > >
> > > I presume you're loading your data via online Puts via the MR job (as
> > > opposed to generating HFiles). What are you hoping to gain from a
> > > coprocessor implementation vs the 6 MR jobs? Have you pre-split your
> > > tables? Can the RegionServer(s) handle all the concurrent mappers?
> > >
> > > -n
> > >
> > > On Mon, Jul 2, 2012 at 11:58 AM, Sever Fundatureanu <
> > > fundatureanu.sever@gmail.com> wrote:
> > >
> > > > I agree that increasing the timeout is not the best option, I will
> work
> > > > both on better balancing the load and maybe doing it in increments
> like
> > > you
> > > > suggested. However for now I want a quick fix to the problem.
> > > >
> > > > Just to see if I understand this right: a zookeeper node redirects my
> > > > client to a region server node and then my client talk directly to
> this
> > > > region server; now the timeout happens on the client while talking to
> > the
> > > > RS right? It expects some kind of confirmation and it times out.. if
> > this
> > > > is the case how can I increase this timeout? I only found in the
> > > > documentation "zookeeper.session.timeout" which is the timeout
> between
> > > > zookeeper and HBase.
> > > >
> > > > Thanks,
> > > > Sever
> > > >
> > > > On Mon, Jul 2, 2012 at 8:19 PM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org
> > > > > wrote:
> > > >
> > > > > Hi Sever,
> > > > >
> > > > > It seems one of the nodes in your cluster is overwhelmed with the
> > load
> > > > > you are giving him.
> > > > >
> > > > > So IMO, you have two options here:
> > > > > First, you can try to reduce the load. I mean, split the bulk in
> > > > > multiple smaller bulks and load them one by one to give the time to
> > > > > your cluster to dispatch it correctly.
> > > > > Second, you can inscreade the timeone from 60s to 120s. But you
> might
> > > > > face the same issue with 120s so I really recommand the fist
> option.
> > > > >
> > > > > JM
> > > > >
> > > > > 2012/7/2, Sever Fundatureanu <fu...@gmail.com>:
> > > > > > Can someone please help me with this?
> > > > > >
> > > > > > Thanks,
> > > > > > Sever
> > > > > >
> > > > > > On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
> > > > > > fundatureanu.sever@gmail.com> wrote:
> > > > > >
> > > > > >> My keys are built of 4 8-byte Ids. I am currently doing the
> load
> > > with
> > > > > MR
> > > > > >> but I get a timeout when doing the loadIncrementalFiles call:
> > > > > >>
> > > > > >> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles:
> > Encountered
> > > > > >> unrecoverable error from region server
> > > > > >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> > > after
> > > > > >> attempts=10, exceptions:
> > > > > >> Sun Jun 24 21:29:01 CEST 2012,
> > > > > >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9
> > ,
> > > > > >> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
> > > > > >> 10.141.0.79:60020
> > > > > >> failed on socket timeout exception:
> > java.net.SocketTimeoutException:
> > > > > >> 60000
> > > > > >> millis timeout while waiting for channel to be ready for read.
> ch
> > :
> > > > > >> java.nio.channels.SocketChannel[co
> > > > > >> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
> > > > > >> 10.141.0.79:60020]
> > > > > >>
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
> > > > > >> at
> > > > > >>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > > > >> at
> > java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > >> at
> > > > > >>
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > >> at java.lang.Thread.run(Thread.java:662)
> > > > > >> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles:
> > Encountered
> > > > > >> unrecoverable error from region server
> > > > > >>
> > > > > >> Is there a way in which I can increase the timeout period?
> > > > > >>
> > > > > >> Thank you,
> > > > > >>
> > > > > >> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell
> > > > > >> <ap...@apache.org>wrote:
> > > > > >>
> > > > > >>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
> > > > > >>> <fu...@gmail.com> wrote:
> > > > > >>> > I have to bulkload 6 tables which contain the same
> information
> > > but
> > > > > >>> > with
> > > > > >>> a
> > > > > >>> > different order to cover all possible access patterns. Would
> it
> > > be
> > > > a
> > > > > >>> good
> > > > > >>> > idea to do only one load and use co-processors to populate
> the
> > > > other
> > > > > >>> > tables, instead of doing the traditional MR bulkload which
> > would
> > > > > >>> require 6
> > > > > >>> > separate jobs?
> > > > > >>>
> > > > > >>> Without knowing more than you've said, it seems better to use
> MR
> > to
> > > > > >>> build all input.
> > > > > >>>
> > > > > >>> Best regards,
> > > > > >>>
> > > > > >>> - Andy
> > > > > >>>
> > > > > >>> Problems worthy of attack prove their worth by hitting back. -
> > Piet
> > > > > >>> Hein (via Tom White)
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Sever Fundatureanu
> > > > > >>
> > > > > >> Vrije Universiteit Amsterdam
> > > > > >> E-mail: fundatureanu.sever@gmail.com
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sever Fundatureanu
> > > > > >
> > > > > > Vrije Universiteit Amsterdam
> > > > > > E-mail: fundatureanu.sever@gmail.com
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sever Fundatureanu
> > > >
> > > > Vrije Universiteit Amsterdam
> > > > E-mail: fundatureanu.sever@gmail.com
> > > >
> > >
> >
> >
> >
> > --
> > Sever Fundatureanu
> >
> > Vrije Universiteit Amsterdam
> > E-mail: fundatureanu.sever@gmail.com
> >
>
--
Sever Fundatureanu
Vrije Universiteit Amsterdam
E-mail: fundatureanu.sever@gmail.com
Re: HBase bulk load through co-processors
Posted by Nick Dimiduk <nd...@gmail.com>.
On Sat, Jul 7, 2012 at 2:58 AM, Sever Fundatureanu <
fundatureanu.sever@gmail.com> wrote:
> Also does anybody know what is the flow in the system when a coprocessor
> from one RS make a Put call with a row key which falls on another RS? I.e.
> do the Region servers communicate directly between each other?
>
In this case, the coprocessor in your RS is acting like any other HBase
client. Puts will write from the coproc to the target RS like a normal
write. That is, of course, assuming I understand your implementation.
-n
On Fri, Jul 6, 2012 at 10:16 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Sever,
> >
> > I presume you're loading your data via online Puts via the MR job (as
> > opposed to generating HFiles). What are you hoping to gain from a
> > coprocessor implementation vs the 6 MR jobs? Have you pre-split your
> > tables? Can the RegionServer(s) handle all the concurrent mappers?
> >
> > -n
> >
> > On Mon, Jul 2, 2012 at 11:58 AM, Sever Fundatureanu <
> > fundatureanu.sever@gmail.com> wrote:
> >
> > > I agree that increasing the timeout is not the best option, I will work
> > > both on better balancing the load and maybe doing it in increments like
> > you
> > > suggested. However for now I want a quick fix to the problem.
> > >
> > > Just to see if I understand this right: a zookeeper node redirects my
> > > client to a region server node and then my client talk directly to this
> > > region server; now the timeout happens on the client while talking to
> the
> > > RS right? It expects some kind of confirmation and it times out.. if
> this
> > > is the case how can I increase this timeout? I only found in the
> > > documentation "zookeeper.session.timeout" which is the timeout between
> > > zookeeper and HBase.
> > >
> > > Thanks,
> > > Sever
> > >
> > > On Mon, Jul 2, 2012 at 8:19 PM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org
> > > > wrote:
> > >
> > > > Hi Sever,
> > > >
> > > > It seems one of the nodes in your cluster is overwhelmed with the
> load
> > > > you are giving him.
> > > >
> > > > So IMO, you have two options here:
> > > > First, you can try to reduce the load. I mean, split the bulk in
> > > > multiple smaller bulks and load them one by one to give the time to
> > > > your cluster to dispatch it correctly.
> > > > Second, you can inscreade the timeone from 60s to 120s. But you might
> > > > face the same issue with 120s so I really recommand the fist option.
> > > >
> > > > JM
> > > >
> > > > 2012/7/2, Sever Fundatureanu <fu...@gmail.com>:
> > > > > Can someone please help me with this?
> > > > >
> > > > > Thanks,
> > > > > Sever
> > > > >
> > > > > On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
> > > > > fundatureanu.sever@gmail.com> wrote:
> > > > >
> > > > >> My keys are built of 4 8-byte Ids. I am currently doing the load
> > with
> > > > MR
> > > > >> but I get a timeout when doing the loadIncrementalFiles call:
> > > > >>
> > > > >> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles:
> Encountered
> > > > >> unrecoverable error from region server
> > > > >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> > after
> > > > >> attempts=10, exceptions:
> > > > >> Sun Jun 24 21:29:01 CEST 2012,
> > > > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9
> ,
> > > > >> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
> > > > >> 10.141.0.79:60020
> > > > >> failed on socket timeout exception:
> java.net.SocketTimeoutException:
> > > > >> 60000
> > > > >> millis timeout while waiting for channel to be ready for read. ch
> :
> > > > >> java.nio.channels.SocketChannel[co
> > > > >> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
> > > > >> 10.141.0.79:60020]
> > > > >>
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
> > > > >> at
> > > > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > > >> at
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > >> at java.lang.Thread.run(Thread.java:662)
> > > > >> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles:
> Encountered
> > > > >> unrecoverable error from region server
> > > > >>
> > > > >> Is there a way in which I can increase the timeout period?
> > > > >>
> > > > >> Thank you,
> > > > >>
> > > > >> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell
> > > > >> <ap...@apache.org>wrote:
> > > > >>
> > > > >>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
> > > > >>> <fu...@gmail.com> wrote:
> > > > >>> > I have to bulkload 6 tables which contain the same information
> > but
> > > > >>> > with
> > > > >>> a
> > > > >>> > different order to cover all possible access patterns. Would it
> > be
> > > a
> > > > >>> good
> > > > >>> > idea to do only one load and use co-processors to populate the
> > > other
> > > > >>> > tables, instead of doing the traditional MR bulkload which
> would
> > > > >>> require 6
> > > > >>> > separate jobs?
> > > > >>>
> > > > >>> Without knowing more than you've said, it seems better to use MR
> to
> > > > >>> build all input.
> > > > >>>
> > > > >>> Best regards,
> > > > >>>
> > > > >>> - Andy
> > > > >>>
> > > > >>> Problems worthy of attack prove their worth by hitting back. -
> Piet
> > > > >>> Hein (via Tom White)
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Sever Fundatureanu
> > > > >>
> > > > >> Vrije Universiteit Amsterdam
> > > > >> E-mail: fundatureanu.sever@gmail.com
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sever Fundatureanu
> > > > >
> > > > > Vrije Universiteit Amsterdam
> > > > > E-mail: fundatureanu.sever@gmail.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sever Fundatureanu
> > >
> > > Vrije Universiteit Amsterdam
> > > E-mail: fundatureanu.sever@gmail.com
> > >
> >
>
>
>
> --
> Sever Fundatureanu
>
> Vrije Universiteit Amsterdam
> E-mail: fundatureanu.sever@gmail.com
>
Re: HBase bulk load through co-processors
Posted by Sever Fundatureanu <fu...@gmail.com>.
Yes Nick, I would like to do a comparison, performance-wise, between this
method of bulkloading through coprocessors from 1 MR job with online Puts
vs. having 6 MR jobs for generating HFiles. The tables are pre-split for
both versions.
For the coprocessors version my strategy is to buffer the Put operations
for the other tables in the co-processors and flush once every FLUSH_LIMIT
puts. I am currently working on making these buffers thread-safe, since the
co-processor functions can be called from multiple RS threads.
The region server handler count is currently set to the default of 10.
These nodes have Dual CPU dual-cores, so 4 hw threads. Of course, on the RS
nodes I also have running the Hadoop and HDFS daemons. Do you think I would
have problems with this setup under heavy loads?
Also does anybody know what is the flow in the system when a coprocessor
from one RS make a Put call with a row key which falls on another RS? I.e.
do the Region servers communicate directly between each other?
Thank you,
Sever
On Fri, Jul 6, 2012 at 10:16 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> Sever,
>
> I presume you're loading your data via online Puts via the MR job (as
> opposed to generating HFiles). What are you hoping to gain from a
> coprocessor implementation vs the 6 MR jobs? Have you pre-split your
> tables? Can the RegionServer(s) handle all the concurrent mappers?
>
> -n
>
> On Mon, Jul 2, 2012 at 11:58 AM, Sever Fundatureanu <
> fundatureanu.sever@gmail.com> wrote:
>
> > I agree that increasing the timeout is not the best option, I will work
> > both on better balancing the load and maybe doing it in increments like
> you
> > suggested. However for now I want a quick fix to the problem.
> >
> > Just to see if I understand this right: a zookeeper node redirects my
> > client to a region server node and then my client talk directly to this
> > region server; now the timeout happens on the client while talking to the
> > RS right? It expects some kind of confirmation and it times out.. if this
> > is the case how can I increase this timeout? I only found in the
> > documentation "zookeeper.session.timeout" which is the timeout between
> > zookeeper and HBase.
> >
> > Thanks,
> > Sever
> >
> > On Mon, Jul 2, 2012 at 8:19 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org
> > > wrote:
> >
> > > Hi Sever,
> > >
> > > It seems one of the nodes in your cluster is overwhelmed with the load
> > > you are giving him.
> > >
> > > So IMO, you have two options here:
> > > First, you can try to reduce the load. I mean, split the bulk in
> > > multiple smaller bulks and load them one by one to give the time to
> > > your cluster to dispatch it correctly.
> > > Second, you can inscreade the timeone from 60s to 120s. But you might
> > > face the same issue with 120s so I really recommand the fist option.
> > >
> > > JM
> > >
> > > 2012/7/2, Sever Fundatureanu <fu...@gmail.com>:
> > > > Can someone please help me with this?
> > > >
> > > > Thanks,
> > > > Sever
> > > >
> > > > On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
> > > > fundatureanu.sever@gmail.com> wrote:
> > > >
> > > >> My keys are built of 4 8-byte Ids. I am currently doing the load
> with
> > > MR
> > > >> but I get a timeout when doing the loadIncrementalFiles call:
> > > >>
> > > >> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> > > >> unrecoverable error from region server
> > > >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after
> > > >> attempts=10, exceptions:
> > > >> Sun Jun 24 21:29:01 CEST 2012,
> > > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9,
> > > >> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
> > > >> 10.141.0.79:60020
> > > >> failed on socket timeout exception: java.net.SocketTimeoutException:
> > > >> 60000
> > > >> millis timeout while waiting for channel to be ready for read. ch :
> > > >> java.nio.channels.SocketChannel[co
> > > >> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
> > > >> 10.141.0.79:60020]
> > > >>
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
> > > >> at
> > > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > >> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > >> at
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > >> at
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > >> at java.lang.Thread.run(Thread.java:662)
> > > >> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> > > >> unrecoverable error from region server
> > > >>
> > > >> Is there a way in which I can increase the timeout period?
> > > >>
> > > >> Thank you,
> > > >>
> > > >> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell
> > > >> <ap...@apache.org>wrote:
> > > >>
> > > >>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
> > > >>> <fu...@gmail.com> wrote:
> > > >>> > I have to bulkload 6 tables which contain the same information
> but
> > > >>> > with
> > > >>> a
> > > >>> > different order to cover all possible access patterns. Would it
> be
> > a
> > > >>> good
> > > >>> > idea to do only one load and use co-processors to populate the
> > other
> > > >>> > tables, instead of doing the traditional MR bulkload which would
> > > >>> require 6
> > > >>> > separate jobs?
> > > >>>
> > > >>> Without knowing more than you've said, it seems better to use MR to
> > > >>> build all input.
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> - Andy
> > > >>>
> > > >>> Problems worthy of attack prove their worth by hitting back. - Piet
> > > >>> Hein (via Tom White)
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Sever Fundatureanu
> > > >>
> > > >> Vrije Universiteit Amsterdam
> > > >> E-mail: fundatureanu.sever@gmail.com
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Sever Fundatureanu
> > > >
> > > > Vrije Universiteit Amsterdam
> > > > E-mail: fundatureanu.sever@gmail.com
> > > >
> > >
> >
> >
> >
> > --
> > Sever Fundatureanu
> >
> > Vrije Universiteit Amsterdam
> > E-mail: fundatureanu.sever@gmail.com
> >
>
--
Sever Fundatureanu
Vrije Universiteit Amsterdam
E-mail: fundatureanu.sever@gmail.com
Re: HBase bulk load through co-processors
Posted by Nick Dimiduk <nd...@gmail.com>.
Sever,
I presume you're loading your data via online Puts via the MR job (as
opposed to generating HFiles). What are you hoping to gain from a
coprocessor implementation vs the 6 MR jobs? Have you pre-split your
tables? Can the RegionServer(s) handle all the concurrent mappers?
-n
On Mon, Jul 2, 2012 at 11:58 AM, Sever Fundatureanu <
fundatureanu.sever@gmail.com> wrote:
> I agree that increasing the timeout is not the best option, I will work
> both on better balancing the load and maybe doing it in increments like you
> suggested. However for now I want a quick fix to the problem.
>
> Just to see if I understand this right: a zookeeper node redirects my
> client to a region server node and then my client talk directly to this
> region server; now the timeout happens on the client while talking to the
> RS right? It expects some kind of confirmation and it times out.. if this
> is the case how can I increase this timeout? I only found in the
> documentation "zookeeper.session.timeout" which is the timeout between
> zookeeper and HBase.
>
> Thanks,
> Sever
>
> On Mon, Jul 2, 2012 at 8:19 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Hi Sever,
> >
> > It seems one of the nodes in your cluster is overwhelmed with the load
> > you are giving him.
> >
> > So IMO, you have two options here:
> > First, you can try to reduce the load. I mean, split the bulk in
> > multiple smaller bulks and load them one by one to give the time to
> > your cluster to dispatch it correctly.
> > Second, you can inscreade the timeone from 60s to 120s. But you might
> > face the same issue with 120s so I really recommand the fist option.
> >
> > JM
> >
> > 2012/7/2, Sever Fundatureanu <fu...@gmail.com>:
> > > Can someone please help me with this?
> > >
> > > Thanks,
> > > Sever
> > >
> > > On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
> > > fundatureanu.sever@gmail.com> wrote:
> > >
> > >> My keys are built of 4 8-byte Ids. I am currently doing the load with
> > MR
> > >> but I get a timeout when doing the loadIncrementalFiles call:
> > >>
> > >> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> > >> unrecoverable error from region server
> > >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> > >> attempts=10, exceptions:
> > >> Sun Jun 24 21:29:01 CEST 2012,
> > >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9,
> > >> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
> > >> 10.141.0.79:60020
> > >> failed on socket timeout exception: java.net.SocketTimeoutException:
> > >> 60000
> > >> millis timeout while waiting for channel to be ready for read. ch :
> > >> java.nio.channels.SocketChannel[co
> > >> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
> > >> 10.141.0.79:60020]
> > >>
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
> > >> at
> > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > >> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > >> at
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > >> at
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > >> at java.lang.Thread.run(Thread.java:662)
> > >> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> > >> unrecoverable error from region server
> > >>
> > >> Is there a way in which I can increase the timeout period?
> > >>
> > >> Thank you,
> > >>
> > >> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell
> > >> <ap...@apache.org>wrote:
> > >>
> > >>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
> > >>> <fu...@gmail.com> wrote:
> > >>> > I have to bulkload 6 tables which contain the same information but
> > >>> > with
> > >>> a
> > >>> > different order to cover all possible access patterns. Would it be
> a
> > >>> good
> > >>> > idea to do only one load and use co-processors to populate the
> other
> > >>> > tables, instead of doing the traditional MR bulkload which would
> > >>> require 6
> > >>> > separate jobs?
> > >>>
> > >>> Without knowing more than you've said, it seems better to use MR to
> > >>> build all input.
> > >>>
> > >>> Best regards,
> > >>>
> > >>> - Andy
> > >>>
> > >>> Problems worthy of attack prove their worth by hitting back. - Piet
> > >>> Hein (via Tom White)
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Sever Fundatureanu
> > >>
> > >> Vrije Universiteit Amsterdam
> > >> E-mail: fundatureanu.sever@gmail.com
> > >>
> > >
> > >
> > >
> > > --
> > > Sever Fundatureanu
> > >
> > > Vrije Universiteit Amsterdam
> > > E-mail: fundatureanu.sever@gmail.com
> > >
> >
>
>
>
> --
> Sever Fundatureanu
>
> Vrije Universiteit Amsterdam
> E-mail: fundatureanu.sever@gmail.com
>
Re: HBase bulk load through co-processors
Posted by Sever Fundatureanu <fu...@gmail.com>.
I agree that increasing the timeout is not the best option, I will work
both on better balancing the load and maybe doing it in increments like you
suggested. However for now I want a quick fix to the problem.
Just to see if I understand this right: a zookeeper node redirects my
client to a region server node and then my client talk directly to this
region server; now the timeout happens on the client while talking to the
RS right? It expects some kind of confirmation and it times out.. if this
is the case how can I increase this timeout? I only found in the
documentation "zookeeper.session.timeout" which is the timeout between
zookeeper and HBase.
Thanks,
Sever
On Mon, Jul 2, 2012 at 8:19 PM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:
> Hi Sever,
>
> It seems one of the nodes in your cluster is overwhelmed with the load
> you are giving him.
>
> So IMO, you have two options here:
> First, you can try to reduce the load. I mean, split the bulk in
> multiple smaller bulks and load them one by one to give the time to
> your cluster to dispatch it correctly.
> Second, you can inscreade the timeone from 60s to 120s. But you might
> face the same issue with 120s so I really recommand the fist option.
>
> JM
>
> 2012/7/2, Sever Fundatureanu <fu...@gmail.com>:
> > Can someone please help me with this?
> >
> > Thanks,
> > Sever
> >
> > On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
> > fundatureanu.sever@gmail.com> wrote:
> >
> >> My keys are built of 4 8-byte Ids. I am currently doing the load with
> MR
> >> but I get a timeout when doing the loadIncrementalFiles call:
> >>
> >> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> >> unrecoverable error from region server
> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> >> attempts=10, exceptions:
> >> Sun Jun 24 21:29:01 CEST 2012,
> >> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9,
> >> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
> >> 10.141.0.79:60020
> >> failed on socket timeout exception: java.net.SocketTimeoutException:
> >> 60000
> >> millis timeout while waiting for channel to be ready for read. ch :
> >> java.nio.channels.SocketChannel[co
> >> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
> >> 10.141.0.79:60020]
> >>
> >> at
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
> >> at
> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:662)
> >> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles: Encountered
> >> unrecoverable error from region server
> >>
> >> Is there a way in which I can increase the timeout period?
> >>
> >> Thank you,
> >>
> >> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell
> >> <ap...@apache.org>wrote:
> >>
> >>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
> >>> <fu...@gmail.com> wrote:
> >>> > I have to bulkload 6 tables which contain the same information but
> >>> > with
> >>> a
> >>> > different order to cover all possible access patterns. Would it be a
> >>> good
> >>> > idea to do only one load and use co-processors to populate the other
> >>> > tables, instead of doing the traditional MR bulkload which would
> >>> require 6
> >>> > separate jobs?
> >>>
> >>> Without knowing more than you've said, it seems better to use MR to
> >>> build all input.
> >>>
> >>> Best regards,
> >>>
> >>> - Andy
> >>>
> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> >>> Hein (via Tom White)
> >>>
> >>
> >>
> >>
> >> --
> >> Sever Fundatureanu
> >>
> >> Vrije Universiteit Amsterdam
> >> E-mail: fundatureanu.sever@gmail.com
> >>
> >
> >
> >
> > --
> > Sever Fundatureanu
> >
> > Vrije Universiteit Amsterdam
> > E-mail: fundatureanu.sever@gmail.com
> >
>
--
Sever Fundatureanu
Vrije Universiteit Amsterdam
E-mail: fundatureanu.sever@gmail.com
Re: HBase bulk load through co-processors
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Sever,
It seems one of the nodes in your cluster is overwhelmed with the load
you are giving him.
So IMO, you have two options here:
First, you can try to reduce the load. I mean, split the bulk in
multiple smaller bulks and load them one by one to give the time to
your cluster to dispatch it correctly.
Second, you can inscreade the timeone from 60s to 120s. But you might
face the same issue with 120s so I really recommand the fist option.
JM
2012/7/2, Sever Fundatureanu <fu...@gmail.com>:
> Can someone please help me with this?
>
> Thanks,
> Sever
>
> On Tue, Jun 26, 2012 at 8:14 PM, Sever Fundatureanu <
> fundatureanu.sever@gmail.com> wrote:
>
>> My keys are built of 4 8-byte Ids. I am currently doing the load with MR
>> but I get a timeout when doing the loadIncrementalFiles call:
>>
>> 12/06/24 21:29:01 ERROR mapreduce.LoadIncrementalHFiles: Encountered
>> unrecoverable error from region server
>> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
>> attempts=10, exceptions:
>> Sun Jun 24 21:29:01 CEST 2012,
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@4699ecf9,
>> java.net.SocketTimeoutException: Call to das3002.cm.cluster/
>> 10.141.0.79:60020
>> failed on socket timeout exception: java.net.SocketTimeoutException:
>> 60000
>> millis timeout while waiting for channel to be ready for read. ch :
>> java.nio.channels.SocketChannel[co
>> nnected local=/10.141.0.254:43240 remote=das3002.cm.cluster/
>> 10.141.0.79:60020]
>>
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:487)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:275)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:273)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> 12/06/24 21:30:52 ERROR mapreduce.LoadIncrementalHFiles: Encountered
>> unrecoverable error from region server
>>
>> Is there a way in which I can increase the timeout period?
>>
>> Thank you,
>>
>> On Tue, Jun 26, 2012 at 7:05 PM, Andrew Purtell
>> <ap...@apache.org>wrote:
>>
>>> On Tue, Jun 26, 2012 at 9:56 AM, Sever Fundatureanu
>>> <fu...@gmail.com> wrote:
>>> > I have to bulkload 6 tables which contain the same information but
>>> > with
>>> a
>>> > different order to cover all possible access patterns. Would it be a
>>> good
>>> > idea to do only one load and use co-processors to populate the other
>>> > tables, instead of doing the traditional MR bulkload which would
>>> require 6
>>> > separate jobs?
>>>
>>> Without knowing more than you've said, it seems better to use MR to
>>> build all input.
>>>
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>> Hein (via Tom White)
>>>
>>
>>
>>
>> --
>> Sever Fundatureanu
>>
>> Vrije Universiteit Amsterdam
>> E-mail: fundatureanu.sever@gmail.com
>>
>
>
>
> --
> Sever Fundatureanu
>
> Vrije Universiteit Amsterdam
> E-mail: fundatureanu.sever@gmail.com
>