You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by John Weatherford <jo...@telescope.tv> on 2013/10/10 03:43:21 UTC
Coprocessor Increments
Hi All,
We have been running into an RPC deadlock issue on HBase and from
investigation, we believe the root of the issue is in us doing cross
region increments from a coprocessor. After some further searching and
reading over this
<http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
we think that we can solve this by doing the increments locally on the
region. My question, is what happens if the row value specified does not
land in the current region. We can obviously do our best to make sure
that it does, but is there any way to be absolutely sure that it is?
This is supposing we use incrementColumnValue() out of the HRegion class
(
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/regionserver/HRegion.html#incrementColumnValue(byte[],
byte[], byte[], long, boolean)
<http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/regionserver/HRegion.html#incrementColumnValue%28byte[],%20byte[],%20byte[],%20long,%20boolean%29>)
Here is the method signature for simplicity
public long*incrementColumnValue*(byte[] row,
byte[] family,
byte[] qualifier,
long amount,
boolean writeToWAL)
If we specify a row, there seems to be no guarantee that row will be
confined to the region the coprocessor is on.
If the call does force the increment to be on the same region, what will
happen if a later call ends up on another region but with the same name.
Contrived Example
Insert rowkey "California-12345" triggers a coprocessor to call
incrementColumnValue() with a rowkey of "California-total" all on Region 1.
This would likely be on an insert on the same region. But as the table
grows, this secondary insert could end up on another region. If it is
confined, then suppose we later insert "California-95424" which still
triggers a call to incrementColumnValue() with a rowkey of
"California-total" all on Region 2.
Are we now left with two rowkeys of "California-total"? One on each
region server? If so, what happens if these two regions are compacted
into one?
Hopefully this all makes sense. We are on Hbase 0.94.10. If we are going
about this all wrong, that could be the issue as well :)
Thanks.
-John Weatherford
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
Right….
Ok, that explains some of the issues.
From a design perspective… it would make sense if the RO updated the same row, or a row in a separate table. But not a different row in the same table.
Outside of the additional RPC overhead, has anyone had a problem updating a row in a different table? If I understand correctly, there wouldn't be a chance of any 'deadlocks', just a bit of latency or the potential for an RPC error if you have a flakey network / nodes…
On Oct 11, 2013, at 12:26 PM, Vladimir Rodionov <vr...@carrieriq.com> wrote:
>>> With respect to the OP's design… does the deadlock occur because he's trying to update a column in a different row within the same table?
>
> Because he is trying to update *row* in a different Region (and potentially in different RS).
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Michael Segel [msegel_hadoop@hotmail.com]
> Sent: Friday, October 11, 2013 9:10 AM
> To: user@hbase.apache.org
> Cc: Vladimir Rodionov
> Subject: Re: Coprocessor Increments
>
>
> Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
>
Re: Coprocessor Increments
Posted by Ted Yu <yu...@gmail.com>.
John:
When RegionObserver increments a well known row in the same region as the
write, a hotspot is created in that region: every write would update that
row.
Bigger issue is that if you allow the regions to split, evaluating the
initial values for the special row in the daughter regions at time of split
would be not so fast. It depends on your use case whether you want to keep
the value in the 'Total' row accurate all the time.
Looks like Tom's method could be a better fit for you.
Cheers
On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
john.weatherford@telescope.tv> wrote:
> OP Here :)
>
> Our current design involves a Region Observer on a table that does
> increments on a second table. We took the approach that Michael said and
> inside the RO, we got a new connection and everything. We believe this is
> causing deadlocks for us. Our next attempt is going to be writing to
> another row in the same table where we will store the increments. If this
> doesn't work, we are going to simply pull the increments out of the RO and
> do them in the application or in Flume.
>
> @Tom Brown
> I would be very interested to hear more about your solution of
> aggregating the increments in another system that is then responsible for
> updating in Hbase.
>
> -jW
>
>
> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>
>> With respect to the OP's design… does the deadlock occur because he's
>>>> trying to update a column in a different row within the same table?
>>>>
>>>
>> Because he is trying to update *row* in a different Region (and
>> potentially in different RS).
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>>
>> ______________________________**__________
>> From: Michael Segel [msegel_hadoop@hotmail.com]
>> Sent: Friday, October 11, 2013 9:10 AM
>> To: user@hbase.apache.org
>> Cc: Vladimir Rodionov
>> Subject: Re: Coprocessor Increments
>>
>>
>> Confidentiality Notice: The information contained in this message,
>> including any attachments hereto, may be confidential and is intended to be
>> read only by the individual or entity to whom this message is addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any form,
>> is strictly prohibited. If you have received this message in error, please
>> immediately notify the sender and/or Notifications@carrieriq.com and
>> delete or destroy any copy of this message and its attachments.
>>
>
Re: Coprocessor Increments
Posted by Michael Segel <mi...@hotmail.com>.
John,
How are you creating your htable connection?
Assuming that the RO is overloading the post_put() method, are you instantiating a new HTable connection each time or are you creating a pool at instantiation and then getting a connection from the pool?
It may be that you're not seeing a deadlock issue in the thread handlers.
Also how fast are you adding rows?
I'm wondering if you're running in to some thread safe issues...
On Oct 11, 2013, at 8:06 PM, John Weatherford <jo...@telescope.tv> wrote:
> OP Here :)
>
> Our current design involves a Region Observer on a table that does increments on a second table. We took the approach that Michael said and inside the RO, we got a new connection and everything. We believe this is causing deadlocks for us. Our next attempt is going to be writing to another row in the same table where we will store the increments. If this doesn't work, we are going to simply pull the increments out of the RO and do them in the application or in Flume.
>
> @Tom Brown
> I would be very interested to hear more about your solution of aggregating the increments in another system that is then responsible for updating in Hbase.
>
> -jW
>
> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>>>> With respect to the OP's design… does the deadlock occur because he's trying to update a column in a different row within the same table?
>>
>> Because he is trying to update *row* in a different Region (and potentially in different RS).
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>>
>> ________________________________________
>> From: Michael Segel [msegel_hadoop@hotmail.com]
>> Sent: Friday, October 11, 2013 9:10 AM
>> To: user@hbase.apache.org
>> Cc: Vladimir Rodionov
>> Subject: Re: Coprocessor Increments
>>
>>
>> Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Coprocessor Increments
Posted by John Weatherford <jo...@telescope.tv>.
Tom,
Thanks for your explanation. It helped me understand why we are
getting deadlocks as well. I (foolishly) thought of tables working on
exclusive regions. In my mind I thought I had a DAG since Table 1 was
always writing to table 2, and the graph never connected back or
cycled. I failed to realized that although the tables were a DAG, the
regions obviously were not.
-John
On Mon 14 Oct 2013 09:36:09 AM PDT, Tom Brown wrote:
> If you allow coprocessors to connect to any other server the connections
> between the HBase nodes can be represented as a directed graph. This means
> that deadlock is possible (when N1 is blocked waiting for N2, and N2 is
> blocked waiting for N1). In order to guarantee that no deadlock will occur,
> you have to be able to represent the connections as a directed acyclic
> graph. And to do this, you cannot have nodes connecting directly to each
> other.
>
> You could also achieve this by carefully assigning your regions. If the
> primary table (PT) is located exclusively on nodes N1..N3, and the
> secondary/index table is located exclusively on nodes N4..N6, (assuming
> updates to the secondary/index table never change the PT) the connections
> would remain a DAG and deadlock would be impossible.
>
> You're right that by putting the business logic into the coprocessor you
> gain the ability to easily allow any group to access your cluster. But that
> access isn't free. To use a SQL analogy: large organizations always protect
> their SQL servers with a DBA. They do this because the potential downsides
> of allowing unsupervised and unstructured access are too great.
>
> YMMV
>
> --Tom
>
>
>
> On Mon, Oct 14, 2013 at 8:50 AM, Michael Segel <ms...@hotmail.com>wrote:
>
>> Anil,
>>
>> I wasn't suggesting that you can't do what you're doing, but you end up
>> running in to the risks which coprocessors are supposed to remove. The
>> standard YMMV always applies.
>>
>> You have a cluster… another team in your company wants to use the cluster.
>> So instead of the cluster being a single resource for your app/team, it now
>> becomes a shared resource. So now you have people accessing HBase for
>> multiple apps.
>>
>> You could then run multiple HBase HMasters with different locations for
>> files, however… this can get messy.
>> HOYA seems to suggest this as the future. If so, then you have to wonder
>> about data locality.
>>
>> Having your app update the primary table and then the secondary index is
>> always a good fallback, however you need to ensure that you understand the
>> risks.
>>
>> With respect to secondary indexes… if you decouple the writes… you can get
>> better throughput. Note that the code becomes a bit more complex because
>> you're going to have to introduce a couple of different things. But thats
>> something for a different discussion…
>>
>>
>> On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com> wrote:
>>
>>> Inline.
>>>
>>> On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <
>> msegel_hadoop@hotmail.com>wrote:
>>>
>>>> Ok…
>>>>
>>>> Sure you can have your app update the secondary index table.
>>>> The only issue with that is if someone updates the base table outside of
>>>> your app,
>>>> they may or may not increment the secondary index.
>>>>
>>> Anil: We dont allow people to write data into HBase from their own HBase
>>> client. We control the writes into HBase. So, we dont have the problem of
>>> secondary index not getting written.
>>> For example, If you expose a restful web service you can easily control
>> the
>>> writes to HBase. Even, if user requests to write one row in "main table",
>>> you application can have the logic to writing in "Secondary index"
>> tables.
>>> In this way, it is transparent to users also. You can add/remove seconday
>>> indexes as you want.
>>>
>>>> Note that your secondary index doesn't have to be an inverted table, but
>>>> could be SOLR, LUCENE or something else.
>>>>
>>> Anil:As of now, we are happy with Inverted tables as they fit to our use
>>> case.
>>>
>>>>
>>>> So you really want to secondary indexes on the server.
>>>>
>>>> There are a couple of things that could improve the performance,
>> although
>>>> the write to the secondary index would most likely lag under heavy load.
>>>>
>>>>
>>>> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com> wrote:
>>>>
>>>>> John,
>>>>>
>>>>> My 2 cents:
>>>>> I tried implementing Secondary Index by using Region Observers on Put.
>> It
>>>>> works well under low load. But, under heavy load the RO could not keep
>> up
>>>>> with load cross region server writes.
>>>>> Then, i decided not to use RO as per Andrew's explanation and I moved
>>>> all
>>>>> the logic of building secondary index tables on my HBase Client . Since
>>>>> then, the system has been running fine under heavy load.
>>>>> IMO, if you will use RO and do cross RS read/write then perhaps this
>> will
>>>>> become your bottleneck in HBase.
>>>>> Is it possible for you to avoid RO and control the writes/updates from
>>>>> client side?
>>>>>
>>>>> Thanks,
>>>>> Anil Gupta
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
>>>>> john.weatherford@telescope.tv> wrote:
>>>>>
>>>>>> OP Here :)
>>>>>>
>>>>>> Our current design involves a Region Observer on a table that does
>>>>>> increments on a second table. We took the approach that Michael said
>> and
>>>>>> inside the RO, we got a new connection and everything. We believe this
>>>> is
>>>>>> causing deadlocks for us. Our next attempt is going to be writing to
>>>>>> another row in the same table where we will store the increments. If
>>>> this
>>>>>> doesn't work, we are going to simply pull the increments out of the RO
>>>> and
>>>>>> do them in the application or in Flume.
>>>>>>
>>>>>> @Tom Brown
>>>>>> I would be very interested to hear more about your solution of
>>>>>> aggregating the increments in another system that is then responsible
>>>> for
>>>>>> updating in Hbase.
>>>>>>
>>>>>> -jW
>>>>>>
>>>>>>
>>>>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>>>>>>
>>>>>>> With respect to the OP's design… does the deadlock occur because he's
>>>>>>>>> trying to update a column in a different row within the same table?
>>>>>>>>>
>>>>>>>>
>>>>>>> Because he is trying to update *row* in a different Region (and
>>>>>>> potentially in different RS).
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Vladimir Rodionov
>>>>>>> Principal Platform Engineer
>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>
>>>>>>> ______________________________**__________
>>>>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
>>>>>>> Sent: Friday, October 11, 2013 9:10 AM
>>>>>>> To: user@hbase.apache.org
>>>>>>> Cc: Vladimir Rodionov
>>>>>>> Subject: Re: Coprocessor Increments
>>>>>>>
>>>>>>>
>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>> including any attachments hereto, may be confidential and is intended
>>>> to be
>>>>>>> read only by the individual or entity to whom this message is
>>>> addressed. If
>>>>>>> the reader of this message is not the intended recipient or an agent
>> or
>>>>>>> designee of the intended recipient, please note that any review, use,
>>>>>>> disclosure or distribution of this message or its attachments, in any
>>>> form,
>>>>>>> is strictly prohibited. If you have received this message in error,
>>>> please
>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anil Gupta
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>
>>
>
Re: Coprocessor Increments
Posted by anil gupta <an...@gmail.com>.
Hi Ted,
Sure, I would like to revive it. My bad that i didnt wrap up the patch. I
am also in the middle of making this coprocessor handle "nulls first" and
"nulls last" clause. I am targeting to do that in a month or so. Thanks
for reminding me.
~Anil
On Mon, Oct 14, 2013 at 3:34 PM, Ted Yu <yu...@gmail.com> wrote:
> Anil:
> bq. We also use CP's wherever they are appropriate(like HBASE-7474).
>
> HBASE-7474 has been dormant for several months. Do you want to revive it ?
>
> Cheers
>
>
> On Mon, Oct 14, 2013 at 3:25 PM, anil gupta <an...@gmail.com> wrote:
>
> > Inline.
> >
> >
> > On Mon, Oct 14, 2013 at 7:50 AM, Michael Segel <
> msegel_hadoop@hotmail.com
> > >wrote:
> >
> > > Anil,
> > >
> > > I wasn't suggesting that you can't do what you're doing, but you end up
> > > running in to the risks which coprocessors are supposed to remove. The
> > > standard YMMV always applies.
> > >
> > Agree with you. But, as per my knowledge and experience with
> coprocessors,
> > they are meant to be used for operations that are local to RS. Otherwise,
> > you are in danger of running into deadlocks, scalability issues.
> >
> > >
> > > You have a cluster… another team in your company wants to use the
> > cluster.
> > > So instead of the cluster being a single resource for your app/team, it
> > now
> > > becomes a shared resource. So now you have people accessing HBase for
> > > multiple apps.
> > >
> > Well, its a separation of responsibility in this case. We don't want
> teams
> > to step each other toes and at the same time work well as an ecosystem.
> > Rule: Other teams can use same cluster. But they cannot write directly
> into
> > the tables that we own/control. If they want to write into our tables
> then
> > they have to use our HBase Client.
> >
> > >
> > > You could then run multiple HBase HMasters with different locations for
> > > files, however… this can get messy.
> > > HOYA seems to suggest this as the future. If so, then you have to
> wonder
> > > about data locality.
> > >
> > HOYA is not even in beta at present. So, right now we are not thinking
> > about it.
> >
> > >
> > > Having your app update the primary table and then the secondary index
> is
> > > always a good fallback, however you need to ensure that you understand
> > the
> > > risks.
> > >
> > Agree, i understand that there is risk. But, you have to bite the bullet
> > when you are doing something that is not supported out of the box. We
> also
> > use CP's wherever they are appropriate(like HBASE-7474).
> >
> > >
> > > With respect to secondary indexes… if you decouple the writes… you can
> > get
> > > better throughput. Note that the code becomes a bit more complex
> because
> > > you're going to have to introduce a couple of different things. But
> > thats
> > > something for a different discussion…
> > >
> > Whether to use CP or not, depends on the use case. In my opinion, CP's
> are
> > really powerful and an awesome feature in HBase. But, sometimes if not
> used
> > properly(like creating a Cyclic Graph as per Tom's example), they might
> be
> > problematic.
> >
> >
> > >
> > > On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com>
> wrote:
> > >
> > > > Inline.
> > > >
> > > > On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <
> > > msegel_hadoop@hotmail.com>wrote:
> > > >
> > > >> Ok…
> > > >>
> > > >> Sure you can have your app update the secondary index table.
> > > >> The only issue with that is if someone updates the base table
> outside
> > of
> > > >> your app,
> > > >> they may or may not increment the secondary index.
> > > >>
> > > > Anil: We dont allow people to write data into HBase from their own
> > HBase
> > > > client. We control the writes into HBase. So, we dont have the
> problem
> > of
> > > > secondary index not getting written.
> > > > For example, If you expose a restful web service you can easily
> control
> > > the
> > > > writes to HBase. Even, if user requests to write one row in "main
> > table",
> > > > you application can have the logic to writing in "Secondary index"
> > > tables.
> > > > In this way, it is transparent to users also. You can add/remove
> > seconday
> > > > indexes as you want.
> > > >
> > > >> Note that your secondary index doesn't have to be an inverted table,
> > but
> > > >> could be SOLR, LUCENE or something else.
> > > >>
> > > > Anil:As of now, we are happy with Inverted tables as they fit to our
> > use
> > > > case.
> > > >
> > > >>
> > > >> So you really want to secondary indexes on the server.
> > > >>
> > > >> There are a couple of things that could improve the performance,
> > > although
> > > >> the write to the secondary index would most likely lag under heavy
> > load.
> > > >>
> > > >>
> > > >> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com>
> > wrote:
> > > >>
> > > >>> John,
> > > >>>
> > > >>> My 2 cents:
> > > >>> I tried implementing Secondary Index by using Region Observers on
> > Put.
> > > It
> > > >>> works well under low load. But, under heavy load the RO could not
> > keep
> > > up
> > > >>> with load cross region server writes.
> > > >>> Then, i decided not to use RO as per Andrew's explanation and I
> > moved
> > > >> all
> > > >>> the logic of building secondary index tables on my HBase Client .
> > Since
> > > >>> then, the system has been running fine under heavy load.
> > > >>> IMO, if you will use RO and do cross RS read/write then perhaps
> this
> > > will
> > > >>> become your bottleneck in HBase.
> > > >>> Is it possible for you to avoid RO and control the writes/updates
> > from
> > > >>> client side?
> > > >>>
> > > >>> Thanks,
> > > >>> Anil Gupta
> > > >>>
> > > >>>
> > > >>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> > > >>> john.weatherford@telescope.tv> wrote:
> > > >>>
> > > >>>> OP Here :)
> > > >>>>
> > > >>>> Our current design involves a Region Observer on a table that does
> > > >>>> increments on a second table. We took the approach that Michael
> said
> > > and
> > > >>>> inside the RO, we got a new connection and everything. We believe
> > this
> > > >> is
> > > >>>> causing deadlocks for us. Our next attempt is going to be writing
> to
> > > >>>> another row in the same table where we will store the increments.
> If
> > > >> this
> > > >>>> doesn't work, we are going to simply pull the increments out of
> the
> > RO
> > > >> and
> > > >>>> do them in the application or in Flume.
> > > >>>>
> > > >>>> @Tom Brown
> > > >>>> I would be very interested to hear more about your solution of
> > > >>>> aggregating the increments in another system that is then
> > responsible
> > > >> for
> > > >>>> updating in Hbase.
> > > >>>>
> > > >>>> -jW
> > > >>>>
> > > >>>>
> > > >>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
> > > >>>>
> > > >>>>> With respect to the OP's design… does the deadlock occur because
> > he's
> > > >>>>>>> trying to update a column in a different row within the same
> > table?
> > > >>>>>>>
> > > >>>>>>
> > > >>>>> Because he is trying to update *row* in a different Region (and
> > > >>>>> potentially in different RS).
> > > >>>>>
> > > >>>>> Best regards,
> > > >>>>> Vladimir Rodionov
> > > >>>>> Principal Platform Engineer
> > > >>>>> Carrier IQ, www.carrieriq.com
> > > >>>>> e-mail: vrodionov@carrieriq.com
> > > >>>>>
> > > >>>>> ______________________________**__________
> > > >>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
> > > >>>>> Sent: Friday, October 11, 2013 9:10 AM
> > > >>>>> To: user@hbase.apache.org
> > > >>>>> Cc: Vladimir Rodionov
> > > >>>>> Subject: Re: Coprocessor Increments
> > > >>>>>
> > > >>>>>
> > > >>>>> Confidentiality Notice: The information contained in this
> message,
> > > >>>>> including any attachments hereto, may be confidential and is
> > intended
> > > >> to be
> > > >>>>> read only by the individual or entity to whom this message is
> > > >> addressed. If
> > > >>>>> the reader of this message is not the intended recipient or an
> > agent
> > > or
> > > >>>>> designee of the intended recipient, please note that any review,
> > use,
> > > >>>>> disclosure or distribution of this message or its attachments, in
> > any
> > > >> form,
> > > >>>>> is strictly prohibited. If you have received this message in
> > error,
> > > >> please
> > > >>>>> immediately notify the sender and/or
> Notifications@carrieriq.comand
> > > >>>>> delete or destroy any copy of this message and its attachments.
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Thanks & Regards,
> > > >>> Anil Gupta
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > >
> > >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Coprocessor Increments
Posted by Ted Yu <yu...@gmail.com>.
Anil:
bq. We also use CP's wherever they are appropriate(like HBASE-7474).
HBASE-7474 has been dormant for several months. Do you want to revive it ?
Cheers
On Mon, Oct 14, 2013 at 3:25 PM, anil gupta <an...@gmail.com> wrote:
> Inline.
>
>
> On Mon, Oct 14, 2013 at 7:50 AM, Michael Segel <msegel_hadoop@hotmail.com
> >wrote:
>
> > Anil,
> >
> > I wasn't suggesting that you can't do what you're doing, but you end up
> > running in to the risks which coprocessors are supposed to remove. The
> > standard YMMV always applies.
> >
> Agree with you. But, as per my knowledge and experience with coprocessors,
> they are meant to be used for operations that are local to RS. Otherwise,
> you are in danger of running into deadlocks, scalability issues.
>
> >
> > You have a cluster… another team in your company wants to use the
> cluster.
> > So instead of the cluster being a single resource for your app/team, it
> now
> > becomes a shared resource. So now you have people accessing HBase for
> > multiple apps.
> >
> Well, its a separation of responsibility in this case. We don't want teams
> to step each other toes and at the same time work well as an ecosystem.
> Rule: Other teams can use same cluster. But they cannot write directly into
> the tables that we own/control. If they want to write into our tables then
> they have to use our HBase Client.
>
> >
> > You could then run multiple HBase HMasters with different locations for
> > files, however… this can get messy.
> > HOYA seems to suggest this as the future. If so, then you have to wonder
> > about data locality.
> >
> HOYA is not even in beta at present. So, right now we are not thinking
> about it.
>
> >
> > Having your app update the primary table and then the secondary index is
> > always a good fallback, however you need to ensure that you understand
> the
> > risks.
> >
> Agree, i understand that there is risk. But, you have to bite the bullet
> when you are doing something that is not supported out of the box. We also
> use CP's wherever they are appropriate(like HBASE-7474).
>
> >
> > With respect to secondary indexes… if you decouple the writes… you can
> get
> > better throughput. Note that the code becomes a bit more complex because
> > you're going to have to introduce a couple of different things. But
> thats
> > something for a different discussion…
> >
> Whether to use CP or not, depends on the use case. In my opinion, CP's are
> really powerful and an awesome feature in HBase. But, sometimes if not used
> properly(like creating a Cyclic Graph as per Tom's example), they might be
> problematic.
>
>
> >
> > On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com> wrote:
> >
> > > Inline.
> > >
> > > On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <
> > msegel_hadoop@hotmail.com>wrote:
> > >
> > >> Ok…
> > >>
> > >> Sure you can have your app update the secondary index table.
> > >> The only issue with that is if someone updates the base table outside
> of
> > >> your app,
> > >> they may or may not increment the secondary index.
> > >>
> > > Anil: We dont allow people to write data into HBase from their own
> HBase
> > > client. We control the writes into HBase. So, we dont have the problem
> of
> > > secondary index not getting written.
> > > For example, If you expose a restful web service you can easily control
> > the
> > > writes to HBase. Even, if user requests to write one row in "main
> table",
> > > you application can have the logic to writing in "Secondary index"
> > tables.
> > > In this way, it is transparent to users also. You can add/remove
> seconday
> > > indexes as you want.
> > >
> > >> Note that your secondary index doesn't have to be an inverted table,
> but
> > >> could be SOLR, LUCENE or something else.
> > >>
> > > Anil:As of now, we are happy with Inverted tables as they fit to our
> use
> > > case.
> > >
> > >>
> > >> So you really want to secondary indexes on the server.
> > >>
> > >> There are a couple of things that could improve the performance,
> > although
> > >> the write to the secondary index would most likely lag under heavy
> load.
> > >>
> > >>
> > >> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com>
> wrote:
> > >>
> > >>> John,
> > >>>
> > >>> My 2 cents:
> > >>> I tried implementing Secondary Index by using Region Observers on
> Put.
> > It
> > >>> works well under low load. But, under heavy load the RO could not
> keep
> > up
> > >>> with load cross region server writes.
> > >>> Then, i decided not to use RO as per Andrew's explanation and I
> moved
> > >> all
> > >>> the logic of building secondary index tables on my HBase Client .
> Since
> > >>> then, the system has been running fine under heavy load.
> > >>> IMO, if you will use RO and do cross RS read/write then perhaps this
> > will
> > >>> become your bottleneck in HBase.
> > >>> Is it possible for you to avoid RO and control the writes/updates
> from
> > >>> client side?
> > >>>
> > >>> Thanks,
> > >>> Anil Gupta
> > >>>
> > >>>
> > >>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> > >>> john.weatherford@telescope.tv> wrote:
> > >>>
> > >>>> OP Here :)
> > >>>>
> > >>>> Our current design involves a Region Observer on a table that does
> > >>>> increments on a second table. We took the approach that Michael said
> > and
> > >>>> inside the RO, we got a new connection and everything. We believe
> this
> > >> is
> > >>>> causing deadlocks for us. Our next attempt is going to be writing to
> > >>>> another row in the same table where we will store the increments. If
> > >> this
> > >>>> doesn't work, we are going to simply pull the increments out of the
> RO
> > >> and
> > >>>> do them in the application or in Flume.
> > >>>>
> > >>>> @Tom Brown
> > >>>> I would be very interested to hear more about your solution of
> > >>>> aggregating the increments in another system that is then
> responsible
> > >> for
> > >>>> updating in Hbase.
> > >>>>
> > >>>> -jW
> > >>>>
> > >>>>
> > >>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
> > >>>>
> > >>>>> With respect to the OP's design… does the deadlock occur because
> he's
> > >>>>>>> trying to update a column in a different row within the same
> table?
> > >>>>>>>
> > >>>>>>
> > >>>>> Because he is trying to update *row* in a different Region (and
> > >>>>> potentially in different RS).
> > >>>>>
> > >>>>> Best regards,
> > >>>>> Vladimir Rodionov
> > >>>>> Principal Platform Engineer
> > >>>>> Carrier IQ, www.carrieriq.com
> > >>>>> e-mail: vrodionov@carrieriq.com
> > >>>>>
> > >>>>> ______________________________**__________
> > >>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
> > >>>>> Sent: Friday, October 11, 2013 9:10 AM
> > >>>>> To: user@hbase.apache.org
> > >>>>> Cc: Vladimir Rodionov
> > >>>>> Subject: Re: Coprocessor Increments
> > >>>>>
> > >>>>>
> > >>>>> Confidentiality Notice: The information contained in this message,
> > >>>>> including any attachments hereto, may be confidential and is
> intended
> > >> to be
> > >>>>> read only by the individual or entity to whom this message is
> > >> addressed. If
> > >>>>> the reader of this message is not the intended recipient or an
> agent
> > or
> > >>>>> designee of the intended recipient, please note that any review,
> use,
> > >>>>> disclosure or distribution of this message or its attachments, in
> any
> > >> form,
> > >>>>> is strictly prohibited. If you have received this message in
> error,
> > >> please
> > >>>>> immediately notify the sender and/or Notifications@carrieriq.comand
> > >>>>> delete or destroy any copy of this message and its attachments.
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Thanks & Regards,
> > >>> Anil Gupta
> > >>
> > >>
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> >
> >
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Coprocessor Increments
Posted by anil gupta <an...@gmail.com>.
On Tue, Oct 15, 2013 at 11:12 AM, Michael Segel
<ms...@hotmail.com>wrote:
> Anil,
> > Agree with you. But, as per my knowledge and experience with
> coprocessors,
> > they are meant to be used for operations that are local to RS. Otherwise,
> > you are in danger of running into deadlocks, scalability issues.
>
>
> I also did a quick look at… HBASE-7474…
>
> You start with the assumption that all of your data is within a single
> region.
>
No, i dont. That sorting CP works even if your scan spans multiple RS's. I
do a merge sort at client side in that case. Please look at the code
closely. :)
>
> IMHO, this is a very narrow window for use cases.
>
> Most use cases have data that crosses region boundaries.
>
> From a design perspective… limiting the use case to only within region…
> kinda kills the reason for coprocessors to exist. Even looking back at the
> implementation by Google, they don't appear to have this problem… errr
> limitation.
>
> Sorry… IMHO and YMMV.
>
>
> On Oct 14, 2013, at 3:25 PM, anil gupta <an...@gmail.com> wrote:
>
> > Inline.
> >
> >
> > On Mon, Oct 14, 2013 at 7:50 AM, Michael Segel <
> msegel_hadoop@hotmail.com>wrote:
> >
> >> Anil,
> >>
> >> I wasn't suggesting that you can't do what you're doing, but you end up
> >> running in to the risks which coprocessors are supposed to remove. The
> >> standard YMMV always applies.
> >>
> > Agree with you. But, as per my knowledge and experience with
> coprocessors,
> > they are meant to be used for operations that are local to RS. Otherwise,
> > you are in danger of running into deadlocks, scalability issues.
> >
> >>
> >> You have a cluster… another team in your company wants to use the
> cluster.
> >> So instead of the cluster being a single resource for your app/team, it
> now
> >> becomes a shared resource. So now you have people accessing HBase for
> >> multiple apps.
> >>
> > Well, its a separation of responsibility in this case. We don't want
> teams
> > to step each other toes and at the same time work well as an ecosystem.
> > Rule: Other teams can use same cluster. But they cannot write directly
> into
> > the tables that we own/control. If they want to write into our tables
> then
> > they have to use our HBase Client.
> >
> >>
> >> You could then run multiple HBase HMasters with different locations for
> >> files, however… this can get messy.
> >> HOYA seems to suggest this as the future. If so, then you have to
> wonder
> >> about data locality.
> >>
> > HOYA is not even in beta at present. So, right now we are not thinking
> > about it.
> >
> >>
> >> Having your app update the primary table and then the secondary index is
> >> always a good fallback, however you need to ensure that you understand
> the
> >> risks.
> >>
> > Agree, i understand that there is risk. But, you have to bite the bullet
> > when you are doing something that is not supported out of the box. We
> also
> > use CP's wherever they are appropriate(like HBASE-7474).
> >
> >>
> >> With respect to secondary indexes… if you decouple the writes… you can
> get
> >> better throughput. Note that the code becomes a bit more complex because
> >> you're going to have to introduce a couple of different things. But
> thats
> >> something for a different discussion…
> >>
> > Whether to use CP or not, depends on the use case. In my opinion, CP's
> are
> > really powerful and an awesome feature in HBase. But, sometimes if not
> used
> > properly(like creating a Cyclic Graph as per Tom's example), they might
> be
> > problematic.
> >
> >
> >>
> >> On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com> wrote:
> >>
> >>> Inline.
> >>>
> >>> On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <
> >> msegel_hadoop@hotmail.com>wrote:
> >>>
> >>>> Ok…
> >>>>
> >>>> Sure you can have your app update the secondary index table.
> >>>> The only issue with that is if someone updates the base table outside
> of
> >>>> your app,
> >>>> they may or may not increment the secondary index.
> >>>>
> >>> Anil: We dont allow people to write data into HBase from their own
> HBase
> >>> client. We control the writes into HBase. So, we dont have the problem
> of
> >>> secondary index not getting written.
> >>> For example, If you expose a restful web service you can easily control
> >> the
> >>> writes to HBase. Even, if user requests to write one row in "main
> table",
> >>> you application can have the logic to writing in "Secondary index"
> >> tables.
> >>> In this way, it is transparent to users also. You can add/remove
> seconday
> >>> indexes as you want.
> >>>
> >>>> Note that your secondary index doesn't have to be an inverted table,
> but
> >>>> could be SOLR, LUCENE or something else.
> >>>>
> >>> Anil:As of now, we are happy with Inverted tables as they fit to our
> use
> >>> case.
> >>>
> >>>>
> >>>> So you really want to secondary indexes on the server.
> >>>>
> >>>> There are a couple of things that could improve the performance,
> >> although
> >>>> the write to the secondary index would most likely lag under heavy
> load.
> >>>>
> >>>>
> >>>> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com>
> wrote:
> >>>>
> >>>>> John,
> >>>>>
> >>>>> My 2 cents:
> >>>>> I tried implementing Secondary Index by using Region Observers on
> Put.
> >> It
> >>>>> works well under low load. But, under heavy load the RO could not
> keep
> >> up
> >>>>> with load cross region server writes.
> >>>>> Then, i decided not to use RO as per Andrew's explanation and I
> moved
> >>>> all
> >>>>> the logic of building secondary index tables on my HBase Client .
> Since
> >>>>> then, the system has been running fine under heavy load.
> >>>>> IMO, if you will use RO and do cross RS read/write then perhaps this
> >> will
> >>>>> become your bottleneck in HBase.
> >>>>> Is it possible for you to avoid RO and control the writes/updates
> from
> >>>>> client side?
> >>>>>
> >>>>> Thanks,
> >>>>> Anil Gupta
> >>>>>
> >>>>>
> >>>>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> >>>>> john.weatherford@telescope.tv> wrote:
> >>>>>
> >>>>>> OP Here :)
> >>>>>>
> >>>>>> Our current design involves a Region Observer on a table that does
> >>>>>> increments on a second table. We took the approach that Michael said
> >> and
> >>>>>> inside the RO, we got a new connection and everything. We believe
> this
> >>>> is
> >>>>>> causing deadlocks for us. Our next attempt is going to be writing to
> >>>>>> another row in the same table where we will store the increments. If
> >>>> this
> >>>>>> doesn't work, we are going to simply pull the increments out of the
> RO
> >>>> and
> >>>>>> do them in the application or in Flume.
> >>>>>>
> >>>>>> @Tom Brown
> >>>>>> I would be very interested to hear more about your solution of
> >>>>>> aggregating the increments in another system that is then
> responsible
> >>>> for
> >>>>>> updating in Hbase.
> >>>>>>
> >>>>>> -jW
> >>>>>>
> >>>>>>
> >>>>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
> >>>>>>
> >>>>>>> With respect to the OP's design… does the deadlock occur because
> he's
> >>>>>>>>> trying to update a column in a different row within the same
> table?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>> Because he is trying to update *row* in a different Region (and
> >>>>>>> potentially in different RS).
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Vladimir Rodionov
> >>>>>>> Principal Platform Engineer
> >>>>>>> Carrier IQ, www.carrieriq.com
> >>>>>>> e-mail: vrodionov@carrieriq.com
> >>>>>>>
> >>>>>>> ______________________________**__________
> >>>>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
> >>>>>>> Sent: Friday, October 11, 2013 9:10 AM
> >>>>>>> To: user@hbase.apache.org
> >>>>>>> Cc: Vladimir Rodionov
> >>>>>>> Subject: Re: Coprocessor Increments
> >>>>>>>
> >>>>>>>
> >>>>>>> Confidentiality Notice: The information contained in this message,
> >>>>>>> including any attachments hereto, may be confidential and is
> intended
> >>>> to be
> >>>>>>> read only by the individual or entity to whom this message is
> >>>> addressed. If
> >>>>>>> the reader of this message is not the intended recipient or an
> agent
> >> or
> >>>>>>> designee of the intended recipient, please note that any review,
> use,
> >>>>>>> disclosure or distribution of this message or its attachments, in
> any
> >>>> form,
> >>>>>>> is strictly prohibited. If you have received this message in
> error,
> >>>> please
> >>>>>>> immediately notify the sender and/or Notifications@carrieriq.comand
> >>>>>>> delete or destroy any copy of this message and its attachments.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Thanks & Regards,
> >>>>> Anil Gupta
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Anil Gupta
> >>
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
--
Thanks & Regards,
Anil Gupta
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
Anil,
> Agree with you. But, as per my knowledge and experience with coprocessors,
> they are meant to be used for operations that are local to RS. Otherwise,
> you are in danger of running into deadlocks, scalability issues.
I also did a quick look at… HBASE-7474…
You start with the assumption that all of your data is within a single region.
IMHO, this is a very narrow window for use cases.
Most use cases have data that crosses region boundaries.
From a design perspective… limiting the use case to only within region… kinda kills the reason for coprocessors to exist. Even looking back at the implementation by Google, they don't appear to have this problem… errr limitation.
Sorry… IMHO and YMMV.
On Oct 14, 2013, at 3:25 PM, anil gupta <an...@gmail.com> wrote:
> Inline.
>
>
> On Mon, Oct 14, 2013 at 7:50 AM, Michael Segel <ms...@hotmail.com>wrote:
>
>> Anil,
>>
>> I wasn't suggesting that you can't do what you're doing, but you end up
>> running in to the risks which coprocessors are supposed to remove. The
>> standard YMMV always applies.
>>
> Agree with you. But, as per my knowledge and experience with coprocessors,
> they are meant to be used for operations that are local to RS. Otherwise,
> you are in danger of running into deadlocks, scalability issues.
>
>>
>> You have a cluster… another team in your company wants to use the cluster.
>> So instead of the cluster being a single resource for your app/team, it now
>> becomes a shared resource. So now you have people accessing HBase for
>> multiple apps.
>>
> Well, its a separation of responsibility in this case. We don't want teams
> to step each other toes and at the same time work well as an ecosystem.
> Rule: Other teams can use same cluster. But they cannot write directly into
> the tables that we own/control. If they want to write into our tables then
> they have to use our HBase Client.
>
>>
>> You could then run multiple HBase HMasters with different locations for
>> files, however… this can get messy.
>> HOYA seems to suggest this as the future. If so, then you have to wonder
>> about data locality.
>>
> HOYA is not even in beta at present. So, right now we are not thinking
> about it.
>
>>
>> Having your app update the primary table and then the secondary index is
>> always a good fallback, however you need to ensure that you understand the
>> risks.
>>
> Agree, i understand that there is risk. But, you have to bite the bullet
> when you are doing something that is not supported out of the box. We also
> use CP's wherever they are appropriate(like HBASE-7474).
>
>>
>> With respect to secondary indexes… if you decouple the writes… you can get
>> better throughput. Note that the code becomes a bit more complex because
>> you're going to have to introduce a couple of different things. But thats
>> something for a different discussion…
>>
> Whether to use CP or not, depends on the use case. In my opinion, CP's are
> really powerful and an awesome feature in HBase. But, sometimes if not used
> properly(like creating a Cyclic Graph as per Tom's example), they might be
> problematic.
>
>
>>
>> On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com> wrote:
>>
>>> Inline.
>>>
>>> On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <
>> msegel_hadoop@hotmail.com>wrote:
>>>
>>>> Ok…
>>>>
>>>> Sure you can have your app update the secondary index table.
>>>> The only issue with that is if someone updates the base table outside of
>>>> your app,
>>>> they may or may not increment the secondary index.
>>>>
>>> Anil: We dont allow people to write data into HBase from their own HBase
>>> client. We control the writes into HBase. So, we dont have the problem of
>>> secondary index not getting written.
>>> For example, If you expose a restful web service you can easily control
>> the
>>> writes to HBase. Even, if user requests to write one row in "main table",
>>> you application can have the logic to writing in "Secondary index"
>> tables.
>>> In this way, it is transparent to users also. You can add/remove seconday
>>> indexes as you want.
>>>
>>>> Note that your secondary index doesn't have to be an inverted table, but
>>>> could be SOLR, LUCENE or something else.
>>>>
>>> Anil:As of now, we are happy with Inverted tables as they fit to our use
>>> case.
>>>
>>>>
>>>> So you really want to secondary indexes on the server.
>>>>
>>>> There are a couple of things that could improve the performance,
>> although
>>>> the write to the secondary index would most likely lag under heavy load.
>>>>
>>>>
>>>> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com> wrote:
>>>>
>>>>> John,
>>>>>
>>>>> My 2 cents:
>>>>> I tried implementing Secondary Index by using Region Observers on Put.
>> It
>>>>> works well under low load. But, under heavy load the RO could not keep
>> up
>>>>> with load cross region server writes.
>>>>> Then, i decided not to use RO as per Andrew's explanation and I moved
>>>> all
>>>>> the logic of building secondary index tables on my HBase Client . Since
>>>>> then, the system has been running fine under heavy load.
>>>>> IMO, if you will use RO and do cross RS read/write then perhaps this
>> will
>>>>> become your bottleneck in HBase.
>>>>> Is it possible for you to avoid RO and control the writes/updates from
>>>>> client side?
>>>>>
>>>>> Thanks,
>>>>> Anil Gupta
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
>>>>> john.weatherford@telescope.tv> wrote:
>>>>>
>>>>>> OP Here :)
>>>>>>
>>>>>> Our current design involves a Region Observer on a table that does
>>>>>> increments on a second table. We took the approach that Michael said
>> and
>>>>>> inside the RO, we got a new connection and everything. We believe this
>>>> is
>>>>>> causing deadlocks for us. Our next attempt is going to be writing to
>>>>>> another row in the same table where we will store the increments. If
>>>> this
>>>>>> doesn't work, we are going to simply pull the increments out of the RO
>>>> and
>>>>>> do them in the application or in Flume.
>>>>>>
>>>>>> @Tom Brown
>>>>>> I would be very interested to hear more about your solution of
>>>>>> aggregating the increments in another system that is then responsible
>>>> for
>>>>>> updating in Hbase.
>>>>>>
>>>>>> -jW
>>>>>>
>>>>>>
>>>>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>>>>>>
>>>>>>> With respect to the OP's design… does the deadlock occur because he's
>>>>>>>>> trying to update a column in a different row within the same table?
>>>>>>>>>
>>>>>>>>
>>>>>>> Because he is trying to update *row* in a different Region (and
>>>>>>> potentially in different RS).
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Vladimir Rodionov
>>>>>>> Principal Platform Engineer
>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>
>>>>>>> ______________________________**__________
>>>>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
>>>>>>> Sent: Friday, October 11, 2013 9:10 AM
>>>>>>> To: user@hbase.apache.org
>>>>>>> Cc: Vladimir Rodionov
>>>>>>> Subject: Re: Coprocessor Increments
>>>>>>>
>>>>>>>
>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>> including any attachments hereto, may be confidential and is intended
>>>> to be
>>>>>>> read only by the individual or entity to whom this message is
>>>> addressed. If
>>>>>>> the reader of this message is not the intended recipient or an agent
>> or
>>>>>>> designee of the intended recipient, please note that any review, use,
>>>>>>> disclosure or distribution of this message or its attachments, in any
>>>> form,
>>>>>>> is strictly prohibited. If you have received this message in error,
>>>> please
>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anil Gupta
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: Coprocessor Increments
Posted by anil gupta <an...@gmail.com>.
Inline.
On Mon, Oct 14, 2013 at 7:50 AM, Michael Segel <ms...@hotmail.com>wrote:
> Anil,
>
> I wasn't suggesting that you can't do what you're doing, but you end up
> running in to the risks which coprocessors are supposed to remove. The
> standard YMMV always applies.
>
Agree with you. But, as per my knowledge and experience with coprocessors,
they are meant to be used for operations that are local to RS. Otherwise,
you are in danger of running into deadlocks, scalability issues.
>
> You have a cluster… another team in your company wants to use the cluster.
> So instead of the cluster being a single resource for your app/team, it now
> becomes a shared resource. So now you have people accessing HBase for
> multiple apps.
>
Well, its a separation of responsibility in this case. We don't want teams
to step each other toes and at the same time work well as an ecosystem.
Rule: Other teams can use same cluster. But they cannot write directly into
the tables that we own/control. If they want to write into our tables then
they have to use our HBase Client.
>
> You could then run multiple HBase HMasters with different locations for
> files, however… this can get messy.
> HOYA seems to suggest this as the future. If so, then you have to wonder
> about data locality.
>
HOYA is not even in beta at present. So, right now we are not thinking
about it.
>
> Having your app update the primary table and then the secondary index is
> always a good fallback, however you need to ensure that you understand the
> risks.
>
Agree, i understand that there is risk. But, you have to bite the bullet
when you are doing something that is not supported out of the box. We also
use CP's wherever they are appropriate(like HBASE-7474).
>
> With respect to secondary indexes… if you decouple the writes… you can get
> better throughput. Note that the code becomes a bit more complex because
> you're going to have to introduce a couple of different things. But thats
> something for a different discussion…
>
Whether to use CP or not, depends on the use case. In my opinion, CP's are
really powerful and an awesome feature in HBase. But, sometimes if not used
properly(like creating a Cyclic Graph as per Tom's example), they might be
problematic.
>
> On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com> wrote:
>
> > Inline.
> >
> > On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <
> msegel_hadoop@hotmail.com>wrote:
> >
> >> Ok…
> >>
> >> Sure you can have your app update the secondary index table.
> >> The only issue with that is if someone updates the base table outside of
> >> your app,
> >> they may or may not increment the secondary index.
> >>
> > Anil: We dont allow people to write data into HBase from their own HBase
> > client. We control the writes into HBase. So, we dont have the problem of
> > secondary index not getting written.
> > For example, If you expose a restful web service you can easily control
> the
> > writes to HBase. Even, if user requests to write one row in "main table",
> > you application can have the logic to writing in "Secondary index"
> tables.
> > In this way, it is transparent to users also. You can add/remove seconday
> > indexes as you want.
> >
> >> Note that your secondary index doesn't have to be an inverted table, but
> >> could be SOLR, LUCENE or something else.
> >>
> > Anil:As of now, we are happy with Inverted tables as they fit to our use
> > case.
> >
> >>
> >> So you really want to secondary indexes on the server.
> >>
> >> There are a couple of things that could improve the performance,
> although
> >> the write to the secondary index would most likely lag under heavy load.
> >>
> >>
> >> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com> wrote:
> >>
> >>> John,
> >>>
> >>> My 2 cents:
> >>> I tried implementing Secondary Index by using Region Observers on Put.
> It
> >>> works well under low load. But, under heavy load the RO could not keep
> up
> >>> with load cross region server writes.
> >>> Then, i decided not to use RO as per Andrew's explanation and I moved
> >> all
> >>> the logic of building secondary index tables on my HBase Client . Since
> >>> then, the system has been running fine under heavy load.
> >>> IMO, if you will use RO and do cross RS read/write then perhaps this
> will
> >>> become your bottleneck in HBase.
> >>> Is it possible for you to avoid RO and control the writes/updates from
> >>> client side?
> >>>
> >>> Thanks,
> >>> Anil Gupta
> >>>
> >>>
> >>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> >>> john.weatherford@telescope.tv> wrote:
> >>>
> >>>> OP Here :)
> >>>>
> >>>> Our current design involves a Region Observer on a table that does
> >>>> increments on a second table. We took the approach that Michael said
> and
> >>>> inside the RO, we got a new connection and everything. We believe this
> >> is
> >>>> causing deadlocks for us. Our next attempt is going to be writing to
> >>>> another row in the same table where we will store the increments. If
> >> this
> >>>> doesn't work, we are going to simply pull the increments out of the RO
> >> and
> >>>> do them in the application or in Flume.
> >>>>
> >>>> @Tom Brown
> >>>> I would be very interested to hear more about your solution of
> >>>> aggregating the increments in another system that is then responsible
> >> for
> >>>> updating in Hbase.
> >>>>
> >>>> -jW
> >>>>
> >>>>
> >>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
> >>>>
> >>>>> With respect to the OP's design… does the deadlock occur because he's
> >>>>>>> trying to update a column in a different row within the same table?
> >>>>>>>
> >>>>>>
> >>>>> Because he is trying to update *row* in a different Region (and
> >>>>> potentially in different RS).
> >>>>>
> >>>>> Best regards,
> >>>>> Vladimir Rodionov
> >>>>> Principal Platform Engineer
> >>>>> Carrier IQ, www.carrieriq.com
> >>>>> e-mail: vrodionov@carrieriq.com
> >>>>>
> >>>>> ______________________________**__________
> >>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
> >>>>> Sent: Friday, October 11, 2013 9:10 AM
> >>>>> To: user@hbase.apache.org
> >>>>> Cc: Vladimir Rodionov
> >>>>> Subject: Re: Coprocessor Increments
> >>>>>
> >>>>>
> >>>>> Confidentiality Notice: The information contained in this message,
> >>>>> including any attachments hereto, may be confidential and is intended
> >> to be
> >>>>> read only by the individual or entity to whom this message is
> >> addressed. If
> >>>>> the reader of this message is not the intended recipient or an agent
> or
> >>>>> designee of the intended recipient, please note that any review, use,
> >>>>> disclosure or distribution of this message or its attachments, in any
> >> form,
> >>>>> is strictly prohibited. If you have received this message in error,
> >> please
> >>>>> immediately notify the sender and/or Notifications@carrieriq.com and
> >>>>> delete or destroy any copy of this message and its attachments.
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Anil Gupta
> >>
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
--
Thanks & Regards,
Anil Gupta
Re: Coprocessor Increments
Posted by Tom Brown <to...@gmail.com>.
If you allow coprocessors to connect to any other server the connections
between the HBase nodes can be represented as a directed graph. This means
that deadlock is possible (when N1 is blocked waiting for N2, and N2 is
blocked waiting for N1). In order to guarantee that no deadlock will occur,
you have to be able to represent the connections as a directed acyclic
graph. And to do this, you cannot have nodes connecting directly to each
other.
You could also achieve this by carefully assigning your regions. If the
primary table (PT) is located exclusively on nodes N1..N3, and the
secondary/index table is located exclusively on nodes N4..N6, (assuming
updates to the secondary/index table never change the PT) the connections
would remain a DAG and deadlock would be impossible.
You're right that by putting the business logic into the coprocessor you
gain the ability to easily allow any group to access your cluster. But that
access isn't free. To use a SQL analogy: large organizations always protect
their SQL servers with a DBA. They do this because the potential downsides
of allowing unsupervised and unstructured access are too great.
YMMV
--Tom
On Mon, Oct 14, 2013 at 8:50 AM, Michael Segel <ms...@hotmail.com>wrote:
> Anil,
>
> I wasn't suggesting that you can't do what you're doing, but you end up
> running in to the risks which coprocessors are supposed to remove. The
> standard YMMV always applies.
>
> You have a cluster… another team in your company wants to use the cluster.
> So instead of the cluster being a single resource for your app/team, it now
> becomes a shared resource. So now you have people accessing HBase for
> multiple apps.
>
> You could then run multiple HBase HMasters with different locations for
> files, however… this can get messy.
> HOYA seems to suggest this as the future. If so, then you have to wonder
> about data locality.
>
> Having your app update the primary table and then the secondary index is
> always a good fallback, however you need to ensure that you understand the
> risks.
>
> With respect to secondary indexes… if you decouple the writes… you can get
> better throughput. Note that the code becomes a bit more complex because
> you're going to have to introduce a couple of different things. But thats
> something for a different discussion…
>
>
> On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com> wrote:
>
> > Inline.
> >
> > On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <
> msegel_hadoop@hotmail.com>wrote:
> >
> >> Ok…
> >>
> >> Sure you can have your app update the secondary index table.
> >> The only issue with that is if someone updates the base table outside of
> >> your app,
> >> they may or may not increment the secondary index.
> >>
> > Anil: We dont allow people to write data into HBase from their own HBase
> > client. We control the writes into HBase. So, we dont have the problem of
> > secondary index not getting written.
> > For example, If you expose a restful web service you can easily control
> the
> > writes to HBase. Even, if user requests to write one row in "main table",
> > you application can have the logic to writing in "Secondary index"
> tables.
> > In this way, it is transparent to users also. You can add/remove seconday
> > indexes as you want.
> >
> >> Note that your secondary index doesn't have to be an inverted table, but
> >> could be SOLR, LUCENE or something else.
> >>
> > Anil:As of now, we are happy with Inverted tables as they fit to our use
> > case.
> >
> >>
> >> So you really want to secondary indexes on the server.
> >>
> >> There are a couple of things that could improve the performance,
> although
> >> the write to the secondary index would most likely lag under heavy load.
> >>
> >>
> >> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com> wrote:
> >>
> >>> John,
> >>>
> >>> My 2 cents:
> >>> I tried implementing Secondary Index by using Region Observers on Put.
> It
> >>> works well under low load. But, under heavy load the RO could not keep
> up
> >>> with load cross region server writes.
> >>> Then, i decided not to use RO as per Andrew's explanation and I moved
> >> all
> >>> the logic of building secondary index tables on my HBase Client . Since
> >>> then, the system has been running fine under heavy load.
> >>> IMO, if you will use RO and do cross RS read/write then perhaps this
> will
> >>> become your bottleneck in HBase.
> >>> Is it possible for you to avoid RO and control the writes/updates from
> >>> client side?
> >>>
> >>> Thanks,
> >>> Anil Gupta
> >>>
> >>>
> >>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> >>> john.weatherford@telescope.tv> wrote:
> >>>
> >>>> OP Here :)
> >>>>
> >>>> Our current design involves a Region Observer on a table that does
> >>>> increments on a second table. We took the approach that Michael said
> and
> >>>> inside the RO, we got a new connection and everything. We believe this
> >> is
> >>>> causing deadlocks for us. Our next attempt is going to be writing to
> >>>> another row in the same table where we will store the increments. If
> >> this
> >>>> doesn't work, we are going to simply pull the increments out of the RO
> >> and
> >>>> do them in the application or in Flume.
> >>>>
> >>>> @Tom Brown
> >>>> I would be very interested to hear more about your solution of
> >>>> aggregating the increments in another system that is then responsible
> >> for
> >>>> updating in Hbase.
> >>>>
> >>>> -jW
> >>>>
> >>>>
> >>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
> >>>>
> >>>>> With respect to the OP's design… does the deadlock occur because he's
> >>>>>>> trying to update a column in a different row within the same table?
> >>>>>>>
> >>>>>>
> >>>>> Because he is trying to update *row* in a different Region (and
> >>>>> potentially in different RS).
> >>>>>
> >>>>> Best regards,
> >>>>> Vladimir Rodionov
> >>>>> Principal Platform Engineer
> >>>>> Carrier IQ, www.carrieriq.com
> >>>>> e-mail: vrodionov@carrieriq.com
> >>>>>
> >>>>> ______________________________**__________
> >>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
> >>>>> Sent: Friday, October 11, 2013 9:10 AM
> >>>>> To: user@hbase.apache.org
> >>>>> Cc: Vladimir Rodionov
> >>>>> Subject: Re: Coprocessor Increments
> >>>>>
> >>>>>
> >>>>> Confidentiality Notice: The information contained in this message,
> >>>>> including any attachments hereto, may be confidential and is intended
> >> to be
> >>>>> read only by the individual or entity to whom this message is
> >> addressed. If
> >>>>> the reader of this message is not the intended recipient or an agent
> or
> >>>>> designee of the intended recipient, please note that any review, use,
> >>>>> disclosure or distribution of this message or its attachments, in any
> >> form,
> >>>>> is strictly prohibited. If you have received this message in error,
> >> please
> >>>>> immediately notify the sender and/or Notifications@carrieriq.com and
> >>>>> delete or destroy any copy of this message and its attachments.
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Anil Gupta
> >>
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
Anil,
I wasn't suggesting that you can't do what you're doing, but you end up running in to the risks which coprocessors are supposed to remove. The standard YMMV always applies.
You have a cluster… another team in your company wants to use the cluster. So instead of the cluster being a single resource for your app/team, it now becomes a shared resource. So now you have people accessing HBase for multiple apps.
You could then run multiple HBase HMasters with different locations for files, however… this can get messy.
HOYA seems to suggest this as the future. If so, then you have to wonder about data locality.
Having your app update the primary table and then the secondary index is always a good fallback, however you need to ensure that you understand the risks.
With respect to secondary indexes… if you decouple the writes… you can get better throughput. Note that the code becomes a bit more complex because you're going to have to introduce a couple of different things. But thats something for a different discussion…
On Oct 13, 2013, at 10:15 AM, anil gupta <an...@gmail.com> wrote:
> Inline.
>
> On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <ms...@hotmail.com>wrote:
>
>> Ok…
>>
>> Sure you can have your app update the secondary index table.
>> The only issue with that is if someone updates the base table outside of
>> your app,
>> they may or may not increment the secondary index.
>>
> Anil: We dont allow people to write data into HBase from their own HBase
> client. We control the writes into HBase. So, we dont have the problem of
> secondary index not getting written.
> For example, If you expose a restful web service you can easily control the
> writes to HBase. Even, if user requests to write one row in "main table",
> you application can have the logic to writing in "Secondary index" tables.
> In this way, it is transparent to users also. You can add/remove seconday
> indexes as you want.
>
>> Note that your secondary index doesn't have to be an inverted table, but
>> could be SOLR, LUCENE or something else.
>>
> Anil:As of now, we are happy with Inverted tables as they fit to our use
> case.
>
>>
>> So you really want to secondary indexes on the server.
>>
>> There are a couple of things that could improve the performance, although
>> the write to the secondary index would most likely lag under heavy load.
>>
>>
>> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com> wrote:
>>
>>> John,
>>>
>>> My 2 cents:
>>> I tried implementing Secondary Index by using Region Observers on Put. It
>>> works well under low load. But, under heavy load the RO could not keep up
>>> with load cross region server writes.
>>> Then, i decided not to use RO as per Andrew's explanation and I moved
>> all
>>> the logic of building secondary index tables on my HBase Client . Since
>>> then, the system has been running fine under heavy load.
>>> IMO, if you will use RO and do cross RS read/write then perhaps this will
>>> become your bottleneck in HBase.
>>> Is it possible for you to avoid RO and control the writes/updates from
>>> client side?
>>>
>>> Thanks,
>>> Anil Gupta
>>>
>>>
>>> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
>>> john.weatherford@telescope.tv> wrote:
>>>
>>>> OP Here :)
>>>>
>>>> Our current design involves a Region Observer on a table that does
>>>> increments on a second table. We took the approach that Michael said and
>>>> inside the RO, we got a new connection and everything. We believe this
>> is
>>>> causing deadlocks for us. Our next attempt is going to be writing to
>>>> another row in the same table where we will store the increments. If
>> this
>>>> doesn't work, we are going to simply pull the increments out of the RO
>> and
>>>> do them in the application or in Flume.
>>>>
>>>> @Tom Brown
>>>> I would be very interested to hear more about your solution of
>>>> aggregating the increments in another system that is then responsible
>> for
>>>> updating in Hbase.
>>>>
>>>> -jW
>>>>
>>>>
>>>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>>>>
>>>>> With respect to the OP's design… does the deadlock occur because he's
>>>>>>> trying to update a column in a different row within the same table?
>>>>>>>
>>>>>>
>>>>> Because he is trying to update *row* in a different Region (and
>>>>> potentially in different RS).
>>>>>
>>>>> Best regards,
>>>>> Vladimir Rodionov
>>>>> Principal Platform Engineer
>>>>> Carrier IQ, www.carrieriq.com
>>>>> e-mail: vrodionov@carrieriq.com
>>>>>
>>>>> ______________________________**__________
>>>>> From: Michael Segel [msegel_hadoop@hotmail.com]
>>>>> Sent: Friday, October 11, 2013 9:10 AM
>>>>> To: user@hbase.apache.org
>>>>> Cc: Vladimir Rodionov
>>>>> Subject: Re: Coprocessor Increments
>>>>>
>>>>>
>>>>> Confidentiality Notice: The information contained in this message,
>>>>> including any attachments hereto, may be confidential and is intended
>> to be
>>>>> read only by the individual or entity to whom this message is
>> addressed. If
>>>>> the reader of this message is not the intended recipient or an agent or
>>>>> designee of the intended recipient, please note that any review, use,
>>>>> disclosure or distribution of this message or its attachments, in any
>> form,
>>>>> is strictly prohibited. If you have received this message in error,
>> please
>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>> delete or destroy any copy of this message and its attachments.
>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: Coprocessor Increments
Posted by anil gupta <an...@gmail.com>.
Inline.
On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <ms...@hotmail.com>wrote:
> Ok…
>
> Sure you can have your app update the secondary index table.
> The only issue with that is if someone updates the base table outside of
> your app,
> they may or may not increment the secondary index.
>
Anil: We dont allow people to write data into HBase from their own HBase
client. We control the writes into HBase. So, we dont have the problem of
secondary index not getting written.
For example, If you expose a restful web service you can easily control the
writes to HBase. Even, if user requests to write one row in "main table",
you application can have the logic to writing in "Secondary index" tables.
In this way, it is transparent to users also. You can add/remove seconday
indexes as you want.
> Note that your secondary index doesn't have to be an inverted table, but
> could be SOLR, LUCENE or something else.
>
Anil:As of now, we are happy with Inverted tables as they fit to our use
case.
>
> So you really want to secondary indexes on the server.
>
> There are a couple of things that could improve the performance, although
> the write to the secondary index would most likely lag under heavy load.
>
>
> On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com> wrote:
>
> > John,
> >
> > My 2 cents:
> > I tried implementing Secondary Index by using Region Observers on Put. It
> > works well under low load. But, under heavy load the RO could not keep up
> > with load cross region server writes.
> > Then, i decided not to use RO as per Andrew's explanation and I moved
> all
> > the logic of building secondary index tables on my HBase Client . Since
> > then, the system has been running fine under heavy load.
> > IMO, if you will use RO and do cross RS read/write then perhaps this will
> > become your bottleneck in HBase.
> > Is it possible for you to avoid RO and control the writes/updates from
> > client side?
> >
> > Thanks,
> > Anil Gupta
> >
> >
> > On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> > john.weatherford@telescope.tv> wrote:
> >
> >> OP Here :)
> >>
> >> Our current design involves a Region Observer on a table that does
> >> increments on a second table. We took the approach that Michael said and
> >> inside the RO, we got a new connection and everything. We believe this
> is
> >> causing deadlocks for us. Our next attempt is going to be writing to
> >> another row in the same table where we will store the increments. If
> this
> >> doesn't work, we are going to simply pull the increments out of the RO
> and
> >> do them in the application or in Flume.
> >>
> >> @Tom Brown
> >> I would be very interested to hear more about your solution of
> >> aggregating the increments in another system that is then responsible
> for
> >> updating in Hbase.
> >>
> >> -jW
> >>
> >>
> >> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
> >>
> >>> With respect to the OP's design… does the deadlock occur because he's
> >>>>> trying to update a column in a different row within the same table?
> >>>>>
> >>>>
> >>> Because he is trying to update *row* in a different Region (and
> >>> potentially in different RS).
> >>>
> >>> Best regards,
> >>> Vladimir Rodionov
> >>> Principal Platform Engineer
> >>> Carrier IQ, www.carrieriq.com
> >>> e-mail: vrodionov@carrieriq.com
> >>>
> >>> ______________________________**__________
> >>> From: Michael Segel [msegel_hadoop@hotmail.com]
> >>> Sent: Friday, October 11, 2013 9:10 AM
> >>> To: user@hbase.apache.org
> >>> Cc: Vladimir Rodionov
> >>> Subject: Re: Coprocessor Increments
> >>>
> >>>
> >>> Confidentiality Notice: The information contained in this message,
> >>> including any attachments hereto, may be confidential and is intended
> to be
> >>> read only by the individual or entity to whom this message is
> addressed. If
> >>> the reader of this message is not the intended recipient or an agent or
> >>> designee of the intended recipient, please note that any review, use,
> >>> disclosure or distribution of this message or its attachments, in any
> form,
> >>> is strictly prohibited. If you have received this message in error,
> please
> >>> immediately notify the sender and/or Notifications@carrieriq.com and
> >>> delete or destroy any copy of this message and its attachments.
> >>>
> >>
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
>
--
Thanks & Regards,
Anil Gupta
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
Ok…
Sure you can have your app update the secondary index table.
The only issue with that is if someone updates the base table outside of your app,
they may or may not increment the secondary index.
Note that your secondary index doesn't have to be an inverted table, but could be SOLR, LUCENE or something else.
So you really want to secondary indexes on the server.
There are a couple of things that could improve the performance, although the write to the secondary index would most likely lag under heavy load.
On Oct 12, 2013, at 11:27 PM, anil gupta <an...@gmail.com> wrote:
> John,
>
> My 2 cents:
> I tried implementing Secondary Index by using Region Observers on Put. It
> works well under low load. But, under heavy load the RO could not keep up
> with load cross region server writes.
> Then, i decided not to use RO as per Andrew's explanation and I moved all
> the logic of building secondary index tables on my HBase Client . Since
> then, the system has been running fine under heavy load.
> IMO, if you will use RO and do cross RS read/write then perhaps this will
> become your bottleneck in HBase.
> Is it possible for you to avoid RO and control the writes/updates from
> client side?
>
> Thanks,
> Anil Gupta
>
>
> On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> john.weatherford@telescope.tv> wrote:
>
>> OP Here :)
>>
>> Our current design involves a Region Observer on a table that does
>> increments on a second table. We took the approach that Michael said and
>> inside the RO, we got a new connection and everything. We believe this is
>> causing deadlocks for us. Our next attempt is going to be writing to
>> another row in the same table where we will store the increments. If this
>> doesn't work, we are going to simply pull the increments out of the RO and
>> do them in the application or in Flume.
>>
>> @Tom Brown
>> I would be very interested to hear more about your solution of
>> aggregating the increments in another system that is then responsible for
>> updating in Hbase.
>>
>> -jW
>>
>>
>> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>>
>>> With respect to the OP's design… does the deadlock occur because he's
>>>>> trying to update a column in a different row within the same table?
>>>>>
>>>>
>>> Because he is trying to update *row* in a different Region (and
>>> potentially in different RS).
>>>
>>> Best regards,
>>> Vladimir Rodionov
>>> Principal Platform Engineer
>>> Carrier IQ, www.carrieriq.com
>>> e-mail: vrodionov@carrieriq.com
>>>
>>> ______________________________**__________
>>> From: Michael Segel [msegel_hadoop@hotmail.com]
>>> Sent: Friday, October 11, 2013 9:10 AM
>>> To: user@hbase.apache.org
>>> Cc: Vladimir Rodionov
>>> Subject: Re: Coprocessor Increments
>>>
>>>
>>> Confidentiality Notice: The information contained in this message,
>>> including any attachments hereto, may be confidential and is intended to be
>>> read only by the individual or entity to whom this message is addressed. If
>>> the reader of this message is not the intended recipient or an agent or
>>> designee of the intended recipient, please note that any review, use,
>>> disclosure or distribution of this message or its attachments, in any form,
>>> is strictly prohibited. If you have received this message in error, please
>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>> delete or destroy any copy of this message and its attachments.
>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: Coprocessor Increments
Posted by anil gupta <an...@gmail.com>.
John,
My 2 cents:
I tried implementing Secondary Index by using Region Observers on Put. It
works well under low load. But, under heavy load the RO could not keep up
with load cross region server writes.
Then, i decided not to use RO as per Andrew's explanation and I moved all
the logic of building secondary index tables on my HBase Client . Since
then, the system has been running fine under heavy load.
IMO, if you will use RO and do cross RS read/write then perhaps this will
become your bottleneck in HBase.
Is it possible for you to avoid RO and control the writes/updates from
client side?
Thanks,
Anil Gupta
On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
john.weatherford@telescope.tv> wrote:
> OP Here :)
>
> Our current design involves a Region Observer on a table that does
> increments on a second table. We took the approach that Michael said and
> inside the RO, we got a new connection and everything. We believe this is
> causing deadlocks for us. Our next attempt is going to be writing to
> another row in the same table where we will store the increments. If this
> doesn't work, we are going to simply pull the increments out of the RO and
> do them in the application or in Flume.
>
> @Tom Brown
> I would be very interested to hear more about your solution of
> aggregating the increments in another system that is then responsible for
> updating in Hbase.
>
> -jW
>
>
> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>
>> With respect to the OP's design… does the deadlock occur because he's
>>>> trying to update a column in a different row within the same table?
>>>>
>>>
>> Because he is trying to update *row* in a different Region (and
>> potentially in different RS).
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>>
>> ______________________________**__________
>> From: Michael Segel [msegel_hadoop@hotmail.com]
>> Sent: Friday, October 11, 2013 9:10 AM
>> To: user@hbase.apache.org
>> Cc: Vladimir Rodionov
>> Subject: Re: Coprocessor Increments
>>
>>
>> Confidentiality Notice: The information contained in this message,
>> including any attachments hereto, may be confidential and is intended to be
>> read only by the individual or entity to whom this message is addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any form,
>> is strictly prohibited. If you have received this message in error, please
>> immediately notify the sender and/or Notifications@carrieriq.com and
>> delete or destroy any copy of this message and its attachments.
>>
>
--
Thanks & Regards,
Anil Gupta
Re: Coprocessor Increments
Posted by John Weatherford <jo...@telescope.tv>.
OP Here :)
Our current design involves a Region Observer on a table that does
increments on a second table. We took the approach that Michael said
and inside the RO, we got a new connection and everything. We believe
this is causing deadlocks for us. Our next attempt is going to be
writing to another row in the same table where we will store the
increments. If this doesn't work, we are going to simply pull the
increments out of the RO and do them in the application or in Flume.
@Tom Brown
I would be very interested to hear more about your solution of
aggregating the increments in another system that is then responsible
for updating in Hbase.
-jW
On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
>>> With respect to the OP's design… does the deadlock occur because he's trying to update a column in a different row within the same table?
>
> Because he is trying to update *row* in a different Region (and potentially in different RS).
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Michael Segel [msegel_hadoop@hotmail.com]
> Sent: Friday, October 11, 2013 9:10 AM
> To: user@hbase.apache.org
> Cc: Vladimir Rodionov
> Subject: Re: Coprocessor Increments
>
>
> Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: Coprocessor Increments
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
>>With respect to the OP's design… does the deadlock occur because he's trying to update a column in a different row within the same table?
Because he is trying to update *row* in a different Region (and potentially in different RS).
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Michael Segel [msegel_hadoop@hotmail.com]
Sent: Friday, October 11, 2013 9:10 AM
To: user@hbase.apache.org
Cc: Vladimir Rodionov
Subject: Re: Coprocessor Increments
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
ok…
A couple of things…
First, it looks like when tuning HBase, you're not accounting for the additional overhead caused by unknown coprocessors. I guess even if you take a SWAG, you could end up still running out of resources…
Second… I'm a little confused.
In the OP's problem… is he trying to increment a counter in the same table, or is his counter in a different table?
This would then become a discussion of schema design.
Next, I would have to agree with you about the need to refactor HBase code. Reviewing and refactoring is always a good thing if you have time. ;-)
With respect to the OP's design… does the deadlock occur because he's trying to update a column in a different row within the same table?
The reason I ask is that if I were to implement an RO that updates a row in a second table, upon instantiation, i would create an HTable object referencing the second table and then use that to handle the data. Am I wrong to assume that in doing so, that I wouldn't be using the internal RPC threads and that the update would look like an external client?
Since my mind is pretty much mush by now… (long week, no sleep), is it better to accept the higher overhead by making the RO a client itself, or should I risk the potential of deadlock?
So what am I missing?
Thx
On Oct 10, 2013, at 6:52 PM, Vladimir Rodionov <vr...@carrieriq.com> wrote:
> Michael,
>
> RS has 3 different pools of threads (Readers (accept RPC), Handlers (process requests), third pool (do not remember name) of threads send request results back to callers )
> All of them communicate to through internal Queues. Cumbersome and needs to be refactored, of course.
>
> All these pools are configurable (max threads). Readers and writers are fine - they will never get into any deadlock situation.
> Handlers are different. In my example, I described the situation when ALL handlers in RS1 are waiting on RPC calls to RS2 and
> ALL handlers in RS2 are waiting on RPC calls to RS2. For the sake of simplicity:
>
> RS1 and RS2 - two region servers
> handlers count = 1 - just one thread in a pool.
>
> 1. RS2 coprocessor sends RPC to RS1. From single handler thread. Total available handlers in RS2 = 0.
> 2. RS1 receives request from RS2 CP2 (coprocessor).
> 3. RS1 handler1 receives request. Total available handlers in RS1 = 0;
> 4. RS1 coprocessor makes RPC call back to RS2
> 5. RS2 Reader thread places request into Handler pool queue, but there is no handlers available (see 1). Deadlock.
> and the request MUST timeout (fail) eventually. If it does not fail - there is the issue in HBase RPC.
>
> But technically, its a deadlock.
>
>
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Michael Segel [msegel_hadoop@hotmail.com]
> Sent: Thursday, October 10, 2013 2:57 PM
> To: user@hbase.apache.org
> Subject: Re: Coprocessor Increments
>
> Ok…
>
> So… I see you followed up to your earlier post.
>
> Lets walk through this to make sure we're on the same page.
>
> You put() row 1 in table 1.
> The post_put() wants to insert a value in table 2.
>
> Now suppose I have an htable pool of connections in my Region Observer.
> (Is this overkill? I'm treating the RO as if its a client connecting to HBase.)
>
> RO performs a put() into the second table.
>
> The RPC handlers are a Region server resource, yes?
>
> So I can always increase them from the default (10) … but the point is that I can have 10 clients updating table A and then have a couple of different regions on the RS for the table making RO requests.
>
> Is that the case?
>
>
> On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov <vr...@carrieriq.com> wrote:
>
>> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>>
>> Each RPC call is served by a thread from RPC-handlers pool (which is 10? by default).
>>
>> Making RPC call from within handler's therad is:
>>
>> A. expensive
>> B. may result in some deadlock -type of situations when no more incoming RPC calls can be accepted because
>> everyone is connected to everyone in a spaghetti way and waiting on RPC calls to complete
>>
>> Let us say we have 2 region servers for simplicity:
>> RS1 and RS2. Both have pool of handler threads = 10
>>
>> What happen when all 10 handlers in RS1 are trying to RPC RS2 and all 10 handlers are trying to RPC RS2?
>>
>> The same deadlock. Its all probabilistic .
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>>
>> ________________________________________
>> From: Vladimir Rodionov
>> Sent: Thursday, October 10, 2013 12:09 PM
>> To: user@hbase.apache.org
>> Subject: RE: Coprocessor Increments
>>
>> Classical deadlock :
>>
>> CP-Region1 updates counter in CP-Region2 (waits on RPC)
>> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>>
>> I think its an anti-pattern. Do not do cross region calls in region CP code.
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>>
>> ________________________________________
>> From: Michael Segel [msegel_hadoop@hotmail.com]
>> Sent: Thursday, October 10, 2013 9:55 AM
>> To: user@hbase.apache.org
>> Cc: Ted Yu
>> Subject: Re: Coprocessor Increments
>>
>> I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.
>>
>> Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
>>
>> But the key is to really understanding and confirming what's causing the Deadlock in detail.
>>
>> On Oct 10, 2013, at 11:15 AM, John Weatherford <jo...@telescope.tv> wrote:
>>
>>> Michael,
>>> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>>>
>>>
>>>
>>> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>>>> Can we just take a quick pause…
>>>>
>>>> John you wrote the following:
>>>> "We have been running into an RPC deadlock issue on HBase and from
>>>> investigation, we believe the root of the issue is in us doing cross
>>>> region increments from a coprocessor. After some further searching and
>>>> reading over this
>>>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
>>>> we think that we can solve this by doing the increments locally on the
>>>> region. "
>>>>
>>>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>>>
>>>> Can either you or Andrew explain in detail what is meant by the RPC
>>>> deadlock?
>>>>
>>>> This goes back to rethink how to implement coprocessors.
>>>>
>>>>
>>>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>>>> <john.weatherford@telescope.tv <ma...@telescope.tv>>
>>>> wrote:
>>>>
>>>>> Thank you ted. I might have to rethink my key design to accomodate
>>>>> this. With this design and using the totals in keys as you suggested,
>>>>> I would have to scan the entire "California" group to find the
>>>>> totals. Thank you very much for your help.
>>>>>
>>>>> -John
>>>>>
>>>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>>>> John:
>>>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>>>> 'California-**
>>>>>> 95424' is the start key of region 2. You can choose
>>>>>> 'California-**12346#total'
>>>>>> to be the row key where increment is done for region 1 and
>>>>>> 'California-**95424#total'
>>>>>> to be the row key where increment is done for region 2.
>>>>>>
>>>>>> w.r.t. verification of whether given row key is indeed inside the
>>>>>> hosting
>>>>>> region, see the following:
>>>>>>
>>>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>>>
>>>>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>>>>
>>>>>> throw new WrongRegionException("Requested row out of range for " +
>>>>>>
>>>>>> op + " on HRegion " + this + ", startKey='" +
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>>>> john.weatherford@telescope.tv
>>>>>> <ma...@telescope.tv>> wrote:
>>>>>>
>>>>>>> First, Thank you everyone for the quick response.
>>>>>>>
>>>>>>> Ted:
>>>>>>> The custom split policy could be an interesting solution.
>>>>>>> Regarding the
>>>>>>> other option you sent, if I pull the end row key, I could construct
>>>>>>> an end
>>>>>>> key, but if I suffix the row key with the end key of the region,
>>>>>>> would that
>>>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>>>> suffixed key
>>>>>>> now be "California-total-California-**12346" and the other region's
>>>>>>> counter would be "California-total-California-**95424" and both of
>>>>>>> these
>>>>>>> keys would actually end up on the second region since the sorting
>>>>>>> of the
>>>>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>>>>
>>>>>>> Part of the question is that I don't understand if calling
>>>>>>> incrementColumnValue() on the region will always execute on the region
>>>>>>> called, regardles of rowkey. If so, what happens when those regions are
>>>>>>> merged?
>>>>>>>
>>>>>>> Thanks again for the help!
>>>>>>>
>>>>>>>
>>>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>>>>
>>>>>>>> bq. 'California-total' row in each region
>>>>>>>>
>>>>>>>> Of course such row key needs to be suffixed with either start or
>>>>>>>> end key
>>>>>>>> of
>>>>>>>> the corresponding region.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>
>>>>>>>> John:
>>>>>>>>> Looks like you need a 'California-total' row in each region.
>>>>>>>>>
>>>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Contrived Example
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>>>>>
>>>>>>>>>>> Region 1.
>>>>>>>>>>
>>>>>>>>>> This would likely be on an insert on the same region. But as
>>>>>>>>>> the table
>>>>>>>>>>>> grows, this secondary insert could end up on another region.
>>>>>>>>>>>> If it is
>>>>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>>>>> which still
>>>>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>>>>>> region server? If so, what happens if these two regions are
>>>>>>>>>>>> compacted
>>>>>>>>>>>> into one?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>>>>>> split
>>>>>>>>>> is complete.
>>>>>>>>>>
>>>>>>>>>> Vladimir Rodionov
>>>>>>>>>> Principal Platform Engineer
>>>>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>>>>
>>>>>>>>>> ______________________________**__________
>>>>>>>>>>
>>>>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>>>>> including any attachments hereto, may be confidential and is
>>>>>>>>>> intended
>>>>>>>>>> to be
>>>>>>>>>> read only by the individual or entity to whom this message is
>>>>>>>>>> addressed. If
>>>>>>>>>> the reader of this message is not the intended recipient or an
>>>>>>>>>> agent or
>>>>>>>>>> designee of the intended recipient, please note that any review,
>>>>>>>>>> use,
>>>>>>>>>> disclosure or distribution of this message or its attachments,
>>>>>>>>>> in any
>>>>>>>>>> form,
>>>>>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>>>>>> please
>>>>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>> Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
>>
>
>
RE: Coprocessor Increments
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Michael,
RS has 3 different pools of threads (Readers (accept RPC), Handlers (process requests), third pool (do not remember name) of threads send request results back to callers )
All of them communicate to through internal Queues. Cumbersome and needs to be refactored, of course.
All these pools are configurable (max threads). Readers and writers are fine - they will never get into any deadlock situation.
Handlers are different. In my example, I described the situation when ALL handlers in RS1 are waiting on RPC calls to RS2 and
ALL handlers in RS2 are waiting on RPC calls to RS2. For the sake of simplicity:
RS1 and RS2 - two region servers
handlers count = 1 - just one thread in a pool.
1. RS2 coprocessor sends RPC to RS1. From single handler thread. Total available handlers in RS2 = 0.
2. RS1 receives request from RS2 CP2 (coprocessor).
3. RS1 handler1 receives request. Total available handlers in RS1 = 0;
4. RS1 coprocessor makes RPC call back to RS2
5. RS2 Reader thread places request into Handler pool queue, but there is no handlers available (see 1). Deadlock.
and the request MUST timeout (fail) eventually. If it does not fail - there is the issue in HBase RPC.
But technically, its a deadlock.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Michael Segel [msegel_hadoop@hotmail.com]
Sent: Thursday, October 10, 2013 2:57 PM
To: user@hbase.apache.org
Subject: Re: Coprocessor Increments
Ok…
So… I see you followed up to your earlier post.
Lets walk through this to make sure we're on the same page.
You put() row 1 in table 1.
The post_put() wants to insert a value in table 2.
Now suppose I have an htable pool of connections in my Region Observer.
(Is this overkill? I'm treating the RO as if its a client connecting to HBase.)
RO performs a put() into the second table.
The RPC handlers are a Region server resource, yes?
So I can always increase them from the default (10) … but the point is that I can have 10 clients updating table A and then have a couple of different regions on the RS for the table making RO requests.
Is that the case?
On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov <vr...@carrieriq.com> wrote:
> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>
> Each RPC call is served by a thread from RPC-handlers pool (which is 10? by default).
>
> Making RPC call from within handler's therad is:
>
> A. expensive
> B. may result in some deadlock -type of situations when no more incoming RPC calls can be accepted because
> everyone is connected to everyone in a spaghetti way and waiting on RPC calls to complete
>
> Let us say we have 2 region servers for simplicity:
> RS1 and RS2. Both have pool of handler threads = 10
>
> What happen when all 10 handlers in RS1 are trying to RPC RS2 and all 10 handlers are trying to RPC RS2?
>
> The same deadlock. Its all probabilistic .
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Thursday, October 10, 2013 12:09 PM
> To: user@hbase.apache.org
> Subject: RE: Coprocessor Increments
>
> Classical deadlock :
>
> CP-Region1 updates counter in CP-Region2 (waits on RPC)
> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>
> I think its an anti-pattern. Do not do cross region calls in region CP code.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Michael Segel [msegel_hadoop@hotmail.com]
> Sent: Thursday, October 10, 2013 9:55 AM
> To: user@hbase.apache.org
> Cc: Ted Yu
> Subject: Re: Coprocessor Increments
>
> I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.
>
> Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
>
> But the key is to really understanding and confirming what's causing the Deadlock in detail.
>
> On Oct 10, 2013, at 11:15 AM, John Weatherford <jo...@telescope.tv> wrote:
>
>> Michael,
>> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>>
>>
>>
>> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>>> Can we just take a quick pause…
>>>
>>> John you wrote the following:
>>> "We have been running into an RPC deadlock issue on HBase and from
>>> investigation, we believe the root of the issue is in us doing cross
>>> region increments from a coprocessor. After some further searching and
>>> reading over this
>>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
>>> we think that we can solve this by doing the increments locally on the
>>> region. "
>>>
>>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>>
>>> Can either you or Andrew explain in detail what is meant by the RPC
>>> deadlock?
>>>
>>> This goes back to rethink how to implement coprocessors.
>>>
>>>
>>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>>> <john.weatherford@telescope.tv <ma...@telescope.tv>>
>>> wrote:
>>>
>>>> Thank you ted. I might have to rethink my key design to accomodate
>>>> this. With this design and using the totals in keys as you suggested,
>>>> I would have to scan the entire "California" group to find the
>>>> totals. Thank you very much for your help.
>>>>
>>>> -John
>>>>
>>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>>> John:
>>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>>> 'California-**
>>>>> 95424' is the start key of region 2. You can choose
>>>>> 'California-**12346#total'
>>>>> to be the row key where increment is done for region 1 and
>>>>> 'California-**95424#total'
>>>>> to be the row key where increment is done for region 2.
>>>>>
>>>>> w.r.t. verification of whether given row key is indeed inside the
>>>>> hosting
>>>>> region, see the following:
>>>>>
>>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>>
>>>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>>>
>>>>> throw new WrongRegionException("Requested row out of range for " +
>>>>>
>>>>> op + " on HRegion " + this + ", startKey='" +
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>>> john.weatherford@telescope.tv
>>>>> <ma...@telescope.tv>> wrote:
>>>>>
>>>>>> First, Thank you everyone for the quick response.
>>>>>>
>>>>>> Ted:
>>>>>> The custom split policy could be an interesting solution.
>>>>>> Regarding the
>>>>>> other option you sent, if I pull the end row key, I could construct
>>>>>> an end
>>>>>> key, but if I suffix the row key with the end key of the region,
>>>>>> would that
>>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>>> suffixed key
>>>>>> now be "California-total-California-**12346" and the other region's
>>>>>> counter would be "California-total-California-**95424" and both of
>>>>>> these
>>>>>> keys would actually end up on the second region since the sorting
>>>>>> of the
>>>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>>>
>>>>>> Part of the question is that I don't understand if calling
>>>>>> incrementColumnValue() on the region will always execute on the region
>>>>>> called, regardles of rowkey. If so, what happens when those regions are
>>>>>> merged?
>>>>>>
>>>>>> Thanks again for the help!
>>>>>>
>>>>>>
>>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>>>
>>>>>>> bq. 'California-total' row in each region
>>>>>>>
>>>>>>> Of course such row key needs to be suffixed with either start or
>>>>>>> end key
>>>>>>> of
>>>>>>> the corresponding region.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>> John:
>>>>>>>> Looks like you need a 'California-total' row in each region.
>>>>>>>>
>>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>> Contrived Example
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>>>>
>>>>>>>>>> Region 1.
>>>>>>>>>
>>>>>>>>> This would likely be on an insert on the same region. But as
>>>>>>>>> the table
>>>>>>>>>>> grows, this secondary insert could end up on another region.
>>>>>>>>>>> If it is
>>>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>>>> which still
>>>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>>>>> region server? If so, what happens if these two regions are
>>>>>>>>>>> compacted
>>>>>>>>>>> into one?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>>>>> split
>>>>>>>>> is complete.
>>>>>>>>>
>>>>>>>>> Vladimir Rodionov
>>>>>>>>> Principal Platform Engineer
>>>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>>>
>>>>>>>>> ______________________________**__________
>>>>>>>>>
>>>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>>>> including any attachments hereto, may be confidential and is
>>>>>>>>> intended
>>>>>>>>> to be
>>>>>>>>> read only by the individual or entity to whom this message is
>>>>>>>>> addressed. If
>>>>>>>>> the reader of this message is not the intended recipient or an
>>>>>>>>> agent or
>>>>>>>>> designee of the intended recipient, please note that any review,
>>>>>>>>> use,
>>>>>>>>> disclosure or distribution of this message or its attachments,
>>>>>>>>> in any
>>>>>>>>> form,
>>>>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>>>>> please
>>>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
>
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
Ok…
So… I see you followed up to your earlier post.
Lets walk through this to make sure we're on the same page.
You put() row 1 in table 1.
The post_put() wants to insert a value in table 2.
Now suppose I have an htable pool of connections in my Region Observer.
(Is this overkill? I'm treating the RO as if its a client connecting to HBase.)
RO performs a put() into the second table.
The RPC handlers are a Region server resource, yes?
So I can always increase them from the default (10) … but the point is that I can have 10 clients updating table A and then have a couple of different regions on the RS for the table making RO requests.
Is that the case?
On Oct 10, 2013, at 2:23 PM, Vladimir Rodionov <vr...@carrieriq.com> wrote:
> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>
> Each RPC call is served by a thread from RPC-handlers pool (which is 10? by default).
>
> Making RPC call from within handler's therad is:
>
> A. expensive
> B. may result in some deadlock -type of situations when no more incoming RPC calls can be accepted because
> everyone is connected to everyone in a spaghetti way and waiting on RPC calls to complete
>
> Let us say we have 2 region servers for simplicity:
> RS1 and RS2. Both have pool of handler threads = 10
>
> What happen when all 10 handlers in RS1 are trying to RPC RS2 and all 10 handlers are trying to RPC RS2?
>
> The same deadlock. Its all probabilistic .
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Thursday, October 10, 2013 12:09 PM
> To: user@hbase.apache.org
> Subject: RE: Coprocessor Increments
>
> Classical deadlock :
>
> CP-Region1 updates counter in CP-Region2 (waits on RPC)
> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>
> I think its an anti-pattern. Do not do cross region calls in region CP code.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Michael Segel [msegel_hadoop@hotmail.com]
> Sent: Thursday, October 10, 2013 9:55 AM
> To: user@hbase.apache.org
> Cc: Ted Yu
> Subject: Re: Coprocessor Increments
>
> I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.
>
> Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
>
> But the key is to really understanding and confirming what's causing the Deadlock in detail.
>
> On Oct 10, 2013, at 11:15 AM, John Weatherford <jo...@telescope.tv> wrote:
>
>> Michael,
>> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>>
>>
>>
>> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>>> Can we just take a quick pause…
>>>
>>> John you wrote the following:
>>> "We have been running into an RPC deadlock issue on HBase and from
>>> investigation, we believe the root of the issue is in us doing cross
>>> region increments from a coprocessor. After some further searching and
>>> reading over this
>>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
>>> we think that we can solve this by doing the increments locally on the
>>> region. "
>>>
>>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>>
>>> Can either you or Andrew explain in detail what is meant by the RPC
>>> deadlock?
>>>
>>> This goes back to rethink how to implement coprocessors.
>>>
>>>
>>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>>> <john.weatherford@telescope.tv <ma...@telescope.tv>>
>>> wrote:
>>>
>>>> Thank you ted. I might have to rethink my key design to accomodate
>>>> this. With this design and using the totals in keys as you suggested,
>>>> I would have to scan the entire "California" group to find the
>>>> totals. Thank you very much for your help.
>>>>
>>>> -John
>>>>
>>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>>> John:
>>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>>> 'California-**
>>>>> 95424' is the start key of region 2. You can choose
>>>>> 'California-**12346#total'
>>>>> to be the row key where increment is done for region 1 and
>>>>> 'California-**95424#total'
>>>>> to be the row key where increment is done for region 2.
>>>>>
>>>>> w.r.t. verification of whether given row key is indeed inside the
>>>>> hosting
>>>>> region, see the following:
>>>>>
>>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>>
>>>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>>>
>>>>> throw new WrongRegionException("Requested row out of range for " +
>>>>>
>>>>> op + " on HRegion " + this + ", startKey='" +
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>>> john.weatherford@telescope.tv
>>>>> <ma...@telescope.tv>> wrote:
>>>>>
>>>>>> First, Thank you everyone for the quick response.
>>>>>>
>>>>>> Ted:
>>>>>> The custom split policy could be an interesting solution.
>>>>>> Regarding the
>>>>>> other option you sent, if I pull the end row key, I could construct
>>>>>> an end
>>>>>> key, but if I suffix the row key with the end key of the region,
>>>>>> would that
>>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>>> suffixed key
>>>>>> now be "California-total-California-**12346" and the other region's
>>>>>> counter would be "California-total-California-**95424" and both of
>>>>>> these
>>>>>> keys would actually end up on the second region since the sorting
>>>>>> of the
>>>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>>>
>>>>>> Part of the question is that I don't understand if calling
>>>>>> incrementColumnValue() on the region will always execute on the region
>>>>>> called, regardles of rowkey. If so, what happens when those regions are
>>>>>> merged?
>>>>>>
>>>>>> Thanks again for the help!
>>>>>>
>>>>>>
>>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>>>
>>>>>>> bq. 'California-total' row in each region
>>>>>>>
>>>>>>> Of course such row key needs to be suffixed with either start or
>>>>>>> end key
>>>>>>> of
>>>>>>> the corresponding region.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>> John:
>>>>>>>> Looks like you need a 'California-total' row in each region.
>>>>>>>>
>>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>> Contrived Example
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>>>>
>>>>>>>>>> Region 1.
>>>>>>>>>
>>>>>>>>> This would likely be on an insert on the same region. But as
>>>>>>>>> the table
>>>>>>>>>>> grows, this secondary insert could end up on another region.
>>>>>>>>>>> If it is
>>>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>>>> which still
>>>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>>>>> region server? If so, what happens if these two regions are
>>>>>>>>>>> compacted
>>>>>>>>>>> into one?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>>>>> split
>>>>>>>>> is complete.
>>>>>>>>>
>>>>>>>>> Vladimir Rodionov
>>>>>>>>> Principal Platform Engineer
>>>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>>>
>>>>>>>>> ______________________________**__________
>>>>>>>>>
>>>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>>>> including any attachments hereto, may be confidential and is
>>>>>>>>> intended
>>>>>>>>> to be
>>>>>>>>> read only by the individual or entity to whom this message is
>>>>>>>>> addressed. If
>>>>>>>>> the reader of this message is not the intended recipient or an
>>>>>>>>> agent or
>>>>>>>>> designee of the intended recipient, please note that any review,
>>>>>>>>> use,
>>>>>>>>> disclosure or distribution of this message or its attachments,
>>>>>>>>> in any
>>>>>>>>> form,
>>>>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>>>>> please
>>>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
>
> Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
>
Re: Coprocessor Increments
Posted by Tom Brown <to...@gmail.com>.
We used to do our updates through coprocessors, but it was not scalable. We
extracted the update code into a separate system, added row transaction
IDs, and haven't looked back.
For each incoming message, we compute the set of updates that message will
generate. With a batch of messages, we merge updates to the same row
together (for example, 3 increments of 1 each becomes 1 increment of 3).
This means fewer overall writes to HBase, and no risk of cross-regionserver
communication deadlocks.
--Tom
On Thu, Oct 10, 2013 at 1:23 PM, Vladimir Rodionov
<vr...@carrieriq.com>wrote:
> Nope. It is not so obvious, but definitely the anti-pattern is still there.
>
> Each RPC call is served by a thread from RPC-handlers pool (which is 10?
> by default).
>
> Making RPC call from within handler's therad is:
>
> A. expensive
> B. may result in some deadlock -type of situations when no more incoming
> RPC calls can be accepted because
> everyone is connected to everyone in a spaghetti way and waiting on RPC
> calls to complete
>
> Let us say we have 2 region servers for simplicity:
> RS1 and RS2. Both have pool of handler threads = 10
>
> What happen when all 10 handlers in RS1 are trying to RPC RS2 and all 10
> handlers are trying to RPC RS2?
>
> The same deadlock. Its all probabilistic .
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Vladimir Rodionov
> Sent: Thursday, October 10, 2013 12:09 PM
> To: user@hbase.apache.org
> Subject: RE: Coprocessor Increments
>
> Classical deadlock :
>
> CP-Region1 updates counter in CP-Region2 (waits on RPC)
> CP-Region2 updates counter in CP-Region1 (waits on RPC)
>
> I think its an anti-pattern. Do not do cross region calls in region CP
> code.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Michael Segel [msegel_hadoop@hotmail.com]
> Sent: Thursday, October 10, 2013 9:55 AM
> To: user@hbase.apache.org
> Cc: Ted Yu
> Subject: Re: Coprocessor Increments
>
> I think Andrew has a handle on it… my take is that you end up running out
> of resources to handle an RPC connection while within your coprocessor and
> you're waiting for a resource to be free and it can't because another
> coprocessor has an RPC resource and is also waiting for a free resource.
>
> Maybe its an over simplification, but if that's the case… you could always
> try thing to limit the RPC call, which would delay updating the counter.
> (Which may not be a problem) or redesign the coprocessors so that the
> coprocessors don't share the same RPC resources.
>
> But the key is to really understanding and confirming what's causing the
> Deadlock in detail.
>
> On Oct 10, 2013, at 11:15 AM, John Weatherford <
> john.weatherford@telescope.tv> wrote:
>
> > Michael,
> > I would also really like to know how this issue is caused also. I can't
> even give a solid way to reproduce our deadlock. It *seems* to happen more
> under load, but nothing can be proved yet. While google-ing and looking for
> an answer I came across that old message post (
> http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E).
> This seemed to line up with what we are doing, so we _hope_ this will be a
> fix for us, but we aren't entirely sure.
> >
> >
> >
> > On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
> >> Can we just take a quick pause…
> >>
> >> John you wrote the following:
> >> "We have been running into an RPC deadlock issue on HBase and from
> >> investigation, we believe the root of the issue is in us doing cross
> >> region increments from a coprocessor. After some further searching and
> >> reading over this
> >> <
> http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E
> >
> >> we think that we can solve this by doing the increments locally on the
> >> region. "
> >>
> >> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
> >>
> >> Can either you or Andrew explain in detail what is meant by the RPC
> >> deadlock?
> >>
> >> This goes back to rethink how to implement coprocessors.
> >>
> >>
> >> On Oct 9, 2013, at 11:03 PM, John Weatherford
> >> <john.weatherford@telescope.tv <ma...@telescope.tv>>
> >> wrote:
> >>
> >>> Thank you ted. I might have to rethink my key design to accomodate
> >>> this. With this design and using the totals in keys as you suggested,
> >>> I would have to scan the entire "California" group to find the
> >>> totals. Thank you very much for your help.
> >>>
> >>> -John
> >>>
> >>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
> >>>> John:
> >>>> Suppose 'California-**12346' is the start key of region 1 and
> >>>> 'California-**
> >>>> 95424' is the start key of region 2. You can choose
> >>>> 'California-**12346#total'
> >>>> to be the row key where increment is done for region 1 and
> >>>> 'California-**95424#total'
> >>>> to be the row key where increment is done for region 2.
> >>>>
> >>>> w.r.t. verification of whether given row key is indeed inside the
> >>>> hosting
> >>>> region, see the following:
> >>>>
> >>>> void checkRow(final byte [] row, String op) throws IOException {
> >>>>
> >>>> if (!rowIsInRange(getRegionInfo(), row)) {
> >>>>
> >>>> throw new WrongRegionException("Requested row out of range for " +
> >>>>
> >>>> op + " on HRegion " + this + ", startKey='" +
> >>>>
> >>>> Cheers
> >>>>
> >>>>
> >>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
> >>>> john.weatherford@telescope.tv
> >>>> <ma...@telescope.tv>> wrote:
> >>>>
> >>>>> First, Thank you everyone for the quick response.
> >>>>>
> >>>>> Ted:
> >>>>> The custom split policy could be an interesting solution.
> >>>>> Regarding the
> >>>>> other option you sent, if I pull the end row key, I could construct
> >>>>> an end
> >>>>> key, but if I suffix the row key with the end key of the region,
> >>>>> would that
> >>>>> actually solve my problem? In the contrived case, wouldn't the
> >>>>> suffixed key
> >>>>> now be "California-total-California-**12346" and the other region's
> >>>>> counter would be "California-total-California-**95424" and both of
> >>>>> these
> >>>>> keys would actually end up on the second region since the sorting
> >>>>> of the
> >>>>> keys would impose that "California-t*" comes after
> "California-[1-9]".
> >>>>>
> >>>>> Part of the question is that I don't understand if calling
> >>>>> incrementColumnValue() on the region will always execute on the
> region
> >>>>> called, regardles of rowkey. If so, what happens when those regions
> are
> >>>>> merged?
> >>>>>
> >>>>> Thanks again for the help!
> >>>>>
> >>>>>
> >>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
> >>>>>
> >>>>>> bq. 'California-total' row in each region
> >>>>>>
> >>>>>> Of course such row key needs to be suffixed with either start or
> >>>>>> end key
> >>>>>> of
> >>>>>> the corresponding region.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
> >>>>>> <ma...@gmail.com>> wrote:
> >>>>>>
> >>>>>> John:
> >>>>>>> Looks like you need a 'California-total' row in each region.
> >>>>>>>
> >>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
> >>>>>>> http://hbase.apache.org/book.**html#d0e6541<
> http://hbase.apache.org/book.html#d0e6541>
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
> >>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>
> >>>>>>> Contrived Example
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
> >>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all
> on
> >>>>>>>>>>
> >>>>>>>>> Region 1.
> >>>>>>>>
> >>>>>>>> This would likely be on an insert on the same region. But as
> >>>>>>>> the table
> >>>>>>>>>> grows, this secondary insert could end up on another region.
> >>>>>>>>>> If it is
> >>>>>>>>>> confined, then suppose we later insert "California-95424"
> >>>>>>>>>> which still
> >>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
> >>>>>>>>>> "California-total" all on Region 2.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Are we now left with two rowkeys of "California-total"? One on
> each
> >>>>>>>>>> region server? If so, what happens if these two regions are
> >>>>>>>>>> compacted
> >>>>>>>>>> into one?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Nope, Your "California-total" will migrate to Region 2 after
> region
> >>>>>>>> split
> >>>>>>>> is complete.
> >>>>>>>>
> >>>>>>>> Vladimir Rodionov
> >>>>>>>> Principal Platform Engineer
> >>>>>>>> Carrier IQ, www.carrieriq.com
> >>>>>>>> e-mail: vrodionov@carrieriq.com
> >>>>>>>>
> >>>>>>>> ______________________________**__________
> >>>>>>>>
> >>>>>>>> Confidentiality Notice: The information contained in this
> message,
> >>>>>>>> including any attachments hereto, may be confidential and is
> >>>>>>>> intended
> >>>>>>>> to be
> >>>>>>>> read only by the individual or entity to whom this message is
> >>>>>>>> addressed. If
> >>>>>>>> the reader of this message is not the intended recipient or an
> >>>>>>>> agent or
> >>>>>>>> designee of the intended recipient, please note that any review,
> >>>>>>>> use,
> >>>>>>>> disclosure or distribution of this message or its attachments,
> >>>>>>>> in any
> >>>>>>>> form,
> >>>>>>>> is strictly prohibited. If you have received this message in
> error,
> >>>>>>>> please
> >>>>>>>> immediately notify the sender and/or Notifications@carrieriq.comand
> >>>>>>>> delete or destroy any copy of this message and its attachments.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> >
>
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
RE: Coprocessor Increments
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Nope. It is not so obvious, but definitely the anti-pattern is still there.
Each RPC call is served by a thread from RPC-handlers pool (which is 10? by default).
Making RPC call from within handler's therad is:
A. expensive
B. may result in some deadlock -type of situations when no more incoming RPC calls can be accepted because
everyone is connected to everyone in a spaghetti way and waiting on RPC calls to complete
Let us say we have 2 region servers for simplicity:
RS1 and RS2. Both have pool of handler threads = 10
What happen when all 10 handlers in RS1 are trying to RPC RS2 and all 10 handlers are trying to RPC RS2?
The same deadlock. Its all probabilistic .
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Vladimir Rodionov
Sent: Thursday, October 10, 2013 12:09 PM
To: user@hbase.apache.org
Subject: RE: Coprocessor Increments
Classical deadlock :
CP-Region1 updates counter in CP-Region2 (waits on RPC)
CP-Region2 updates counter in CP-Region1 (waits on RPC)
I think its an anti-pattern. Do not do cross region calls in region CP code.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Michael Segel [msegel_hadoop@hotmail.com]
Sent: Thursday, October 10, 2013 9:55 AM
To: user@hbase.apache.org
Cc: Ted Yu
Subject: Re: Coprocessor Increments
I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.
Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
But the key is to really understanding and confirming what's causing the Deadlock in detail.
On Oct 10, 2013, at 11:15 AM, John Weatherford <jo...@telescope.tv> wrote:
> Michael,
> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>
>
>
> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>> Can we just take a quick pause…
>>
>> John you wrote the following:
>> "We have been running into an RPC deadlock issue on HBase and from
>> investigation, we believe the root of the issue is in us doing cross
>> region increments from a coprocessor. After some further searching and
>> reading over this
>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
>> we think that we can solve this by doing the increments locally on the
>> region. "
>>
>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>
>> Can either you or Andrew explain in detail what is meant by the RPC
>> deadlock?
>>
>> This goes back to rethink how to implement coprocessors.
>>
>>
>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>> <john.weatherford@telescope.tv <ma...@telescope.tv>>
>> wrote:
>>
>>> Thank you ted. I might have to rethink my key design to accomodate
>>> this. With this design and using the totals in keys as you suggested,
>>> I would have to scan the entire "California" group to find the
>>> totals. Thank you very much for your help.
>>>
>>> -John
>>>
>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>> John:
>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>> 'California-**
>>>> 95424' is the start key of region 2. You can choose
>>>> 'California-**12346#total'
>>>> to be the row key where increment is done for region 1 and
>>>> 'California-**95424#total'
>>>> to be the row key where increment is done for region 2.
>>>>
>>>> w.r.t. verification of whether given row key is indeed inside the
>>>> hosting
>>>> region, see the following:
>>>>
>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>
>>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>>
>>>> throw new WrongRegionException("Requested row out of range for " +
>>>>
>>>> op + " on HRegion " + this + ", startKey='" +
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>> john.weatherford@telescope.tv
>>>> <ma...@telescope.tv>> wrote:
>>>>
>>>>> First, Thank you everyone for the quick response.
>>>>>
>>>>> Ted:
>>>>> The custom split policy could be an interesting solution.
>>>>> Regarding the
>>>>> other option you sent, if I pull the end row key, I could construct
>>>>> an end
>>>>> key, but if I suffix the row key with the end key of the region,
>>>>> would that
>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>> suffixed key
>>>>> now be "California-total-California-**12346" and the other region's
>>>>> counter would be "California-total-California-**95424" and both of
>>>>> these
>>>>> keys would actually end up on the second region since the sorting
>>>>> of the
>>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>>
>>>>> Part of the question is that I don't understand if calling
>>>>> incrementColumnValue() on the region will always execute on the region
>>>>> called, regardles of rowkey. If so, what happens when those regions are
>>>>> merged?
>>>>>
>>>>> Thanks again for the help!
>>>>>
>>>>>
>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>>
>>>>>> bq. 'California-total' row in each region
>>>>>>
>>>>>> Of course such row key needs to be suffixed with either start or
>>>>>> end key
>>>>>> of
>>>>>> the corresponding region.
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>>> <ma...@gmail.com>> wrote:
>>>>>>
>>>>>> John:
>>>>>>> Looks like you need a 'California-total' row in each region.
>>>>>>>
>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> Contrived Example
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>>>
>>>>>>>>> Region 1.
>>>>>>>>
>>>>>>>> This would likely be on an insert on the same region. But as
>>>>>>>> the table
>>>>>>>>>> grows, this secondary insert could end up on another region.
>>>>>>>>>> If it is
>>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>>> which still
>>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>>>> region server? If so, what happens if these two regions are
>>>>>>>>>> compacted
>>>>>>>>>> into one?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>>>> split
>>>>>>>> is complete.
>>>>>>>>
>>>>>>>> Vladimir Rodionov
>>>>>>>> Principal Platform Engineer
>>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>>
>>>>>>>> ______________________________**__________
>>>>>>>>
>>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>>> including any attachments hereto, may be confidential and is
>>>>>>>> intended
>>>>>>>> to be
>>>>>>>> read only by the individual or entity to whom this message is
>>>>>>>> addressed. If
>>>>>>>> the reader of this message is not the intended recipient or an
>>>>>>>> agent or
>>>>>>>> designee of the intended recipient, please note that any review,
>>>>>>>> use,
>>>>>>>> disclosure or distribution of this message or its attachments,
>>>>>>>> in any
>>>>>>>> form,
>>>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>>>> please
>>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: Coprocessor Increments
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Classical deadlock :
CP-Region1 updates counter in CP-Region2 (waits on RPC)
CP-Region2 updates counter in CP-Region1 (waits on RPC)
I think its an anti-pattern. Do not do cross region calls in region CP code.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Michael Segel [msegel_hadoop@hotmail.com]
Sent: Thursday, October 10, 2013 9:55 AM
To: user@hbase.apache.org
Cc: Ted Yu
Subject: Re: Coprocessor Increments
I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.
Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
But the key is to really understanding and confirming what's causing the Deadlock in detail.
On Oct 10, 2013, at 11:15 AM, John Weatherford <jo...@telescope.tv> wrote:
> Michael,
> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>
>
>
> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>> Can we just take a quick pause…
>>
>> John you wrote the following:
>> "We have been running into an RPC deadlock issue on HBase and from
>> investigation, we believe the root of the issue is in us doing cross
>> region increments from a coprocessor. After some further searching and
>> reading over this
>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
>> we think that we can solve this by doing the increments locally on the
>> region. "
>>
>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>
>> Can either you or Andrew explain in detail what is meant by the RPC
>> deadlock?
>>
>> This goes back to rethink how to implement coprocessors.
>>
>>
>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>> <john.weatherford@telescope.tv <ma...@telescope.tv>>
>> wrote:
>>
>>> Thank you ted. I might have to rethink my key design to accomodate
>>> this. With this design and using the totals in keys as you suggested,
>>> I would have to scan the entire "California" group to find the
>>> totals. Thank you very much for your help.
>>>
>>> -John
>>>
>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>> John:
>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>> 'California-**
>>>> 95424' is the start key of region 2. You can choose
>>>> 'California-**12346#total'
>>>> to be the row key where increment is done for region 1 and
>>>> 'California-**95424#total'
>>>> to be the row key where increment is done for region 2.
>>>>
>>>> w.r.t. verification of whether given row key is indeed inside the
>>>> hosting
>>>> region, see the following:
>>>>
>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>
>>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>>
>>>> throw new WrongRegionException("Requested row out of range for " +
>>>>
>>>> op + " on HRegion " + this + ", startKey='" +
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>> john.weatherford@telescope.tv
>>>> <ma...@telescope.tv>> wrote:
>>>>
>>>>> First, Thank you everyone for the quick response.
>>>>>
>>>>> Ted:
>>>>> The custom split policy could be an interesting solution.
>>>>> Regarding the
>>>>> other option you sent, if I pull the end row key, I could construct
>>>>> an end
>>>>> key, but if I suffix the row key with the end key of the region,
>>>>> would that
>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>> suffixed key
>>>>> now be "California-total-California-**12346" and the other region's
>>>>> counter would be "California-total-California-**95424" and both of
>>>>> these
>>>>> keys would actually end up on the second region since the sorting
>>>>> of the
>>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>>
>>>>> Part of the question is that I don't understand if calling
>>>>> incrementColumnValue() on the region will always execute on the region
>>>>> called, regardles of rowkey. If so, what happens when those regions are
>>>>> merged?
>>>>>
>>>>> Thanks again for the help!
>>>>>
>>>>>
>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>>
>>>>>> bq. 'California-total' row in each region
>>>>>>
>>>>>> Of course such row key needs to be suffixed with either start or
>>>>>> end key
>>>>>> of
>>>>>> the corresponding region.
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>>> <ma...@gmail.com>> wrote:
>>>>>>
>>>>>> John:
>>>>>>> Looks like you need a 'California-total' row in each region.
>>>>>>>
>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> Contrived Example
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>>>
>>>>>>>>> Region 1.
>>>>>>>>
>>>>>>>> This would likely be on an insert on the same region. But as
>>>>>>>> the table
>>>>>>>>>> grows, this secondary insert could end up on another region.
>>>>>>>>>> If it is
>>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>>> which still
>>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>>>> region server? If so, what happens if these two regions are
>>>>>>>>>> compacted
>>>>>>>>>> into one?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>>>> split
>>>>>>>> is complete.
>>>>>>>>
>>>>>>>> Vladimir Rodionov
>>>>>>>> Principal Platform Engineer
>>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>>
>>>>>>>> ______________________________**__________
>>>>>>>>
>>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>>> including any attachments hereto, may be confidential and is
>>>>>>>> intended
>>>>>>>> to be
>>>>>>>> read only by the individual or entity to whom this message is
>>>>>>>> addressed. If
>>>>>>>> the reader of this message is not the intended recipient or an
>>>>>>>> agent or
>>>>>>>> designee of the intended recipient, please note that any review,
>>>>>>>> use,
>>>>>>>> disclosure or distribution of this message or its attachments,
>>>>>>>> in any
>>>>>>>> form,
>>>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>>>> please
>>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
I think Andrew has a handle on it… my take is that you end up running out of resources to handle an RPC connection while within your coprocessor and you're waiting for a resource to be free and it can't because another coprocessor has an RPC resource and is also waiting for a free resource.
Maybe its an over simplification, but if that's the case… you could always try thing to limit the RPC call, which would delay updating the counter. (Which may not be a problem) or redesign the coprocessors so that the coprocessors don't share the same RPC resources.
But the key is to really understanding and confirming what's causing the Deadlock in detail.
On Oct 10, 2013, at 11:15 AM, John Weatherford <jo...@telescope.tv> wrote:
> Michael,
> I would also really like to know how this issue is caused also. I can't even give a solid way to reproduce our deadlock. It *seems* to happen more under load, but nothing can be proved yet. While google-ing and looking for an answer I came across that old message post (http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E). This seemed to line up with what we are doing, so we _hope_ this will be a fix for us, but we aren't entirely sure.
>
>
>
> On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
>> Can we just take a quick pause…
>>
>> John you wrote the following:
>> "We have been running into an RPC deadlock issue on HBase and from
>> investigation, we believe the root of the issue is in us doing cross
>> region increments from a coprocessor. After some further searching and
>> reading over this
>> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
>> we think that we can solve this by doing the increments locally on the
>> region. "
>>
>> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>>
>> Can either you or Andrew explain in detail what is meant by the RPC
>> deadlock?
>>
>> This goes back to rethink how to implement coprocessors.
>>
>>
>> On Oct 9, 2013, at 11:03 PM, John Weatherford
>> <john.weatherford@telescope.tv <ma...@telescope.tv>>
>> wrote:
>>
>>> Thank you ted. I might have to rethink my key design to accomodate
>>> this. With this design and using the totals in keys as you suggested,
>>> I would have to scan the entire "California" group to find the
>>> totals. Thank you very much for your help.
>>>
>>> -John
>>>
>>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>>> John:
>>>> Suppose 'California-**12346' is the start key of region 1 and
>>>> 'California-**
>>>> 95424' is the start key of region 2. You can choose
>>>> 'California-**12346#total'
>>>> to be the row key where increment is done for region 1 and
>>>> 'California-**95424#total'
>>>> to be the row key where increment is done for region 2.
>>>>
>>>> w.r.t. verification of whether given row key is indeed inside the
>>>> hosting
>>>> region, see the following:
>>>>
>>>> void checkRow(final byte [] row, String op) throws IOException {
>>>>
>>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>>
>>>> throw new WrongRegionException("Requested row out of range for " +
>>>>
>>>> op + " on HRegion " + this + ", startKey='" +
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>>> john.weatherford@telescope.tv
>>>> <ma...@telescope.tv>> wrote:
>>>>
>>>>> First, Thank you everyone for the quick response.
>>>>>
>>>>> Ted:
>>>>> The custom split policy could be an interesting solution.
>>>>> Regarding the
>>>>> other option you sent, if I pull the end row key, I could construct
>>>>> an end
>>>>> key, but if I suffix the row key with the end key of the region,
>>>>> would that
>>>>> actually solve my problem? In the contrived case, wouldn't the
>>>>> suffixed key
>>>>> now be "California-total-California-**12346" and the other region's
>>>>> counter would be "California-total-California-**95424" and both of
>>>>> these
>>>>> keys would actually end up on the second region since the sorting
>>>>> of the
>>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>>
>>>>> Part of the question is that I don't understand if calling
>>>>> incrementColumnValue() on the region will always execute on the region
>>>>> called, regardles of rowkey. If so, what happens when those regions are
>>>>> merged?
>>>>>
>>>>> Thanks again for the help!
>>>>>
>>>>>
>>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>>
>>>>>> bq. 'California-total' row in each region
>>>>>>
>>>>>> Of course such row key needs to be suffixed with either start or
>>>>>> end key
>>>>>> of
>>>>>> the corresponding region.
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>>> <ma...@gmail.com>> wrote:
>>>>>>
>>>>>> John:
>>>>>>> Looks like you need a 'California-total' row in each region.
>>>>>>>
>>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> Contrived Example
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>>>
>>>>>>>>> Region 1.
>>>>>>>>
>>>>>>>> This would likely be on an insert on the same region. But as
>>>>>>>> the table
>>>>>>>>>> grows, this secondary insert could end up on another region.
>>>>>>>>>> If it is
>>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>>> which still
>>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>>>> region server? If so, what happens if these two regions are
>>>>>>>>>> compacted
>>>>>>>>>> into one?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>>>> split
>>>>>>>> is complete.
>>>>>>>>
>>>>>>>> Vladimir Rodionov
>>>>>>>> Principal Platform Engineer
>>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>>
>>>>>>>> ______________________________**__________
>>>>>>>>
>>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>>> including any attachments hereto, may be confidential and is
>>>>>>>> intended
>>>>>>>> to be
>>>>>>>> read only by the individual or entity to whom this message is
>>>>>>>> addressed. If
>>>>>>>> the reader of this message is not the intended recipient or an
>>>>>>>> agent or
>>>>>>>> designee of the intended recipient, please note that any review,
>>>>>>>> use,
>>>>>>>> disclosure or distribution of this message or its attachments,
>>>>>>>> in any
>>>>>>>> form,
>>>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>>>> please
>>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: Coprocessor Increments
Posted by John Weatherford <jo...@telescope.tv>.
Michael,
I would also really like to know how this issue is caused also. I
can't even give a solid way to reproduce our deadlock. It *seems* to
happen more under load, but nothing can be proved yet. While google-ing
and looking for an answer I came across that old message post
(http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E).
This seemed to line up with what we are doing, so we _hope_ this will
be a fix for us, but we aren't entirely sure.
On Thu 10 Oct 2013 07:57:46 AM PDT, Michael Segel wrote:
> Can we just take a quick pause…
>
> John you wrote the following:
> "We have been running into an RPC deadlock issue on HBase and from
> investigation, we believe the root of the issue is in us doing cross
> region increments from a coprocessor. After some further searching and
> reading over this
> <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E>
> we think that we can solve this by doing the increments locally on the
> region. "
>
> Which goes back to some thing Andrew wrote concerning an RPC deadlock.
>
> Can either you or Andrew explain in detail what is meant by the RPC
> deadlock?
>
> This goes back to rethink how to implement coprocessors.
>
>
> On Oct 9, 2013, at 11:03 PM, John Weatherford
> <john.weatherford@telescope.tv <ma...@telescope.tv>>
> wrote:
>
>> Thank you ted. I might have to rethink my key design to accomodate
>> this. With this design and using the totals in keys as you suggested,
>> I would have to scan the entire "California" group to find the
>> totals. Thank you very much for your help.
>>
>> -John
>>
>> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>>> John:
>>> Suppose 'California-**12346' is the start key of region 1 and
>>> 'California-**
>>> 95424' is the start key of region 2. You can choose
>>> 'California-**12346#total'
>>> to be the row key where increment is done for region 1 and
>>> 'California-**95424#total'
>>> to be the row key where increment is done for region 2.
>>>
>>> w.r.t. verification of whether given row key is indeed inside the
>>> hosting
>>> region, see the following:
>>>
>>> void checkRow(final byte [] row, String op) throws IOException {
>>>
>>> if (!rowIsInRange(getRegionInfo(), row)) {
>>>
>>> throw new WrongRegionException("Requested row out of range for " +
>>>
>>> op + " on HRegion " + this + ", startKey='" +
>>>
>>> Cheers
>>>
>>>
>>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>>> john.weatherford@telescope.tv
>>> <ma...@telescope.tv>> wrote:
>>>
>>>> First, Thank you everyone for the quick response.
>>>>
>>>> Ted:
>>>> The custom split policy could be an interesting solution.
>>>> Regarding the
>>>> other option you sent, if I pull the end row key, I could construct
>>>> an end
>>>> key, but if I suffix the row key with the end key of the region,
>>>> would that
>>>> actually solve my problem? In the contrived case, wouldn't the
>>>> suffixed key
>>>> now be "California-total-California-**12346" and the other region's
>>>> counter would be "California-total-California-**95424" and both of
>>>> these
>>>> keys would actually end up on the second region since the sorting
>>>> of the
>>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>>
>>>> Part of the question is that I don't understand if calling
>>>> incrementColumnValue() on the region will always execute on the region
>>>> called, regardles of rowkey. If so, what happens when those regions are
>>>> merged?
>>>>
>>>> Thanks again for the help!
>>>>
>>>>
>>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>>
>>>>> bq. 'California-total' row in each region
>>>>>
>>>>> Of course such row key needs to be suffixed with either start or
>>>>> end key
>>>>> of
>>>>> the corresponding region.
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yuzhihong@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>>
>>>>> John:
>>>>>> Looks like you need a 'California-total' row in each region.
>>>>>>
>>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>>> vrodionov@carrieriq.com <ma...@carrieriq.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> Contrived Example
>>>>>>>>>
>>>>>>>>
>>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>>
>>>>>>>> Region 1.
>>>>>>>
>>>>>>> This would likely be on an insert on the same region. But as
>>>>>>> the table
>>>>>>>>> grows, this secondary insert could end up on another region.
>>>>>>>>> If it is
>>>>>>>>> confined, then suppose we later insert "California-95424"
>>>>>>>>> which still
>>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>>> "California-total" all on Region 2.
>>>>>>>>>
>>>>>>>>
>>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>>> region server? If so, what happens if these two regions are
>>>>>>>>> compacted
>>>>>>>>> into one?
>>>>>>>>>
>>>>>>>>
>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>
>>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>>> split
>>>>>>> is complete.
>>>>>>>
>>>>>>> Vladimir Rodionov
>>>>>>> Principal Platform Engineer
>>>>>>> Carrier IQ, www.carrieriq.com
>>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>>
>>>>>>> ______________________________**__________
>>>>>>>
>>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>>> including any attachments hereto, may be confidential and is
>>>>>>> intended
>>>>>>> to be
>>>>>>> read only by the individual or entity to whom this message is
>>>>>>> addressed. If
>>>>>>> the reader of this message is not the intended recipient or an
>>>>>>> agent or
>>>>>>> designee of the intended recipient, please note that any review,
>>>>>>> use,
>>>>>>> disclosure or distribution of this message or its attachments,
>>>>>>> in any
>>>>>>> form,
>>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>>> please
>>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Coprocessor Increments
Posted by Michael Segel <ms...@hotmail.com>.
Can we just take a quick pause…
John you wrote the following:
"We have been running into an RPC deadlock issue on HBase and from investigation, we believe the root of the issue is in us doing cross region increments from a coprocessor. After some further searching and reading over this <http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5Cn3EkZDJZ_Z-2QT8xOZ+Q@mail.gmail.com%3E> we think that we can solve this by doing the increments locally on the region. "
Which goes back to some thing Andrew wrote concerning an RPC deadlock.
Can either you or Andrew explain in detail what is meant by the RPC deadlock?
This goes back to rethink how to implement coprocessors.
On Oct 9, 2013, at 11:03 PM, John Weatherford <jo...@telescope.tv> wrote:
> Thank you ted. I might have to rethink my key design to accomodate this. With this design and using the totals in keys as you suggested, I would have to scan the entire "California" group to find the totals. Thank you very much for your help.
>
> -John
>
> On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
>> John:
>> Suppose 'California-**12346' is the start key of region 1 and 'California-**
>> 95424' is the start key of region 2. You can choose 'California-**12346#total'
>> to be the row key where increment is done for region 1 and
>> 'California-**95424#total'
>> to be the row key where increment is done for region 2.
>>
>> w.r.t. verification of whether given row key is indeed inside the hosting
>> region, see the following:
>>
>> void checkRow(final byte [] row, String op) throws IOException {
>>
>> if (!rowIsInRange(getRegionInfo(), row)) {
>>
>> throw new WrongRegionException("Requested row out of range for " +
>>
>> op + " on HRegion " + this + ", startKey='" +
>>
>> Cheers
>>
>>
>> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
>> john.weatherford@telescope.tv> wrote:
>>
>>> First, Thank you everyone for the quick response.
>>>
>>> Ted:
>>> The custom split policy could be an interesting solution. Regarding the
>>> other option you sent, if I pull the end row key, I could construct an end
>>> key, but if I suffix the row key with the end key of the region, would that
>>> actually solve my problem? In the contrived case, wouldn't the suffixed key
>>> now be "California-total-California-**12346" and the other region's
>>> counter would be "California-total-California-**95424" and both of these
>>> keys would actually end up on the second region since the sorting of the
>>> keys would impose that "California-t*" comes after "California-[1-9]".
>>>
>>> Part of the question is that I don't understand if calling
>>> incrementColumnValue() on the region will always execute on the region
>>> called, regardles of rowkey. If so, what happens when those regions are
>>> merged?
>>>
>>> Thanks again for the help!
>>>
>>>
>>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>>
>>>> bq. 'California-total' row in each region
>>>>
>>>> Of course such row key needs to be suffixed with either start or end key
>>>> of
>>>> the corresponding region.
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>> John:
>>>>> Looks like you need a 'California-total' row in each region.
>>>>>
>>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>>> vrodionov@carrieriq.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> Contrived Example
>>>>>>>>
>>>>>>>
>>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>>
>>>>>>> Region 1.
>>>>>>
>>>>>> This would likely be on an insert on the same region. But as the table
>>>>>>>> grows, this secondary insert could end up on another region. If it is
>>>>>>>> confined, then suppose we later insert "California-95424" which still
>>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>>> "California-total" all on Region 2.
>>>>>>>>
>>>>>>>
>>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>>> region server? If so, what happens if these two regions are compacted
>>>>>>>> into one?
>>>>>>>>
>>>>>>>
>>>>>> Best regards,
>>>>>>>>
>>>>>>>
>>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>>> split
>>>>>> is complete.
>>>>>>
>>>>>> Vladimir Rodionov
>>>>>> Principal Platform Engineer
>>>>>> Carrier IQ, www.carrieriq.com
>>>>>> e-mail: vrodionov@carrieriq.com
>>>>>>
>>>>>> ______________________________**__________
>>>>>>
>>>>>> Confidentiality Notice: The information contained in this message,
>>>>>> including any attachments hereto, may be confidential and is intended
>>>>>> to be
>>>>>> read only by the individual or entity to whom this message is
>>>>>> addressed. If
>>>>>> the reader of this message is not the intended recipient or an agent or
>>>>>> designee of the intended recipient, please note that any review, use,
>>>>>> disclosure or distribution of this message or its attachments, in any
>>>>>> form,
>>>>>> is strictly prohibited. If you have received this message in error,
>>>>>> please
>>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>>> delete or destroy any copy of this message and its attachments.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>
>
Re: Coprocessor Increments
Posted by John Weatherford <jo...@telescope.tv>.
Thank you ted. I might have to rethink my key design to accomodate
this. With this design and using the totals in keys as you suggested, I
would have to scan the entire "California" group to find the totals.
Thank you very much for your help.
-John
On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
> John:
> Suppose 'California-**12346' is the start key of region 1 and 'California-**
> 95424' is the start key of region 2. You can choose 'California-**12346#total'
> to be the row key where increment is done for region 1 and
> 'California-**95424#total'
> to be the row key where increment is done for region 2.
>
> w.r.t. verification of whether given row key is indeed inside the hosting
> region, see the following:
>
> void checkRow(final byte [] row, String op) throws IOException {
>
> if (!rowIsInRange(getRegionInfo(), row)) {
>
> throw new WrongRegionException("Requested row out of range for " +
>
> op + " on HRegion " + this + ", startKey='" +
>
> Cheers
>
>
> On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
> john.weatherford@telescope.tv> wrote:
>
>> First, Thank you everyone for the quick response.
>>
>> Ted:
>> The custom split policy could be an interesting solution. Regarding the
>> other option you sent, if I pull the end row key, I could construct an end
>> key, but if I suffix the row key with the end key of the region, would that
>> actually solve my problem? In the contrived case, wouldn't the suffixed key
>> now be "California-total-California-**12346" and the other region's
>> counter would be "California-total-California-**95424" and both of these
>> keys would actually end up on the second region since the sorting of the
>> keys would impose that "California-t*" comes after "California-[1-9]".
>>
>> Part of the question is that I don't understand if calling
>> incrementColumnValue() on the region will always execute on the region
>> called, regardles of rowkey. If so, what happens when those regions are
>> merged?
>>
>> Thanks again for the help!
>>
>>
>> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>>
>>> bq. 'California-total' row in each region
>>>
>>> Of course such row key needs to be suffixed with either start or end key
>>> of
>>> the corresponding region.
>>>
>>>
>>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> John:
>>>> Looks like you need a 'California-total' row in each region.
>>>>
>>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>>> vrodionov@carrieriq.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> Contrived Example
>>>>>>>
>>>>>>
>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>>
>>>>>> Region 1.
>>>>>
>>>>> This would likely be on an insert on the same region. But as the table
>>>>>>> grows, this secondary insert could end up on another region. If it is
>>>>>>> confined, then suppose we later insert "California-95424" which still
>>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>>> "California-total" all on Region 2.
>>>>>>>
>>>>>>
>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>>> region server? If so, what happens if these two regions are compacted
>>>>>>> into one?
>>>>>>>
>>>>>>
>>>>> Best regards,
>>>>>>>
>>>>>>
>>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>>> split
>>>>> is complete.
>>>>>
>>>>> Vladimir Rodionov
>>>>> Principal Platform Engineer
>>>>> Carrier IQ, www.carrieriq.com
>>>>> e-mail: vrodionov@carrieriq.com
>>>>>
>>>>> ______________________________**__________
>>>>>
>>>>> Confidentiality Notice: The information contained in this message,
>>>>> including any attachments hereto, may be confidential and is intended
>>>>> to be
>>>>> read only by the individual or entity to whom this message is
>>>>> addressed. If
>>>>> the reader of this message is not the intended recipient or an agent or
>>>>> designee of the intended recipient, please note that any review, use,
>>>>> disclosure or distribution of this message or its attachments, in any
>>>>> form,
>>>>> is strictly prohibited. If you have received this message in error,
>>>>> please
>>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>>> delete or destroy any copy of this message and its attachments.
>>>>>
>>>>>
>>>>
>>>>
>>>
>
Re: Coprocessor Increments
Posted by Ted Yu <yu...@gmail.com>.
John:
Suppose 'California-**12346' is the start key of region 1 and 'California-**
95424' is the start key of region 2. You can choose 'California-**12346#total'
to be the row key where increment is done for region 1 and
'California-**95424#total'
to be the row key where increment is done for region 2.
w.r.t. verification of whether given row key is indeed inside the hosting
region, see the following:
void checkRow(final byte [] row, String op) throws IOException {
if (!rowIsInRange(getRegionInfo(), row)) {
throw new WrongRegionException("Requested row out of range for " +
op + " on HRegion " + this + ", startKey='" +
Cheers
On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford <
john.weatherford@telescope.tv> wrote:
> First, Thank you everyone for the quick response.
>
> Ted:
> The custom split policy could be an interesting solution. Regarding the
> other option you sent, if I pull the end row key, I could construct an end
> key, but if I suffix the row key with the end key of the region, would that
> actually solve my problem? In the contrived case, wouldn't the suffixed key
> now be "California-total-California-**12346" and the other region's
> counter would be "California-total-California-**95424" and both of these
> keys would actually end up on the second region since the sorting of the
> keys would impose that "California-t*" comes after "California-[1-9]".
>
> Part of the question is that I don't understand if calling
> incrementColumnValue() on the region will always execute on the region
> called, regardles of rowkey. If so, what happens when those regions are
> merged?
>
> Thanks again for the help!
>
>
> On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
>
>> bq. 'California-total' row in each region
>>
>> Of course such row key needs to be suffixed with either start or end key
>> of
>> the corresponding region.
>>
>>
>> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> John:
>>> Looks like you need a 'California-total' row in each region.
>>>
>>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>>> http://hbase.apache.org/book.**html#d0e6541<http://hbase.apache.org/book.html#d0e6541>
>>>
>>> Cheers
>>>
>>>
>>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <
>>> vrodionov@carrieriq.com
>>>
>>>> wrote:
>>>>
>>>
>>> Contrived Example
>>>>>>
>>>>>
>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>>>>>
>>>>> Region 1.
>>>>
>>>> This would likely be on an insert on the same region. But as the table
>>>>>> grows, this secondary insert could end up on another region. If it is
>>>>>> confined, then suppose we later insert "California-95424" which still
>>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>>> "California-total" all on Region 2.
>>>>>>
>>>>>
>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>>> region server? If so, what happens if these two regions are compacted
>>>>>> into one?
>>>>>>
>>>>>
>>>> Best regards,
>>>>>>
>>>>>
>>>> Nope, Your "California-total" will migrate to Region 2 after region
>>>> split
>>>> is complete.
>>>>
>>>> Vladimir Rodionov
>>>> Principal Platform Engineer
>>>> Carrier IQ, www.carrieriq.com
>>>> e-mail: vrodionov@carrieriq.com
>>>>
>>>> ______________________________**__________
>>>>
>>>> Confidentiality Notice: The information contained in this message,
>>>> including any attachments hereto, may be confidential and is intended
>>>> to be
>>>> read only by the individual or entity to whom this message is
>>>> addressed. If
>>>> the reader of this message is not the intended recipient or an agent or
>>>> designee of the intended recipient, please note that any review, use,
>>>> disclosure or distribution of this message or its attachments, in any
>>>> form,
>>>> is strictly prohibited. If you have received this message in error,
>>>> please
>>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>>> delete or destroy any copy of this message and its attachments.
>>>>
>>>>
>>>
>>>
>>
Re: Coprocessor Increments
Posted by John Weatherford <jo...@telescope.tv>.
First, Thank you everyone for the quick response.
Ted:
The custom split policy could be an interesting solution. Regarding
the other option you sent, if I pull the end row key, I could construct
an end key, but if I suffix the row key with the end key of the region,
would that actually solve my problem? In the contrived case, wouldn't
the suffixed key now be "California-total-California-12346" and the
other region's counter would be "California-total-California-95424" and
both of these keys would actually end up on the second region since the
sorting of the keys would impose that "California-t*" comes after
"California-[1-9]".
Part of the question is that I don't understand if calling
incrementColumnValue() on the region will always execute on the region
called, regardles of rowkey. If so, what happens when those regions are
merged?
Thanks again for the help!
On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
> bq. 'California-total' row in each region
>
> Of course such row key needs to be suffixed with either start or end key of
> the corresponding region.
>
>
> On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> John:
>> Looks like you need a 'California-total' row in each region.
>>
>> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
>> http://hbase.apache.org/book.html#d0e6541
>>
>> Cheers
>>
>>
>> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <vrodionov@carrieriq.com
>>> wrote:
>>
>>>>> Contrived Example
>>>
>>>>> Insert rowkey "California-12345" triggers a coprocessor to call
>>>>> incrementColumnValue() with a rowkey of "California-total" all on
>>> Region 1.
>>>
>>>>> This would likely be on an insert on the same region. But as the table
>>>>> grows, this secondary insert could end up on another region. If it is
>>>>> confined, then suppose we later insert "California-95424" which still
>>>>> triggers a call to incrementColumnValue() with a rowkey of
>>>>> "California-total" all on Region 2.
>>>
>>>>> Are we now left with two rowkeys of "California-total"? One on each
>>>>> region server? If so, what happens if these two regions are compacted
>>>>> into one?
>>>
>>>>> Best regards,
>>>
>>> Nope, Your "California-total" will migrate to Region 2 after region split
>>> is complete.
>>>
>>> Vladimir Rodionov
>>> Principal Platform Engineer
>>> Carrier IQ, www.carrieriq.com
>>> e-mail: vrodionov@carrieriq.com
>>>
>>> ________________________________________
>>>
>>> Confidentiality Notice: The information contained in this message,
>>> including any attachments hereto, may be confidential and is intended to be
>>> read only by the individual or entity to whom this message is addressed. If
>>> the reader of this message is not the intended recipient or an agent or
>>> designee of the intended recipient, please note that any review, use,
>>> disclosure or distribution of this message or its attachments, in any form,
>>> is strictly prohibited. If you have received this message in error, please
>>> immediately notify the sender and/or Notifications@carrieriq.com and
>>> delete or destroy any copy of this message and its attachments.
>>>
>>
>>
>
Re: Coprocessor Increments
Posted by Ted Yu <yu...@gmail.com>.
bq. 'California-total' row in each region
Of course such row key needs to be suffixed with either start or end key of
the corresponding region.
On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu <yu...@gmail.com> wrote:
> John:
> Looks like you need a 'California-total' row in each region.
>
> Custom Split Policy can help you achieve that, see 9.7.4.1 in:
> http://hbase.apache.org/book.html#d0e6541
>
> Cheers
>
>
> On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov <vrodionov@carrieriq.com
> > wrote:
>
>> >>Contrived Example
>>
>> >>Insert rowkey "California-12345" triggers a coprocessor to call
>> >>incrementColumnValue() with a rowkey of "California-total" all on
>> Region 1.
>>
>> >>This would likely be on an insert on the same region. But as the table
>> >>grows, this secondary insert could end up on another region. If it is
>> >>confined, then suppose we later insert "California-95424" which still
>> >>triggers a call to incrementColumnValue() with a rowkey of
>> >>"California-total" all on Region 2.
>>
>> >>Are we now left with two rowkeys of "California-total"? One on each
>> >>region server? If so, what happens if these two regions are compacted
>> >>into one?
>>
>> >>Best regards,
>>
>> Nope, Your "California-total" will migrate to Region 2 after region split
>> is complete.
>>
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: vrodionov@carrieriq.com
>>
>> ________________________________________
>>
>> Confidentiality Notice: The information contained in this message,
>> including any attachments hereto, may be confidential and is intended to be
>> read only by the individual or entity to whom this message is addressed. If
>> the reader of this message is not the intended recipient or an agent or
>> designee of the intended recipient, please note that any review, use,
>> disclosure or distribution of this message or its attachments, in any form,
>> is strictly prohibited. If you have received this message in error, please
>> immediately notify the sender and/or Notifications@carrieriq.com and
>> delete or destroy any copy of this message and its attachments.
>>
>
>
Re: Coprocessor Increments
Posted by Ted Yu <yu...@gmail.com>.
John:
Looks like you need a 'California-total' row in each region.
Custom Split Policy can help you achieve that, see 9.7.4.1 in:
http://hbase.apache.org/book.html#d0e6541
Cheers
On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov
<vr...@carrieriq.com>wrote:
> >>Contrived Example
>
> >>Insert rowkey "California-12345" triggers a coprocessor to call
> >>incrementColumnValue() with a rowkey of "California-total" all on
> Region 1.
>
> >>This would likely be on an insert on the same region. But as the table
> >>grows, this secondary insert could end up on another region. If it is
> >>confined, then suppose we later insert "California-95424" which still
> >>triggers a call to incrementColumnValue() with a rowkey of
> >>"California-total" all on Region 2.
>
> >>Are we now left with two rowkeys of "California-total"? One on each
> >>region server? If so, what happens if these two regions are compacted
> >>into one?
>
> >>Best regards,
>
> Nope, Your "California-total" will migrate to Region 2 after region split
> is complete.
>
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
RE: Coprocessor Increments
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
>>Contrived Example
>>Insert rowkey "California-12345" triggers a coprocessor to call
>>incrementColumnValue() with a rowkey of "California-total" all on Region 1.
>>This would likely be on an insert on the same region. But as the table
>>grows, this secondary insert could end up on another region. If it is
>>confined, then suppose we later insert "California-95424" which still
>>triggers a call to incrementColumnValue() with a rowkey of
>>"California-total" all on Region 2.
>>Are we now left with two rowkeys of "California-total"? One on each
>>region server? If so, what happens if these two regions are compacted
>>into one?
>>Best regards,
Nope, Your "California-total" will migrate to Region 2 after region split is complete.
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.