You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by David Wales <da...@gmail.com> on 2017/03/04 12:58:38 UTC
Gemfire proxy cache race condition bug?
Hi there,
I have a client proxy cache with an associated cache listener. An
invocation of the region get method will trigger the afterCreate callback.
I realized this call back was on the same thread that called get and is
different to the thread that dispatches the region updates. So I had a look
at the code and there seems to be no synchronization. I could mark the
cache listener methods as synchronized but this doesn't fully mitigate the
race. How does one get the initial state of an entry in a region in a way
that synchronizes with updates on. Is this a known limitation or a bug as
the documentation talks about cache listener methods being mutually
exclusive?
Much appreciated
Thank you
David
Re: Gemfire proxy cache race condition bug?
Posted by Akihiro Kitada <ak...@pivotal.io>.
Hello David,
Have you set conserve-sockets=false to avoid potential connection level
dead-lock? And also, please refer to the following KB to avoid any types of
dead-locks.
https://discuss.pivotal.io/hc/en-us/articles/201298393
Thanks.
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
Support.Pivotal.io <http://support.pivotal.io/> | Mon-Fri 9:00am to
5:30pm JST | 1-877-477-2269
[image: support] <https://support.pivotal.io/> [image: twitter]
<https://twitter.com/pivotal> [image: linkedin]
<https://www.linkedin.com/company/3048967> [image: facebook]
<https://www.facebook.com/pivotalsoftware> [image: google plus]
<https://plus.google.com/+Pivotal> [image: youtube]
<https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl>
2017-03-04 21:58 GMT+09:00 David Wales <da...@gmail.com>:
> Hi there,
>
> I have a client proxy cache with an associated cache listener. An
> invocation of the region get method will trigger the afterCreate callback.
> I realized this call back was on the same thread that called get and is
> different to the thread that dispatches the region updates. So I had a look
> at the code and there seems to be no synchronization. I could mark the
> cache listener methods as synchronized but this doesn't fully mitigate the
> race. How does one get the initial state of an entry in a region in a way
> that synchronizes with updates on. Is this a known limitation or a bug as
> the documentation talks about cache listener methods being mutually
> exclusive?
>
> Much appreciated
> Thank you
> David
>
Re: Gemfire proxy cache race condition bug?
Posted by Darrel Schneider <ds...@pivotal.io>.
This is a known limitation of CacheListeners on a PROXY region. On other
types of regions the local region entry that the operation is modifying is
synchronized during the operation and during the CacheListener invocation.
That is what the javadocs on the CacheListener calls "holding a lock on the
entry." But since a PROXY region has none of its own state it has no local
region entry to hold a lock on. The javadocs should be changed to mention
this. Did you see this in some other place in the docs?
Another thing to keep in mind is that even when the entry is locked during
CacheListener invocation this does not guarantee a total ordering across
the distributed system. In particular the events you receive on a client
are async with those that occur on the server. You best bet is to use a
CacheListener on the server side on a partitioned region. All the modify
ops go through the single primary.
Another odd thing about a PROXY is that "get" will call afterCreate on a
listener on the proxy even when the get was a hit on the server. This is
because our PROXY logic always looks at the local state (not the server
state) when deciding on what CacheListener method to call. Ideally you
would only have "get" deliver afterCreate when the "get" does a load.
On Mon, Mar 6, 2017 at 8:55 AM, Michael Stolz <ms...@pivotal.io> wrote:
> If you always do whatever processing you are planning to do inside the
> CacheListener and use the "newValue" field that is supplied to that
> callback you will always begin processing in the order the data was
> received. I suppose if you want to ensure that you are completing
> processing in the correct order you would need to synchronize on your
> callback as you said.
>
> To "prime the pump" you can use get as you are now, or registerInterest
> with a list of keys or wildcard on string-based keys. Register interest can
> optionally deliver the initial value followed by all updates in the correct
> order.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: +1-631-835-4771 <(631)%20835-4771>
>
> On Sat, Mar 4, 2017 at 7:58 AM, David Wales <da...@gmail.com>
> wrote:
>
>> Hi there,
>>
>> I have a client proxy cache with an associated cache listener. An
>> invocation of the region get method will trigger the afterCreate callback.
>> I realized this call back was on the same thread that called get and is
>> different to the thread that dispatches the region updates. So I had a
>> look
>> at the code and there seems to be no synchronization. I could mark the
>> cache listener methods as synchronized but this doesn't fully mitigate the
>> race. How does one get the initial state of an entry in a region in a way
>> that synchronizes with updates on. Is this a known limitation or a bug as
>> the documentation talks about cache listener methods being mutually
>> exclusive?
>>
>> Much appreciated
>> Thank you
>> David
>>
>
>
Re: Gemfire proxy cache race condition bug?
Posted by Darrel Schneider <ds...@pivotal.io>.
This is a known limitation of CacheListeners on a PROXY region. On other
types of regions the local region entry that the operation is modifying is
synchronized during the operation and during the CacheListener invocation.
That is what the javadocs on the CacheListener calls "holding a lock on the
entry." But since a PROXY region has none of its own state it has no local
region entry to hold a lock on. The javadocs should be changed to mention
this. Did you see this in some other place in the docs?
Another thing to keep in mind is that even when the entry is locked during
CacheListener invocation this does not guarantee a total ordering across
the distributed system. In particular the events you receive on a client
are async with those that occur on the server. You best bet is to use a
CacheListener on the server side on a partitioned region. All the modify
ops go through the single primary.
Another odd thing about a PROXY is that "get" will call afterCreate on a
listener on the proxy even when the get was a hit on the server. This is
because our PROXY logic always looks at the local state (not the server
state) when deciding on what CacheListener method to call. Ideally you
would only have "get" deliver afterCreate when the "get" does a load.
On Mon, Mar 6, 2017 at 8:55 AM, Michael Stolz <ms...@pivotal.io> wrote:
> If you always do whatever processing you are planning to do inside the
> CacheListener and use the "newValue" field that is supplied to that
> callback you will always begin processing in the order the data was
> received. I suppose if you want to ensure that you are completing
> processing in the correct order you would need to synchronize on your
> callback as you said.
>
> To "prime the pump" you can use get as you are now, or registerInterest
> with a list of keys or wildcard on string-based keys. Register interest can
> optionally deliver the initial value followed by all updates in the correct
> order.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: +1-631-835-4771 <(631)%20835-4771>
>
> On Sat, Mar 4, 2017 at 7:58 AM, David Wales <da...@gmail.com>
> wrote:
>
>> Hi there,
>>
>> I have a client proxy cache with an associated cache listener. An
>> invocation of the region get method will trigger the afterCreate callback.
>> I realized this call back was on the same thread that called get and is
>> different to the thread that dispatches the region updates. So I had a
>> look
>> at the code and there seems to be no synchronization. I could mark the
>> cache listener methods as synchronized but this doesn't fully mitigate the
>> race. How does one get the initial state of an entry in a region in a way
>> that synchronizes with updates on. Is this a known limitation or a bug as
>> the documentation talks about cache listener methods being mutually
>> exclusive?
>>
>> Much appreciated
>> Thank you
>> David
>>
>
>
Re: Gemfire proxy cache race condition bug?
Posted by Michael Stolz <ms...@pivotal.io>.
If you always do whatever processing you are planning to do inside the
CacheListener and use the "newValue" field that is supplied to that
callback you will always begin processing in the order the data was
received. I suppose if you want to ensure that you are completing
processing in the correct order you would need to synchronize on your
callback as you said.
To "prime the pump" you can use get as you are now, or registerInterest
with a list of keys or wildcard on string-based keys. Register interest can
optionally deliver the initial value followed by all updates in the correct
order.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771
On Sat, Mar 4, 2017 at 7:58 AM, David Wales <da...@gmail.com>
wrote:
> Hi there,
>
> I have a client proxy cache with an associated cache listener. An
> invocation of the region get method will trigger the afterCreate callback.
> I realized this call back was on the same thread that called get and is
> different to the thread that dispatches the region updates. So I had a look
> at the code and there seems to be no synchronization. I could mark the
> cache listener methods as synchronized but this doesn't fully mitigate the
> race. How does one get the initial state of an entry in a region in a way
> that synchronizes with updates on. Is this a known limitation or a bug as
> the documentation talks about cache listener methods being mutually
> exclusive?
>
> Much appreciated
> Thank you
> David
>
Re: Gemfire proxy cache race condition bug?
Posted by Michael Stolz <ms...@pivotal.io>.
If you always do whatever processing you are planning to do inside the
CacheListener and use the "newValue" field that is supplied to that
callback you will always begin processing in the order the data was
received. I suppose if you want to ensure that you are completing
processing in the correct order you would need to synchronize on your
callback as you said.
To "prime the pump" you can use get as you are now, or registerInterest
with a list of keys or wildcard on string-based keys. Register interest can
optionally deliver the initial value followed by all updates in the correct
order.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771
On Sat, Mar 4, 2017 at 7:58 AM, David Wales <da...@gmail.com>
wrote:
> Hi there,
>
> I have a client proxy cache with an associated cache listener. An
> invocation of the region get method will trigger the afterCreate callback.
> I realized this call back was on the same thread that called get and is
> different to the thread that dispatches the region updates. So I had a look
> at the code and there seems to be no synchronization. I could mark the
> cache listener methods as synchronized but this doesn't fully mitigate the
> race. How does one get the initial state of an entry in a region in a way
> that synchronizes with updates on. Is this a known limitation or a bug as
> the documentation talks about cache listener methods being mutually
> exclusive?
>
> Much appreciated
> Thank you
> David
>
Re: Gemfire proxy cache race condition bug?
Posted by Akihiro Kitada <ak...@pivotal.io>.
Hello David,
Have you set conserve-sockets=false to avoid potential connection level
dead-lock? And also, please refer to the following KB to avoid any types of
dead-locks.
https://discuss.pivotal.io/hc/en-us/articles/201298393
Thanks.
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
Support.Pivotal.io <http://support.pivotal.io/> | Mon-Fri 9:00am to
5:30pm JST | 1-877-477-2269
[image: support] <https://support.pivotal.io/> [image: twitter]
<https://twitter.com/pivotal> [image: linkedin]
<https://www.linkedin.com/company/3048967> [image: facebook]
<https://www.facebook.com/pivotalsoftware> [image: google plus]
<https://plus.google.com/+Pivotal> [image: youtube]
<https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl>
2017-03-04 21:58 GMT+09:00 David Wales <da...@gmail.com>:
> Hi there,
>
> I have a client proxy cache with an associated cache listener. An
> invocation of the region get method will trigger the afterCreate callback.
> I realized this call back was on the same thread that called get and is
> different to the thread that dispatches the region updates. So I had a look
> at the code and there seems to be no synchronization. I could mark the
> cache listener methods as synchronized but this doesn't fully mitigate the
> race. How does one get the initial state of an entry in a region in a way
> that synchronizes with updates on. Is this a known limitation or a bug as
> the documentation talks about cache listener methods being mutually
> exclusive?
>
> Much appreciated
> Thank you
> David
>