You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Jakov Varenina <ja...@est.tech> on 2020/04/15 05:56:42 UTC
Unable to get behavior described in documentation when using durable
native client
Hi all,
Could you please help me understand behavior of the native client when
configured as durable?
I have been working on a bug GEODE-7944
<https://issues.apache.org/jira/browse/GEODE-7944> which results with
exception "Unable to deserialize membership id java.io.EOFException" on
locator only when debug is enabled. This happens because native client,
only when subscription is enabled, sends towards locator
QueueConnectionRequest that doesn't encapsulate ClientProxyMembershipID
(not properly serialized) and therefore exception occurs when locator
tries to deserialize membership id to log it at debug level.
I was trying to figure out why would locator need
ClientProxyMembershipID from native client and found following paragraph
in the documentation (copied from
https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html):
/For durable subscriptions, the server locator must be able to
locate the servers that host the queues for the durable client. When
a durable client sends a request, the server locator queries all the
available servers to see if they are hosting the subscription region
queue for the durable client. If the server is located, the client
is connected to the server hosting the subscription region queue./
Locator behaves as described in above paragraph only when it receives
///QueueConnectionRequest with ///findDurable flag set to "true" //and
with valid membership i//d. //I noticed that unlike java client, the
native client always sets //findDurable// to //"false" //and therefore
locator will never behave as described in above paragraph when native
client is used.
Does anybody know why native client always sets //findDurable=false//?
BRs,
Jakov
Re: Unable to get behavior described in documentation when using
durable native client
Posted by Jacob Barrett <jb...@pivotal.io>.
This log message about the GFE version is suspicious too. That is a VERY old version with ordinal value 1. The C++ client is supposed to be sending 45 which is GFE 9.0.0 / Geode 1.0.0.
Looks like there might be a handshake protocol misalignment somewhere. Will keep digging.
-Jake
> On Apr 17, 2020, at 7:24 AM, Jacob Barrett <jb...@pivotal.io> wrote:
>
> Can you confirm that when log level less than debug that the IOException goes away and the client appears to function?
>
> -Jake
>
>
>> On Apr 17, 2020, at 1:12 AM, Jakov Varenina <ja...@est.tech> wrote:
>>
>> Hi Jacob,
>>
>> Thanks for your response!
>>
>> Regarding GEODE-7944, "Unable to deserialize *membership id* java.io.EOFException" is not logged but thrown, and it breaks processing of QueueConnectionRequest in locator. This reflects in native client with "No locators found" even though they are available. Happens only when native client with subscription is enabled and locator started with --log-level=debug.
>>
>> I haven't had time to test and analyze in detail native durable client yet. So far I could only confirm that when using native durable client then locator behaves differently than how it is described in documentation (see previous mail) and how java client works:
>>
>> It seems that native durable client always requests from locator all available servers (redundant= -1, findDurable=false) with QueueConnectionRequest. Locator returns them in QueueConnectionResponse ordered by load (best...worst). While for java durable client, locator use *membership id *from QueueConnectionRequest to locate servers that host client queue and send them back in QueueConnectionResponse as described in previous mail. I expect that native durable client is handling re-connection to same servers queue somehow also, but this has to be investigated yet. Any hints or comments related to this would be really appreciated.
>>
>> BRs,
>>
>> Jakov
>>
>> On 15. 04. 2020. 10:07, Jacob Barrett wrote:
>>> Looking back at history the native library has always only ever set that findDurable flag to false. I traced it back to its initial commit. Aside from the annoying log message, does client durable connection work correctly?
>>>
>>>> On Apr 14, 2020, at 10:56 PM, Jakov Varenina <ja...@est.tech> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Could you please help me understand behavior of the native client when configured as durable?
>>>>
>>>> I have been working on a bug GEODE-7944 <https://issues.apache.org/jira/browse/GEODE-7944> which results with exception "Unable to deserialize membership id java.io.EOFException" on locator only when debug is enabled. This happens because native client, only when subscription is enabled, sends towards locator QueueConnectionRequest that doesn't encapsulate ClientProxyMembershipID (not properly serialized) and therefore exception occurs when locator tries to deserialize membership id to log it at debug level.
>>>>
>>>> I was trying to figure out why would locator need ClientProxyMembershipID from native client and found following paragraph in the documentation (copied from https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html):
>>>>
>>>> /For durable subscriptions, the server locator must be able to
>>>> locate the servers that host the queues for the durable client. When
>>>> a durable client sends a request, the server locator queries all the
>>>> available servers to see if they are hosting the subscription region
>>>> queue for the durable client. If the server is located, the client
>>>> is connected to the server hosting the subscription region queue./
>>>>
>>>> Locator behaves as described in above paragraph only when it receives ///QueueConnectionRequest with ///findDurable flag set to "true" //and with valid membership i//d. //I noticed that unlike java client, the native client always sets //findDurable// to //"false" //and therefore locator will never behave as described in above paragraph when native client is used.
>>>>
>>>> Does anybody know why native client always sets //findDurable=false//?
>>>>
>>>> BRs,
>>>>
>>>> Jakov
>
Re: Unable to get behavior described in documentation when using
durable native client
Posted by Jakov Varenina <ja...@est.tech>.
Hi Jacob,
Native durable client reconnects to the servers hosting the queue in a
following way:
1. Always sends QueueConnectionRequest (redundant=-1,
findDurable=false, ClientProxyMembershipID="Not set, except uniqueID
which is hard-coded to 1") requesting all available servers
regardless of the value used in "setSubscriptionRedundancy".
2. Native client then sends its ClientProxyMembershipID in handshake
towards received servers. Each server based on
ClientProxyMembershipID will return within handshake indication if
it is primary, redundant or non-redundant server for the event queue.
* if it is first time durable client connects then all severs will
be non-redundant (non of them hosting subscription region
queue). Then native client algorithm will select primary server
and number of redundant servers based on
"setSubscriptionRedundancy" and perform subscriptions to them
accordingly.
* if it is reconnect case, then the native client will get
indication in handshake which of the servers are primary and
redundant and reconnect to them accordingly.
So it seems that below native client document incorrectly describes
locator behavior when native client is used. Maybe it would be good to
update it to reflect correct behavior?
https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html
...
/When a client registers interest for a region, if the connection
pool does not already have a subscription channel, the connection
pool sends a message to the server locator, and the server locator
chooses servers to host the queue and return those server names to
the client. The client then contacts the chosen servers and asks
them to create the queue./
/..../
/For durable subscriptions, the server locator must be able to
locate the servers that host the queues for the durable client. When
a durable client sends a request, the server locator queries all the
available servers to see if they are hosting the subscription region
queue for the durable client. If the server is located, the client
is connected to the server hosting the subscription region queue./
BRs,
Jakov
On 20. 04. 2020. 08:10, Jakov Varenina wrote:
> Yes I can. IOException is not thrown and the client works in that case.
>
> BRs,
>
> Jakov
>
> On 17. 04. 2020. 16:24, Jacob Barrett wrote:
>> Can you confirm that when log level less than debug that the
>> IOException goes away and the client appears to function?
>>
>> -Jake
>>
>>
>>> On Apr 17, 2020, at 1:12 AM, Jakov Varenina
>>> <ja...@est.tech> wrote:
>>>
>>> Hi Jacob,
>>>
>>> Thanks for your response!
>>>
>>> Regarding GEODE-7944, "Unable to deserialize *membership id*
>>> java.io.EOFException" is not logged but thrown, and it breaks
>>> processing of QueueConnectionRequest in locator. This reflects in
>>> native client with "No locators found" even though they are
>>> available. Happens only when native client with subscription is
>>> enabled and locator started with --log-level=debug.
>>>
>>> I haven't had time to test and analyze in detail native durable
>>> client yet. So far I could only confirm that when using native
>>> durable client then locator behaves differently than how it is
>>> described in documentation (see previous mail) and how java client
>>> works:
>>>
>>> It seems that native durable client always requests from locator all
>>> available servers (redundant= -1, findDurable=false) with
>>> QueueConnectionRequest. Locator returns them in
>>> QueueConnectionResponse ordered by load (best...worst). While for
>>> java durable client, locator use *membership id *from
>>> QueueConnectionRequest to locate servers that host client queue and
>>> send them back in QueueConnectionResponse as described in previous
>>> mail. I expect that native durable client is handling re-connection
>>> to same servers queue somehow also, but this has to be investigated
>>> yet. Any hints or comments related to this would be really appreciated.
>>>
>>> BRs,
>>>
>>> Jakov
>>>
>>> On 15. 04. 2020. 10:07, Jacob Barrett wrote:
>>>> Looking back at history the native library has always only ever set
>>>> that findDurable flag to false. I traced it back to its initial
>>>> commit. Aside from the annoying log message, does client durable
>>>> connection work correctly?
>>>>
>>>>> On Apr 14, 2020, at 10:56 PM, Jakov Varenina
>>>>> <ja...@est.tech> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Could you please help me understand behavior of the native client
>>>>> when configured as durable?
>>>>>
>>>>> I have been working on a bug GEODE-7944
>>>>> <https://issues.apache.org/jira/browse/GEODE-7944> which results
>>>>> with exception "Unable to deserialize membership id
>>>>> java.io.EOFException" on locator only when debug is enabled. This
>>>>> happens because native client, only when subscription is enabled,
>>>>> sends towards locator QueueConnectionRequest that doesn't
>>>>> encapsulate ClientProxyMembershipID (not properly serialized) and
>>>>> therefore exception occurs when locator tries to deserialize
>>>>> membership id to log it at debug level.
>>>>>
>>>>> I was trying to figure out why would locator need
>>>>> ClientProxyMembershipID from native client and found following
>>>>> paragraph in the documentation (copied from
>>>>> https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html):
>>>>>
>>>>> /For durable subscriptions, the server locator must be able to
>>>>> locate the servers that host the queues for the durable client.
>>>>> When
>>>>> a durable client sends a request, the server locator queries
>>>>> all the
>>>>> available servers to see if they are hosting the subscription
>>>>> region
>>>>> queue for the durable client. If the server is located, the client
>>>>> is connected to the server hosting the subscription region queue./
>>>>>
>>>>> Locator behaves as described in above paragraph only when it
>>>>> receives ///QueueConnectionRequest with ///findDurable flag set to
>>>>> "true" //and with valid membership i//d. //I noticed that unlike
>>>>> java client, the native client always sets //findDurable// to
>>>>> //"false" //and therefore locator will never behave as described
>>>>> in above paragraph when native client is used.
>>>>>
>>>>> Does anybody know why native client always sets
>>>>> //findDurable=false//?
>>>>>
>>>>> BRs,
>>>>>
>>>>> Jakov
Re: Unable to get behavior described in documentation when using
durable native client
Posted by Jakov Varenina <ja...@est.tech>.
Yes I can. IOException is not thrown and the client works in that case.
BRs,
Jakov
On 17. 04. 2020. 16:24, Jacob Barrett wrote:
> Can you confirm that when log level less than debug that the IOException goes away and the client appears to function?
>
> -Jake
>
>
>> On Apr 17, 2020, at 1:12 AM, Jakov Varenina <ja...@est.tech> wrote:
>>
>> Hi Jacob,
>>
>> Thanks for your response!
>>
>> Regarding GEODE-7944, "Unable to deserialize *membership id* java.io.EOFException" is not logged but thrown, and it breaks processing of QueueConnectionRequest in locator. This reflects in native client with "No locators found" even though they are available. Happens only when native client with subscription is enabled and locator started with --log-level=debug.
>>
>> I haven't had time to test and analyze in detail native durable client yet. So far I could only confirm that when using native durable client then locator behaves differently than how it is described in documentation (see previous mail) and how java client works:
>>
>> It seems that native durable client always requests from locator all available servers (redundant= -1, findDurable=false) with QueueConnectionRequest. Locator returns them in QueueConnectionResponse ordered by load (best...worst). While for java durable client, locator use *membership id *from QueueConnectionRequest to locate servers that host client queue and send them back in QueueConnectionResponse as described in previous mail. I expect that native durable client is handling re-connection to same servers queue somehow also, but this has to be investigated yet. Any hints or comments related to this would be really appreciated.
>>
>> BRs,
>>
>> Jakov
>>
>> On 15. 04. 2020. 10:07, Jacob Barrett wrote:
>>> Looking back at history the native library has always only ever set that findDurable flag to false. I traced it back to its initial commit. Aside from the annoying log message, does client durable connection work correctly?
>>>
>>>> On Apr 14, 2020, at 10:56 PM, Jakov Varenina <ja...@est.tech> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Could you please help me understand behavior of the native client when configured as durable?
>>>>
>>>> I have been working on a bug GEODE-7944 <https://issues.apache.org/jira/browse/GEODE-7944> which results with exception "Unable to deserialize membership id java.io.EOFException" on locator only when debug is enabled. This happens because native client, only when subscription is enabled, sends towards locator QueueConnectionRequest that doesn't encapsulate ClientProxyMembershipID (not properly serialized) and therefore exception occurs when locator tries to deserialize membership id to log it at debug level.
>>>>
>>>> I was trying to figure out why would locator need ClientProxyMembershipID from native client and found following paragraph in the documentation (copied from https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html):
>>>>
>>>> /For durable subscriptions, the server locator must be able to
>>>> locate the servers that host the queues for the durable client. When
>>>> a durable client sends a request, the server locator queries all the
>>>> available servers to see if they are hosting the subscription region
>>>> queue for the durable client. If the server is located, the client
>>>> is connected to the server hosting the subscription region queue./
>>>>
>>>> Locator behaves as described in above paragraph only when it receives ///QueueConnectionRequest with ///findDurable flag set to "true" //and with valid membership i//d. //I noticed that unlike java client, the native client always sets //findDurable// to //"false" //and therefore locator will never behave as described in above paragraph when native client is used.
>>>>
>>>> Does anybody know why native client always sets //findDurable=false//?
>>>>
>>>> BRs,
>>>>
>>>> Jakov
Re: Unable to get behavior described in documentation when using
durable native client
Posted by Jacob Barrett <jb...@pivotal.io>.
Can you confirm that when log level less than debug that the IOException goes away and the client appears to function?
-Jake
> On Apr 17, 2020, at 1:12 AM, Jakov Varenina <ja...@est.tech> wrote:
>
> Hi Jacob,
>
> Thanks for your response!
>
> Regarding GEODE-7944, "Unable to deserialize *membership id* java.io.EOFException" is not logged but thrown, and it breaks processing of QueueConnectionRequest in locator. This reflects in native client with "No locators found" even though they are available. Happens only when native client with subscription is enabled and locator started with --log-level=debug.
>
> I haven't had time to test and analyze in detail native durable client yet. So far I could only confirm that when using native durable client then locator behaves differently than how it is described in documentation (see previous mail) and how java client works:
>
> It seems that native durable client always requests from locator all available servers (redundant= -1, findDurable=false) with QueueConnectionRequest. Locator returns them in QueueConnectionResponse ordered by load (best...worst). While for java durable client, locator use *membership id *from QueueConnectionRequest to locate servers that host client queue and send them back in QueueConnectionResponse as described in previous mail. I expect that native durable client is handling re-connection to same servers queue somehow also, but this has to be investigated yet. Any hints or comments related to this would be really appreciated.
>
> BRs,
>
> Jakov
>
> On 15. 04. 2020. 10:07, Jacob Barrett wrote:
>> Looking back at history the native library has always only ever set that findDurable flag to false. I traced it back to its initial commit. Aside from the annoying log message, does client durable connection work correctly?
>>
>>> On Apr 14, 2020, at 10:56 PM, Jakov Varenina <ja...@est.tech> wrote:
>>>
>>> Hi all,
>>>
>>> Could you please help me understand behavior of the native client when configured as durable?
>>>
>>> I have been working on a bug GEODE-7944 <https://issues.apache.org/jira/browse/GEODE-7944> which results with exception "Unable to deserialize membership id java.io.EOFException" on locator only when debug is enabled. This happens because native client, only when subscription is enabled, sends towards locator QueueConnectionRequest that doesn't encapsulate ClientProxyMembershipID (not properly serialized) and therefore exception occurs when locator tries to deserialize membership id to log it at debug level.
>>>
>>> I was trying to figure out why would locator need ClientProxyMembershipID from native client and found following paragraph in the documentation (copied from https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html):
>>>
>>> /For durable subscriptions, the server locator must be able to
>>> locate the servers that host the queues for the durable client. When
>>> a durable client sends a request, the server locator queries all the
>>> available servers to see if they are hosting the subscription region
>>> queue for the durable client. If the server is located, the client
>>> is connected to the server hosting the subscription region queue./
>>>
>>> Locator behaves as described in above paragraph only when it receives ///QueueConnectionRequest with ///findDurable flag set to "true" //and with valid membership i//d. //I noticed that unlike java client, the native client always sets //findDurable// to //"false" //and therefore locator will never behave as described in above paragraph when native client is used.
>>>
>>> Does anybody know why native client always sets //findDurable=false//?
>>>
>>> BRs,
>>>
>>> Jakov
Re: Unable to get behavior described in documentation when using
durable native client
Posted by Jakov Varenina <ja...@est.tech>.
Hi Jacob,
Thanks for your response!
Regarding GEODE-7944, "Unable to deserialize *membership id*
java.io.EOFException" is not logged but thrown, and it breaks processing
of QueueConnectionRequest in locator. This reflects in native client
with "No locators found" even though they are available. Happens only
when native client with subscription is enabled and locator started with
--log-level=debug.
I haven't had time to test and analyze in detail native durable client
yet. So far I could only confirm that when using native durable client
then locator behaves differently than how it is described in
documentation (see previous mail) and how java client works:
It seems that native durable client always requests from locator all
available servers (redundant= -1, findDurable=false) with
QueueConnectionRequest. Locator returns them in QueueConnectionResponse
ordered by load (best...worst). While for java durable client, locator
use *membership id *from QueueConnectionRequest to locate servers that
host client queue and send them back in QueueConnectionResponse as
described in previous mail. I expect that native durable client is
handling re-connection to same servers queue somehow also, but this has
to be investigated yet. Any hints or comments related to this would be
really appreciated.
BRs,
Jakov
On 15. 04. 2020. 10:07, Jacob Barrett wrote:
> Looking back at history the native library has always only ever set that findDurable flag to false. I traced it back to its initial commit. Aside from the annoying log message, does client durable connection work correctly?
>
>> On Apr 14, 2020, at 10:56 PM, Jakov Varenina <ja...@est.tech> wrote:
>>
>> Hi all,
>>
>> Could you please help me understand behavior of the native client when configured as durable?
>>
>> I have been working on a bug GEODE-7944 <https://issues.apache.org/jira/browse/GEODE-7944> which results with exception "Unable to deserialize membership id java.io.EOFException" on locator only when debug is enabled. This happens because native client, only when subscription is enabled, sends towards locator QueueConnectionRequest that doesn't encapsulate ClientProxyMembershipID (not properly serialized) and therefore exception occurs when locator tries to deserialize membership id to log it at debug level.
>>
>> I was trying to figure out why would locator need ClientProxyMembershipID from native client and found following paragraph in the documentation (copied from https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html):
>>
>> /For durable subscriptions, the server locator must be able to
>> locate the servers that host the queues for the durable client. When
>> a durable client sends a request, the server locator queries all the
>> available servers to see if they are hosting the subscription region
>> queue for the durable client. If the server is located, the client
>> is connected to the server hosting the subscription region queue./
>>
>> Locator behaves as described in above paragraph only when it receives ///QueueConnectionRequest with ///findDurable flag set to "true" //and with valid membership i//d. //I noticed that unlike java client, the native client always sets //findDurable// to //"false" //and therefore locator will never behave as described in above paragraph when native client is used.
>>
>> Does anybody know why native client always sets //findDurable=false//?
>>
>> BRs,
>>
>> Jakov
Re: Unable to get behavior described in documentation when using
durable native client
Posted by Jacob Barrett <jb...@pivotal.io>.
Looking back at history the native library has always only ever set that findDurable flag to false. I traced it back to its initial commit. Aside from the annoying log message, does client durable connection work correctly?
> On Apr 14, 2020, at 10:56 PM, Jakov Varenina <ja...@est.tech> wrote:
>
> Hi all,
>
> Could you please help me understand behavior of the native client when configured as durable?
>
> I have been working on a bug GEODE-7944 <https://issues.apache.org/jira/browse/GEODE-7944> which results with exception "Unable to deserialize membership id java.io.EOFException" on locator only when debug is enabled. This happens because native client, only when subscription is enabled, sends towards locator QueueConnectionRequest that doesn't encapsulate ClientProxyMembershipID (not properly serialized) and therefore exception occurs when locator tries to deserialize membership id to log it at debug level.
>
> I was trying to figure out why would locator need ClientProxyMembershipID from native client and found following paragraph in the documentation (copied from https://geode.apache.org/docs/geode-native/cpp/112/connection-pools/subscription-properties.html):
>
> /For durable subscriptions, the server locator must be able to
> locate the servers that host the queues for the durable client. When
> a durable client sends a request, the server locator queries all the
> available servers to see if they are hosting the subscription region
> queue for the durable client. If the server is located, the client
> is connected to the server hosting the subscription region queue./
>
> Locator behaves as described in above paragraph only when it receives ///QueueConnectionRequest with ///findDurable flag set to "true" //and with valid membership i//d. //I noticed that unlike java client, the native client always sets //findDurable// to //"false" //and therefore locator will never behave as described in above paragraph when native client is used.
>
> Does anybody know why native client always sets //findDurable=false//?
>
> BRs,
>
> Jakov