[jira] [Commented] (GEODE-8687) Durable client is continuously re-registering CQs on all servers when event de-serialization fails causing resource exhaustion on servers

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GEODE-8687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229837#comment-17229837 ] 

ASF GitHub Bot commented on GEODE-8687:
---------------------------------------

jvarenina opened a new pull request #5730:
URL: https://github.com/apache/geode/pull/5730


   * Improves handling of PdxSerializationException on client at the reception
   of events from subscription queue
   
   * Faulty behavior: At the reception of event for which
   PdxSerializationException is thrown the client would always shutdown
   CacheClientUpdater, destroy subscription queue connection
   and try to perform failover to other server in cluster
   
   * Behaviour with this fix: At the reception of event that provoke
   PdxSerializationException client will only log the exception
   
   Thank you for submitting a contribution to Apache Geode.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
   
   - [x] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?
   
   - [x] Is your initial contribution a single, squashed commit?
   
   - [x] Does `gradlew build` run cleanly?
   
   - [x] Have you written or updated unit tests to verify your changes?
   
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   
   ### Note:
   Please ensure that once the PR is submitted, check Concourse for build issues and
   submit an update to your PR as soon as possible. If you need help, please send an
   email to dev@geode.apache.org.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Durable client is continuously re-registering CQs on all servers when event de-serialization fails causing resource exhaustion on servers 
> ------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: GEODE-8687
>                 URL: https://issues.apache.org/jira/browse/GEODE-8687
>             Project: Geode
>          Issue Type: Bug
>          Components: client/server
>    Affects Versions: 1.13.0
>            Reporter: Jakov Varenina
>            Assignee: Jakov Varenina
>            Priority: Major
>         Attachments: deserialzationFault.log
>
>
> When ReflectionBasedAutoSerializer is wrongly/not set it results with serialization exception on client at the reception of the CQ events. Serialization exception isn't logged which is misleading, and is hard to find that actually ReflectionBasedAutoSerializer isn't set correctly. Only log that can be seen is that client/servers subscription connections are closed due to EOF. This is because client destroys subscriptions connections intentionally, but doesn't log reason (PdxSerializationException) that led to this. It would be good that serialization exceptions are logged as error or warn.
> Client destroys subscription connection and perform server fail-over whenever serialization issue occurs. Additionally when subscription connection for particular server fails multiple times then this server is put in deny list for 10 seconds (this is configurable with {{ping-interval}}). After 10s expire the server is removed from list and it is available for subscription connection which will be destroyed again due serialization issue. This will go indefinitely and approx. every 10s in this case the client subscribes to each servers at least once. Due to serialization issue events aren't sent to client and remain in subscription queues.
> Whenever connection fails due to serialization issue and client is not durable then subscription queue is closed and events are lost.
> The biggest problem arises when client is durable. This is because subscription queue remains on server for configurable period of time (e.g. 300s) waiting for client to reconnect. When client perform fail-over to another server it will create new subscription queue using initial image from old queue that is currently paused. This means that all events from old queue will be transferred to new subscription queue hosted by the current primary server. This will happen on all servers and all of them will have copy of the queue even subscription redundancy isn't configured. The problem here is that client will periodically (every 10s in this case) establish connection to each servers, so configured timeout (e.g. 300s) will never expire, but it will be renewed each time client is registered. This could cause a lots of problems since memory and disk usage (if overflow on queue is configured) will increase on all servers.
> You can find in attached logs for the problematic case with durable client :
> vm0              -> locator
> vm1, vm2   -> servers
> vm3              -> durable client with enabled subscription handling CQ events
> vm4              -> client generating traffic that should trigger registered CQ
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)