You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Mangesh Deshmukh (JIRA)" <ji...@apache.org> on 2017/09/28 19:56:00 UTC
[jira] [Comment Edited] (GEODE-3709) Geode Version: 1.1.1 In one
of the project we a...
[ https://issues.apache.org/jira/browse/GEODE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184742#comment-16184742 ]
Mangesh Deshmukh edited comment on GEODE-3709 at 9/28/17 7:55 PM:
------------------------------------------------------------------
Here are answers to the questions you have posed above:
Q: What is your client CacheListener doing?
A: We have not configured any cache listener. I assume, client must be using a default one.
Q: Does the network traffic stop completely or slow to a crawl?
A: No. Other traffic is quite normal. We can see this from other network related stats.
I will be attaching client and server stats. However, we do not have thread dumps. I see the value of thread dumps in this issue. Next time this happens, will make sure that we take thread dumps and upload it.
client: dca-prd-rstc12 File: gf-rest-stats-12-05.gfs
server: dca-prd-gdcs11 File: myStatisticsArchiveFile-04-01.gfs.
On server, look for CacheClientProxyStatistics - ....10.55.12.224(17709:loaner).....
Region in question: CircularStoreMap
was (Author: mangeshd):
Here are answers to the questions you have posed above:
Q: What is your client CacheListener doing?
A: We have not configured any cache listener. I assume, client must be using a default one.
Q: Does the network traffic stop completely or slow to a crawl?
A: No. Other traffic is quite normal. We can see this from other network related stats.
I will be attaching client and server stats. However, we do not have thread dumps. I see the value of thread dumps in this issue. Next time this happens, will make sure that we take thread dumps and upload it.
client: dca-prd-rstc12 File: gf-rest-stats-12-05.gfs
server: dca-prd-gdcs11 File: myStatisticsArchiveFile-04-01.gfs.
On server, look for CacheClientProxyStatistics - ....10.55.12.224(17709:loaner).....
> Geode Version: 1.1.1 In one of the project we a...
> -----------------------------------------------------
>
> Key: GEODE-3709
> URL: https://issues.apache.org/jira/browse/GEODE-3709
> Project: Geode
> Issue Type: Improvement
> Components: client queues
> Reporter: Gregory Chase
> Attachments: gf-rest-stats-12-05.gfs, myStatisticsArchiveFile-04-01.gfs
>
>
> Geode Version: 1.1.1
> In one of the project we are using Geode. Here is a summary of how we use it.
> - Geode servers have multiple regions.
> - Clients subscribe to the data from these regions.
> - Clients subscribe interest in all the entries, therefore they get updates about all the entries from creation to modification to deletion.
> - One of the regions usually has 5-10 million entries with a TTL of 24 hours. Most entries are added in an hour's span one after other. So when TTL kicks in, they are often destroyed in an hour.
> Problem:
> Every now and then we observe following message:
> Client queue for _gfe_non_durable_client_with_id_x.x.x.x(14229:loner):42754:e4266fc4_2_queue client is full.
> This seems to happen when the TTL kicks in on the region with 5-10 million entries. Entries start getting evicted (deleted); the updates (destroys) now must be sent to clients. We see that the updates do happen for a while but suddenly the updates stop and the queue size starts growing. This is becoming a major issue for smooth functioning of our production setup. Any help will be much appreciated.
> I did some ground work by downloading and looking at the code. I see reference to 2 issues #37581, #51400. But I am unable to view actual JIRA tickets (needs login credentials) Hopefully, it helps someone looking at the issue.
> Here is the pertinent code:
> @Override
> @edu.umd.cs.findbugs.annotations.SuppressWarnings("TLW_TWO_LOCK_WAIT")
> void checkQueueSizeConstraint() throws InterruptedException {
> if (this.haContainer instanceof HAContainerMap && isPrimary()) { // Fix for bug 39413
> if (Thread.interrupted())
> throw new InterruptedException();
> synchronized (this.putGuard) {
> if (putPermits <= 0) {
> synchronized (this.permitMon) {
> if (reconcilePutPermits() <= 0) {
> if (region.getSystem().getConfig().getRemoveUnresponsiveClient()) {
> isClientSlowReciever = true;
> } else {
> try {
> long logFrequency = CacheClientNotifier.DEFAULT_LOG_FREQUENCY;
> CacheClientNotifier ccn = CacheClientNotifier.getInstance();
> if (ccn != null) { // check needed for junit tests
> logFrequency = ccn.getLogFrequency();
> }
> if ((this.maxQueueSizeHitCount % logFrequency) == 0) {
> logger.warn(LocalizedMessage.create(
> LocalizedStrings.HARegionQueue_CLIENT_QUEUE_FOR_0_IS_FULL,
> new Object[] {region.getName()}));
> this.maxQueueSizeHitCount = 0;
> }
> ++this.maxQueueSizeHitCount;
> this.region.checkReadiness(); // fix for bug 37581
> // TODO: wait called while holding two locks
> this.permitMon.wait(CacheClientNotifier.eventEnqueueWaitTime);
> this.region.checkReadiness(); // fix for bug 37581
> // Fix for #51400. Allow the queue to grow beyond its
> // capacity/maxQueueSize, if it is taking a long time to
> // drain the queue, either due to a slower client or the
> // deadlock scenario mentioned in the ticket.
> reconcilePutPermits();
> if ((this.maxQueueSizeHitCount % logFrequency) == 1) {
> logger.info(LocalizedMessage
> .create(LocalizedStrings.HARegionQueue_RESUMING_WITH_PROCESSING_PUTS));
> }
> } catch (InterruptedException ex) {
> // TODO: The line below is meaningless. Comment it out later
> this.permitMon.notifyAll();
> throw ex;
> }
> }
> }
> } // synchronized (this.permitMon)
> } // if (putPermits <= 0)
> --putPermits;
> } // synchronized (this.putGuard)
> }
> }
> *Reporter*: Mangesh Deshmukh
> *E-mail*: [mailto:mdeshmukh@quotient.com]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)