You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by "Rashid Mahmood (Jira)" <ji...@apache.org> on 2020/09/18 15:22:00 UTC
[jira] [Commented] (DIRSERVER-2255) java.lang.OutOfMemoryError:
Java heap space
[ https://issues.apache.org/jira/browse/DIRSERVER-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198396#comment-17198396 ]
Rashid Mahmood commented on DIRSERVER-2255:
-------------------------------------------
We are facing same problem in Multi-Master setup in our Reference Env. Next target is our production environment
We have 2 seperate data centers backed by load balancer. Each data center has a Maste-Slave(Machine1 Master and Machine 2 slave) setup with 20k enrtries.
We need a Master-Master setup between Data Centers, i enabled the consumer on each Data Center's Master instance to poll the other Data Center's Master instance. Those Master instances had already 20K Entries. Then i noticed Replication was not working, any change made on one Master is not replicated on other Master. Memory usage on each Master was reaching to 2GB. Took a heap dump on one instance and found SyncReplSearchListener occupied almost 90% of memory.
What could be the reason???
Couple of more questions:
1). Are both Masters trying to synch all entries? I think so maybe because both newly enabled consumers on each Master are not sending any cookie to the target Master, it makes Master think of sending all entries to consumer.
2). How many consumers one Master can serve, is there any benchmark?
3). Is there any guideline with best practices in setting up Multi-Master for Production System?
4). We have a cron job which makes daily backup of our Data Partition, in case of broken DB on one Master/Slave; should we restore from backup or just create a fresh installation? To make restore with minimun Repk overhead we need to make backup of the cookie entry so that Repl-Server send only those modifications which happend after DB corruption, right???
Our Environment:
Linux Centos 7
ApacheDS 2.0.0.AM24
All consumers are with ads-replrefreshnpersist: TRUE configured.
!9-instances-SyncReplSearchListener .png!
> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
> Key: DIRSERVER-2255
> URL: https://issues.apache.org/jira/browse/DIRSERVER-2255
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: replication
> Affects Versions: 2.0.0.AM25
> Environment: Linux CentOS 7
> Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
> Reporter: li
> Priority: Major
> Attachments: 1.png, 9-instances-SyncReplSearchListener .png, conf.zip, wrapper.conf
>
>
> INFO | jvm 1 | 2018/10/29 12:01:45 | Exception in thread "Thread-4" java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:01:45 | at java.util.Arrays.copyOf(Arrays.java:3236)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:301)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:207)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at java.lang.StringCoding.encode(StringCoding.java:348)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at java.lang.String.getBytes(String.java:941)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1626)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:261)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.api.ldap.model.entry.DefaultAttribute.apply(DefaultAttribute.java:1828)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.api.ldap.model.entry.DefaultEntry.readExternal(DefaultEntry.java:2443)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.ldap.replication.ReplicaEventMessageSerializer.deserialize(ReplicaEventMessageSerializer.java:126)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.deserialize(BPage.java:1188)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.deserialize(BPage.java:81)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:329)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.loadBPage(BPage.java:949)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.childBPage(BPage.java:938)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.insert(BPage.java:380)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BPage.insert(BPage.java:381)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at jdbm.btree.BTree.insert(BTree.java:302)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.put(JdbmTable.java:541)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.log(ReplicaEventLog.java:176)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.ldap.replication.provider.SyncReplSearchListener.entryModified(SyncReplSearchListener.java:314)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.core.event.EventInterceptor.fire(EventInterceptor.java:169)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.core.event.EventInterceptor.modify(EventInterceptor.java:333)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:508)
> INFO | jvm 1 | 2018/10/29 12:01:45 | at org.apache.directory.server.core.subtree.SubentryInterceptor.modify(SubentryInterceptor.java:1249)
> INFO | jvm 1 | 2018/10/29 12:01:47 | Exception in thread "pool-13-thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.directory.api.ldap.model.entry.Value.clone(Value.java:425)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.directory.api.ldap.model.name.Ava.getValue(Ava.java:728)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.directory.api.ldap.model.name.Rdn.<init>(Rdn.java:455)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.directory.api.ldap.model.name.Dn.<init>(Dn.java:197)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.directory.api.ldap.model.entry.DefaultEntry.normalizeDn(DefaultEntry.java:519)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.directory.api.ldap.model.entry.DefaultEntry.<init>(DefaultEntry.java:300)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.directory.ldap.client.api.LdapNetworkConnection.messageReceived(LdapNetworkConnection.java:2757)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:437)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:256)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:49)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1141)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:122)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:650)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:643)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1242)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1231)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> INFO | jvm 1 | 2018/10/29 12:01:47 | at java.lang.Thread.run(Thread.java:748)
> INFO | jvm 1 | 2018/10/29 12:50:50 | Exception in thread "pool-7-thread-5835" java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2018/10/29 12:50:50 | at java.io.BufferedReader.<init>(BufferedReader.java:105)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at java.io.BufferedReader.<init>(BufferedReader.java:116)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.api.ldap.model.schema.parsers.OpenLdapSchemaParser.parseAttributeType(OpenLdapSchemaParser.java:2599)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.api.ldap.model.schema.parsers.AttributeTypeDescriptionSchemaParser.parse(AttributeTypeDescriptionSchemaParser.java:81)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.api.ldap.model.schema.syntaxCheckers.AttributeTypeDescriptionSyntaxChecker.isValidSyntax(AttributeTypeDescriptionSyntaxChecker.java:163)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.api.ldap.model.entry.Value.<init>(Value.java:285)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:449)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1104)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.shared.SchemaService.generateAttributeTypes(SchemaService.java:144)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.shared.SchemaService.generateSchemaSubentry(SchemaService.java:258)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.shared.SchemaService.getSubschemaEntry(SchemaService.java:432)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.schema.SchemaInterceptor.search(SchemaInterceptor.java:1519)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.authz.DefaultAuthorizationInterceptor.search(DefaultAuthorizationInterceptor.java:354)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.authz.AciAuthorizationInterceptor.search(AciAuthorizationInterceptor.java:1247)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.authn.AuthenticationInterceptor.search(AuthenticationInterceptor.java:1394)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.api.interceptor.BaseInterceptor.next(BaseInterceptor.java:598)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.normalization.NormalizationInterceptor.search(NormalizationInterceptor.java:414)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.DefaultOperationManager.search(DefaultOperationManager.java:1781)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.core.shared.DefaultCoreSession.search(DefaultCoreSession.java:1193)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.doSimpleSearch(SearchRequestHandler.java:810)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleIgnoringReferrals(SearchRequestHandler.java:1164)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handleWithReferrals(SearchRequestHandler.java:1262)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:212)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.handlers.request.SearchRequestHandler.handle(SearchRequestHandler.java:92)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:207)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:243)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:223)
> INFO | jvm 1 | 2018/10/29 12:50:50 | at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:1019)
> STATUS | wrapper | 2018/10/29 12:56:37 | TERM trapped. Shutting down.
> INFO | jvm 1 | 2018/10/29 12:56:46 | Error in WrapperListener.stop callback. java.lang.OutOfMemoryError: Java heap space
> INFO | jvm 1 | 2018/10/29 12:56:46 | java.lang.OutOfMemoryError: Java heap space
> INFO | jvm 1 | 2018/10/29 12:56:46 | at java.util.HashMap.resize(HashMap.java:704)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at java.util.HashMap.putVal(HashMap.java:663)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at java.util.HashMap.put(HashMap.java:612)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at jdbm.recman.RecordFile.commit(RecordFile.java:314)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at jdbm.recman.PageManager.close(PageManager.java:329)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at jdbm.recman.BaseRecordManager.close(BaseRecordManager.java:164)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at org.apache.directory.server.ldap.replication.provider.ReplicaEventLog.stop(ReplicaEventLog.java:226)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at org.apache.directory.server.ldap.replication.provider.SyncReplRequestHandler.stop(SyncReplRequestHandler.java:274)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at org.apache.directory.server.ldap.LdapServer.stop(LdapServer.java:668)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at org.apache.directory.server.ApacheDsService.stop(ApacheDsService.java:610)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.stop(ApacheDsTanukiWrapper.java:205)
> INFO | jvm 1 | 2018/10/29 12:56:46 | at org.tanukisoftware.wrapper.WrapperManager$13.run(WrapperManager.java:3134)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@directory.apache.org
For additional commands, e-mail: dev-help@directory.apache.org