You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rodrigo Oliveira <ad...@gmail.com> on 2019/07/29 18:34:59 UTC
[ZOOKEEPER] - Error - HEAP MEMORY
Hi,
After 3 days running, my zookeeper showing this error.
2019-07-29 15:10:41,906 [myid:1] - WARN
[RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
Connection broken for id 4332550065071534382, my id = 1, error =
java.io.IOException: Received packet with invalid packet: 824196618
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
2019-07-29 15:10:41,906 [myid:1] - WARN
[RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
Interrupting SendWorker
2019-07-29 15:10:41,907 [myid:1] - WARN
[SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
2019-07-29 15:10:41,907 [myid:1] - WARN
[SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - Send
worker leaving thread id 4332550065071534382 my id = 1
2019-07-29 15:10:41,917 [myid:1] - INFO [/177.55.55.152:3888
:QuorumCnxManager$Listener@888] - Received connection request /
177.55.55.63:53972
2019-07-29 15:10:41,920 [myid:1] - WARN
[RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
Connection broken for id 4332550065071534382, my id = 1, error =
java.io.IOException: Received packet with invalid packet: 840973834
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
2019-07-29 15:10:41,921 [myid:1] - WARN
[RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
Interrupting SendWorker
2019-07-29 15:10:41,922 [myid:1] - WARN
[SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
2019-07-29 15:10:41,922 [myid:1] - WARN
[SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - Send
worker leaving thread id 4332550065071534382 my id = 1
2019-07-29 15:10:41,932 [myid:1] - INFO [/177.55.55.152:3888
:QuorumCnxManager$Listener@888] - Received connection request /
177.55.55.63:38633
2019-07-29 15:10:41,933 [myid:1] - WARN
[RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1176] -
Connection broken for id 4332550065071534638, my id = 1, error =
java.io.IOException: Received packet with invalid packet: 807419402
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
2019-07-29 15:10:41,933 [myid:1] - WARN
[RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1179] -
Interrupting SendWorker
2019-07-29 15:10:41,934 [myid:1] - WARN
[SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1092] -
Interrupted while waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
2019-07-29 15:10:41,935 [myid:1] - WARN
[SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1102] - Send
worker leaving thread id 4332550065071534638 my id = 1
2019-07-29 15:10:41,981 [myid:1] - WARN
[NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of session
0x0: Len error 1008750420
2019-07-29 15:10:41,987 [myid:1] - WARN
[NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of session
0x0: Len error 1008750420
2019-07-29 15:10:41,992 [myid:1] - WARN
[NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of session
0x0: Len error 1008750420
2019-07-29 15:10:42,008 [myid:1] - WARN
[NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of session
0x0: Len error 1008750420
2019-07-29 15:10:45,375 [myid:1] - WARN
[NIOWorkerThread-11:NIOServerCnxn@366] - Unable to read additional data
from client sessionid 0x400044d531804b6, likely client has closed socket
2019-07-29 15:10:46,062 [myid:1] - WARN
[NIOWorkerThread-4:NIOServerCnxn@366] - Unable to read additional data
from client sessionid 0x300044d504c0471, likely client has closed socket
2019-07-29 15:11:12,965 [myid:1] - WARN [qtp1789550256-23:URIUtil@396] -
/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
byte C0 in state 0
2019-07-29 15:11:17,588 [myid:1] - INFO [/177.153.44.152:3888
:QuorumCnxManager$Listener@888] - Received connection request /
177.55.55.63:33721
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid97598.hprof ...
2019-07-29 15:11:17,590 [myid:1] - WARN
[RecvWorker:1:QuorumCnxManager$RecvWorker@1176] - Connection broken for id
1, my id = 1, error =
java.io.IOException: Received packet with invalid packet: 1966420818
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
2019-07-29 15:11:19,452 [myid:1] - WARN
[RecvWorker:1:QuorumCnxManager$RecvWorker@1179] - Interrupting SendWorker
Heap dump file created [638922790 bytes in 3,132 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
# Executing /bin/sh -c "kill -9 97598"...
That's right. Any tips ou tricks to workaround this error? How to increase
the memory in Zookeeper?
Look my zoo.conf.
[15:30:09] root@eddison0001:/zoop/conf$ cat zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/zoop/zookeeperdata
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
oom_heap_dump_dir=/tmp
zookeeper_server_java_heaps=4G
server.1=177.55.55.152:2888:3888
server.2=177.55.55.153:2888:3888
server.3=177.55.55.154:2888:3888
server.4=177.55.55.155:2888:3888
server.5=177.55.55.156:2888:3888
Any workaround?
Regards,
Re: [ZOOKEEPER] - Error - HEAP MEMORY
Posted by Rodrigo Oliveira <ad...@gmail.com>.
Hi,
Follow the process, and more data about my SOLR + ZOOKEEPER.
root 48425 1 26 Jul29 ? 03:00:39 java -server -Xms28g
-Xmx32g -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4
-XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled
-XX:-OmitStackTraceInFastThrow -verbose:gc -XX:+PrintHeapAtGC
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
-XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-Xloggc:/solr/server/logs/solr_gc.log -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -DzkClientTimeout=15000
-DzkHost=177.55.55.152:2181,177.55.55.153:2181,177.55.55.154:2181,
177.55.55.155:2181,177.55.55.156:2181 -Dsolr.log.dir=/solr/server/logs
-Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC
-Djetty.home=/solr/server -Dsolr.solr.home=/solr/server/solr
-Dsolr.data.home= -Dsolr.install.dir=/solr
-Dsolr.default.confdir=/solr/server/solr/configsets/_default/conf -Xss256k
-Dsolr.jetty.https.port=8983 -Dsolr.log.muteconsole
-XX:OnOutOfMemoryError=/solr/bin/oom_solr.sh 8983 /solr/server/logs -jar
start.jar --module=http
root 48163 1 0 Jul29 ? 00:01:33 java
-Dzookeeper.log.dir=/zoop/bin/../logs
-Dzookeeper.log.file=zookeeper-root-server-eddison0001.log
-Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError
-XX:OnOutOfMemoryError=kill -9 %p -cp
/zoop/bin/../zookeeper-server/target/classes:/zoop/bin/../build/classes:/zoop/bin/../zookeeper-server/target/lib/*.jar:/zoop/bin/../build/lib/*.jar:/zoop/bin/../lib/zookeeper-jute-3.5.5.jar:/zoop/bin/../lib/zookeeper-3.5.5.jar:/zoop/bin/../lib/slf4j-log4j12-1.7.25.jar:/zoop/bin/../lib/slf4j-api-1.7.25.jar:/zoop/bin/../lib/netty-all-4.1.29.Final.jar:/zoop/bin/../lib/log4j-1.2.17.jar:/zoop/bin/../lib/json-simple-1.1.1.jar:/zoop/bin/../lib/jline-2.11.jar:/zoop/bin/../lib/jetty-util-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-servlet-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-server-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-security-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-io-9.4.17.v20190418.jar:/zoop/bin/../lib/jetty-http-9.4.17.v20190418.jar:/zoop/bin/../lib/javax.servlet-api-3.1.0.jar:/zoop/bin/../lib/jackson-databind-2.9.8.jar:/zoop/bin/../lib/jackson-core-2.9.8.jar:/zoop/bin/../lib/jackson-annotations-2.9.0.jar:/zoop/bin/../lib/commons-cli-1.2.jar:/zoop/bin/../lib/audience-annotations-0.5.0.jar:/zoop/bin/../zookeeper-*.jar:/zoop/bin/../zookeeper-server/src/main/resources/lib/*.jar:/zoop/bin/../conf:
-*Xmx1000m -Xmx4096m* -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
org.apache.zookeeper.server.quorum.QuorumPeerMain /zoop/conf/zoo.cfg
root 91749 91497 0 09:43 pts/0 00:00:00 grep --color=auto -i zook
[09:43:55] root@eddison0001:~$
Answer the question, look the ZOOKEEPER process, 2 xmx.
By the way, I changed the SOLR (28xms - 32 xmx) because after 5 days ago
using the SOLR, I received the message about Heap Memory in SOLR. Nowadays
I don't have more message about Heap Memory in Solr.
Regards,
Em ter, 30 de jul de 2019 às 08:41, Jörn Franke <jo...@gmail.com>
escreveu:
> 2 xmx does not make sense,
>
> Your heap seems unusual large usually your heap should be much smaller
> than available memory so solr can use it for index caching which is off-heap
>
> > Am 30.07.2019 um 13:25 schrieb Rodrigo Oliveira <
> adamantina.rodrigo@gmail.com>:
> >
> > Hi,
> >
> > My environment have 5 servers with solr + zookeeper in the same hosts.
> >
> > However, I've had 48 RAM each servers (solr - xms 28 gb and xmx - 32 gb).
> >
> > Looking for my servers and process, in zookeepers has xmx 1000 mb and xmx
> > 4096 mb (last item, was change for me).
> >
> > Why 2 values for xmx?
> >
> > Regards,
> >
> > Em ter, 30 de jul de 2019 04:44, Dominique Bejean <
> dominique.bejean@eolya.fr>
> > escreveu:
> >
> >> Hi,
> >>
> >> I don’t find any documentation about the parameter
> >> zookeeper_server_java_heaps
> >> in zoo.cfg.
> >> The way to control java heap size is either the java.env file of the
> >> zookeeper-env.sh file. In zookeeper-env.sh
> >> SERVER_JVMFLAGS="-Xmx=512m"
> >>
> >> How many RAM on your server ?
> >>
> >> Regards
> >>
> >> Dominique
> >>
> >>
> >>
> >>
> >> Le lun. 29 juil. 2019 à 20:35, Rodrigo Oliveira <
> >> adamantina.rodrigo@gmail.com> a écrit :
> >>
> >>> Hi,
> >>>
> >>> After 3 days running, my zookeeper showing this error.
> >>>
> >>> 2019-07-29 15:10:41,906 [myid:1] - WARN
> >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
> >>> Connection broken for id 4332550065071534382, my id = 1, error =
> >>> java.io.IOException: Received packet with invalid packet: 824196618
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> >>> 2019-07-29 15:10:41,906 [myid:1] - WARN
> >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
> >>> Interrupting SendWorker
> >>> 2019-07-29 15:10:41,907 [myid:1] - WARN
> >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
> >>> Interrupted while waiting for message on queue
> >>> java.lang.InterruptedException
> >>> at
> >>>
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> >>> at
> >>>
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> >>> at
> >>>
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> >>> 2019-07-29 15:10:41,907 [myid:1] - WARN
> >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] -
> >> Send
> >>> worker leaving thread id 4332550065071534382 my id = 1
> >>> 2019-07-29 15:10:41,917 [myid:1] - INFO [/177.55.55.152:3888
> >>> :QuorumCnxManager$Listener@888] - Received connection request /
> >>> 177.55.55.63:53972
> >>> 2019-07-29 15:10:41,920 [myid:1] - WARN
> >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
> >>> Connection broken for id 4332550065071534382, my id = 1, error =
> >>> java.io.IOException: Received packet with invalid packet: 840973834
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> >>> 2019-07-29 15:10:41,921 [myid:1] - WARN
> >>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
> >>> Interrupting SendWorker
> >>> 2019-07-29 15:10:41,922 [myid:1] - WARN
> >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
> >>> Interrupted while waiting for message on queue
> >>> java.lang.InterruptedException
> >>> at
> >>>
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> >>> at
> >>>
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> >>> at
> >>>
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> >>> 2019-07-29 15:10:41,922 [myid:1] - WARN
> >>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] -
> >> Send
> >>> worker leaving thread id 4332550065071534382 my id = 1
> >>> 2019-07-29 15:10:41,932 [myid:1] - INFO [/177.55.55.152:3888
> >>> :QuorumCnxManager$Listener@888] - Received connection request /
> >>> 177.55.55.63:38633
> >>> 2019-07-29 15:10:41,933 [myid:1] - WARN
> >>> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1176] -
> >>> Connection broken for id 4332550065071534638, my id = 1, error =
> >>> java.io.IOException: Received packet with invalid packet: 807419402
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> >>> 2019-07-29 15:10:41,933 [myid:1] - WARN
> >>> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1179] -
> >>> Interrupting SendWorker
> >>> 2019-07-29 15:10:41,934 [myid:1] - WARN
> >>> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1092] -
> >>> Interrupted while waiting for message on queue
> >>> java.lang.InterruptedException
> >>> at
> >>>
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> >>> at
> >>>
> >>>
> >>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> >>> at
> >>>
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> >>> 2019-07-29 15:10:41,935 [myid:1] - WARN
> >>> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1102] -
> >> Send
> >>> worker leaving thread id 4332550065071534638 my id = 1
> >>> 2019-07-29 15:10:41,981 [myid:1] - WARN
> >>> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> >>> session
> >>> 0x0: Len error 1008750420
> >>> 2019-07-29 15:10:41,987 [myid:1] - WARN
> >>> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> >>> session
> >>> 0x0: Len error 1008750420
> >>> 2019-07-29 15:10:41,992 [myid:1] - WARN
> >>> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> >>> session
> >>> 0x0: Len error 1008750420
> >>> 2019-07-29 15:10:42,008 [myid:1] - WARN
> >>> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> >>> session
> >>> 0x0: Len error 1008750420
> >>> 2019-07-29 15:10:45,375 [myid:1] - WARN
> >>> [NIOWorkerThread-11:NIOServerCnxn@366] - Unable to read additional
> data
> >>> from client sessionid 0x400044d531804b6, likely client has closed
> socket
> >>> 2019-07-29 15:10:46,062 [myid:1] - WARN
> >>> [NIOWorkerThread-4:NIOServerCnxn@366] - Unable to read additional data
> >>> from client sessionid 0x300044d504c0471, likely client has closed
> socket
> >>> 2019-07-29 15:11:12,965 [myid:1] - WARN [qtp1789550256-23:URIUtil@396
> ]
> >> -
> >>>
> >>>
> >>
> /%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
> >>> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> >>> byte C0 in state 0
> >>> 2019-07-29 15:11:17,588 [myid:1] - INFO [/177.153.44.152:3888
> >>> :QuorumCnxManager$Listener@888] - Received connection request /
> >>> 177.55.55.63:33721
> >>> java.lang.OutOfMemoryError: Java heap space
> >>> Dumping heap to java_pid97598.hprof ...
> >>> 2019-07-29 15:11:17,590 [myid:1] - WARN
> >>> [RecvWorker:1:QuorumCnxManager$RecvWorker@1176] - Connection broken
> for
> >>> id
> >>> 1, my id = 1, error =
> >>> java.io.IOException: Received packet with invalid packet: 1966420818
> >>> at
> >>>
> >>>
> >>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> >>> 2019-07-29 15:11:19,452 [myid:1] - WARN
> >>> [RecvWorker:1:QuorumCnxManager$RecvWorker@1179] - Interrupting
> >> SendWorker
> >>> Heap dump file created [638922790 bytes in 3,132 secs]
> >>> #
> >>> # java.lang.OutOfMemoryError: Java heap space
> >>> # -XX:OnOutOfMemoryError="kill -9 %p"
> >>> # Executing /bin/sh -c "kill -9 97598"...
> >>>
> >>> That's right. Any tips ou tricks to workaround this error? How to
> >> increase
> >>> the memory in Zookeeper?
> >>>
> >>> Look my zoo.conf.
> >>>
> >>> [15:30:09] root@eddison0001:/zoop/conf$ cat zoo.cfg
> >>> # The number of milliseconds of each tick
> >>> tickTime=2000
> >>> # The number of ticks that the initial
> >>> # synchronization phase can take
> >>> initLimit=10
> >>> # The number of ticks that can pass between
> >>> # sending a request and getting an acknowledgement
> >>> syncLimit=5
> >>> # the directory where the snapshot is stored.
> >>> # do not use /tmp for storage, /tmp here is just
> >>> # example sakes.
> >>> dataDir=/zoop/zookeeperdata
> >>> # the port at which the clients will connect
> >>> clientPort=2181
> >>> # the maximum number of client connections.
> >>> # increase this if you need to handle more clients
> >>> #maxClientCnxns=60
> >>> #
> >>> # Be sure to read the maintenance section of the
> >>> # administrator guide before turning on autopurge.
> >>> #
> >>> #
> >>>
> >>
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >>> #
> >>> # The number of snapshots to retain in dataDir
> >>> #autopurge.snapRetainCount=3
> >>> # Purge task interval in hours
> >>> # Set to "0" to disable auto purge feature
> >>> #autopurge.purgeInterval=1
> >>> oom_heap_dump_dir=/tmp
> >>> zookeeper_server_java_heaps=4G
> >>>
> >>> server.1=177.55.55.152:2888:3888
> >>> server.2=177.55.55.153:2888:3888
> >>> server.3=177.55.55.154:2888:3888
> >>> server.4=177.55.55.155:2888:3888
> >>> server.5=177.55.55.156:2888:3888
> >>>
> >>> Any workaround?
> >>>
> >>> Regards,
> >>>
> >>
>
Re: [ZOOKEEPER] - Error - HEAP MEMORY
Posted by Jörn Franke <jo...@gmail.com>.
2 xmx does not make sense,
Your heap seems unusual large usually your heap should be much smaller than available memory so solr can use it for index caching which is off-heap
> Am 30.07.2019 um 13:25 schrieb Rodrigo Oliveira <ad...@gmail.com>:
>
> Hi,
>
> My environment have 5 servers with solr + zookeeper in the same hosts.
>
> However, I've had 48 RAM each servers (solr - xms 28 gb and xmx - 32 gb).
>
> Looking for my servers and process, in zookeepers has xmx 1000 mb and xmx
> 4096 mb (last item, was change for me).
>
> Why 2 values for xmx?
>
> Regards,
>
> Em ter, 30 de jul de 2019 04:44, Dominique Bejean <do...@eolya.fr>
> escreveu:
>
>> Hi,
>>
>> I don’t find any documentation about the parameter
>> zookeeper_server_java_heaps
>> in zoo.cfg.
>> The way to control java heap size is either the java.env file of the
>> zookeeper-env.sh file. In zookeeper-env.sh
>> SERVER_JVMFLAGS="-Xmx=512m"
>>
>> How many RAM on your server ?
>>
>> Regards
>>
>> Dominique
>>
>>
>>
>>
>> Le lun. 29 juil. 2019 à 20:35, Rodrigo Oliveira <
>> adamantina.rodrigo@gmail.com> a écrit :
>>
>>> Hi,
>>>
>>> After 3 days running, my zookeeper showing this error.
>>>
>>> 2019-07-29 15:10:41,906 [myid:1] - WARN
>>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
>>> Connection broken for id 4332550065071534382, my id = 1, error =
>>> java.io.IOException: Received packet with invalid packet: 824196618
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
>>> 2019-07-29 15:10:41,906 [myid:1] - WARN
>>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
>>> Interrupting SendWorker
>>> 2019-07-29 15:10:41,907 [myid:1] - WARN
>>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
>>> Interrupted while waiting for message on queue
>>> java.lang.InterruptedException
>>> at
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>>> at
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>> at
>>> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
>>> 2019-07-29 15:10:41,907 [myid:1] - WARN
>>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] -
>> Send
>>> worker leaving thread id 4332550065071534382 my id = 1
>>> 2019-07-29 15:10:41,917 [myid:1] - INFO [/177.55.55.152:3888
>>> :QuorumCnxManager$Listener@888] - Received connection request /
>>> 177.55.55.63:53972
>>> 2019-07-29 15:10:41,920 [myid:1] - WARN
>>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
>>> Connection broken for id 4332550065071534382, my id = 1, error =
>>> java.io.IOException: Received packet with invalid packet: 840973834
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
>>> 2019-07-29 15:10:41,921 [myid:1] - WARN
>>> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
>>> Interrupting SendWorker
>>> 2019-07-29 15:10:41,922 [myid:1] - WARN
>>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
>>> Interrupted while waiting for message on queue
>>> java.lang.InterruptedException
>>> at
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>>> at
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>> at
>>> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
>>> 2019-07-29 15:10:41,922 [myid:1] - WARN
>>> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] -
>> Send
>>> worker leaving thread id 4332550065071534382 my id = 1
>>> 2019-07-29 15:10:41,932 [myid:1] - INFO [/177.55.55.152:3888
>>> :QuorumCnxManager$Listener@888] - Received connection request /
>>> 177.55.55.63:38633
>>> 2019-07-29 15:10:41,933 [myid:1] - WARN
>>> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1176] -
>>> Connection broken for id 4332550065071534638, my id = 1, error =
>>> java.io.IOException: Received packet with invalid packet: 807419402
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
>>> 2019-07-29 15:10:41,933 [myid:1] - WARN
>>> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1179] -
>>> Interrupting SendWorker
>>> 2019-07-29 15:10:41,934 [myid:1] - WARN
>>> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1092] -
>>> Interrupted while waiting for message on queue
>>> java.lang.InterruptedException
>>> at
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>>> at
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>> at
>>> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
>>> 2019-07-29 15:10:41,935 [myid:1] - WARN
>>> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1102] -
>> Send
>>> worker leaving thread id 4332550065071534638 my id = 1
>>> 2019-07-29 15:10:41,981 [myid:1] - WARN
>>> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
>>> session
>>> 0x0: Len error 1008750420
>>> 2019-07-29 15:10:41,987 [myid:1] - WARN
>>> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
>>> session
>>> 0x0: Len error 1008750420
>>> 2019-07-29 15:10:41,992 [myid:1] - WARN
>>> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
>>> session
>>> 0x0: Len error 1008750420
>>> 2019-07-29 15:10:42,008 [myid:1] - WARN
>>> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
>>> session
>>> 0x0: Len error 1008750420
>>> 2019-07-29 15:10:45,375 [myid:1] - WARN
>>> [NIOWorkerThread-11:NIOServerCnxn@366] - Unable to read additional data
>>> from client sessionid 0x400044d531804b6, likely client has closed socket
>>> 2019-07-29 15:10:46,062 [myid:1] - WARN
>>> [NIOWorkerThread-4:NIOServerCnxn@366] - Unable to read additional data
>>> from client sessionid 0x300044d504c0471, likely client has closed socket
>>> 2019-07-29 15:11:12,965 [myid:1] - WARN [qtp1789550256-23:URIUtil@396]
>> -
>>>
>>>
>> /%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
>>> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
>>> byte C0 in state 0
>>> 2019-07-29 15:11:17,588 [myid:1] - INFO [/177.153.44.152:3888
>>> :QuorumCnxManager$Listener@888] - Received connection request /
>>> 177.55.55.63:33721
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid97598.hprof ...
>>> 2019-07-29 15:11:17,590 [myid:1] - WARN
>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@1176] - Connection broken for
>>> id
>>> 1, my id = 1, error =
>>> java.io.IOException: Received packet with invalid packet: 1966420818
>>> at
>>>
>>>
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
>>> 2019-07-29 15:11:19,452 [myid:1] - WARN
>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@1179] - Interrupting
>> SendWorker
>>> Heap dump file created [638922790 bytes in 3,132 secs]
>>> #
>>> # java.lang.OutOfMemoryError: Java heap space
>>> # -XX:OnOutOfMemoryError="kill -9 %p"
>>> # Executing /bin/sh -c "kill -9 97598"...
>>>
>>> That's right. Any tips ou tricks to workaround this error? How to
>> increase
>>> the memory in Zookeeper?
>>>
>>> Look my zoo.conf.
>>>
>>> [15:30:09] root@eddison0001:/zoop/conf$ cat zoo.cfg
>>> # The number of milliseconds of each tick
>>> tickTime=2000
>>> # The number of ticks that the initial
>>> # synchronization phase can take
>>> initLimit=10
>>> # The number of ticks that can pass between
>>> # sending a request and getting an acknowledgement
>>> syncLimit=5
>>> # the directory where the snapshot is stored.
>>> # do not use /tmp for storage, /tmp here is just
>>> # example sakes.
>>> dataDir=/zoop/zookeeperdata
>>> # the port at which the clients will connect
>>> clientPort=2181
>>> # the maximum number of client connections.
>>> # increase this if you need to handle more clients
>>> #maxClientCnxns=60
>>> #
>>> # Be sure to read the maintenance section of the
>>> # administrator guide before turning on autopurge.
>>> #
>>> #
>>>
>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>> #
>>> # The number of snapshots to retain in dataDir
>>> #autopurge.snapRetainCount=3
>>> # Purge task interval in hours
>>> # Set to "0" to disable auto purge feature
>>> #autopurge.purgeInterval=1
>>> oom_heap_dump_dir=/tmp
>>> zookeeper_server_java_heaps=4G
>>>
>>> server.1=177.55.55.152:2888:3888
>>> server.2=177.55.55.153:2888:3888
>>> server.3=177.55.55.154:2888:3888
>>> server.4=177.55.55.155:2888:3888
>>> server.5=177.55.55.156:2888:3888
>>>
>>> Any workaround?
>>>
>>> Regards,
>>>
>>
Re: [ZOOKEEPER] - Error - HEAP MEMORY
Posted by Rodrigo Oliveira <ad...@gmail.com>.
Hi,
My environment have 5 servers with solr + zookeeper in the same hosts.
However, I've had 48 RAM each servers (solr - xms 28 gb and xmx - 32 gb).
Looking for my servers and process, in zookeepers has xmx 1000 mb and xmx
4096 mb (last item, was change for me).
Why 2 values for xmx?
Regards,
Em ter, 30 de jul de 2019 04:44, Dominique Bejean <do...@eolya.fr>
escreveu:
> Hi,
>
> I don’t find any documentation about the parameter
> zookeeper_server_java_heaps
> in zoo.cfg.
> The way to control java heap size is either the java.env file of the
> zookeeper-env.sh file. In zookeeper-env.sh
> SERVER_JVMFLAGS="-Xmx=512m"
>
> How many RAM on your server ?
>
> Regards
>
> Dominique
>
>
>
>
> Le lun. 29 juil. 2019 à 20:35, Rodrigo Oliveira <
> adamantina.rodrigo@gmail.com> a écrit :
>
> > Hi,
> >
> > After 3 days running, my zookeeper showing this error.
> >
> > 2019-07-29 15:10:41,906 [myid:1] - WARN
> > [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
> > Connection broken for id 4332550065071534382, my id = 1, error =
> > java.io.IOException: Received packet with invalid packet: 824196618
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> > 2019-07-29 15:10:41,906 [myid:1] - WARN
> > [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
> > Interrupting SendWorker
> > 2019-07-29 15:10:41,907 [myid:1] - WARN
> > [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
> > Interrupted while waiting for message on queue
> > java.lang.InterruptedException
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> > at
> > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> > 2019-07-29 15:10:41,907 [myid:1] - WARN
> > [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] -
> Send
> > worker leaving thread id 4332550065071534382 my id = 1
> > 2019-07-29 15:10:41,917 [myid:1] - INFO [/177.55.55.152:3888
> > :QuorumCnxManager$Listener@888] - Received connection request /
> > 177.55.55.63:53972
> > 2019-07-29 15:10:41,920 [myid:1] - WARN
> > [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
> > Connection broken for id 4332550065071534382, my id = 1, error =
> > java.io.IOException: Received packet with invalid packet: 840973834
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> > 2019-07-29 15:10:41,921 [myid:1] - WARN
> > [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
> > Interrupting SendWorker
> > 2019-07-29 15:10:41,922 [myid:1] - WARN
> > [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
> > Interrupted while waiting for message on queue
> > java.lang.InterruptedException
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> > at
> > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> > 2019-07-29 15:10:41,922 [myid:1] - WARN
> > [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] -
> Send
> > worker leaving thread id 4332550065071534382 my id = 1
> > 2019-07-29 15:10:41,932 [myid:1] - INFO [/177.55.55.152:3888
> > :QuorumCnxManager$Listener@888] - Received connection request /
> > 177.55.55.63:38633
> > 2019-07-29 15:10:41,933 [myid:1] - WARN
> > [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1176] -
> > Connection broken for id 4332550065071534638, my id = 1, error =
> > java.io.IOException: Received packet with invalid packet: 807419402
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> > 2019-07-29 15:10:41,933 [myid:1] - WARN
> > [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1179] -
> > Interrupting SendWorker
> > 2019-07-29 15:10:41,934 [myid:1] - WARN
> > [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1092] -
> > Interrupted while waiting for message on queue
> > java.lang.InterruptedException
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> > at
> >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> > at
> > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> > 2019-07-29 15:10:41,935 [myid:1] - WARN
> > [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1102] -
> Send
> > worker leaving thread id 4332550065071534638 my id = 1
> > 2019-07-29 15:10:41,981 [myid:1] - WARN
> > [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> > session
> > 0x0: Len error 1008750420
> > 2019-07-29 15:10:41,987 [myid:1] - WARN
> > [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> > session
> > 0x0: Len error 1008750420
> > 2019-07-29 15:10:41,992 [myid:1] - WARN
> > [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> > session
> > 0x0: Len error 1008750420
> > 2019-07-29 15:10:42,008 [myid:1] - WARN
> > [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> > session
> > 0x0: Len error 1008750420
> > 2019-07-29 15:10:45,375 [myid:1] - WARN
> > [NIOWorkerThread-11:NIOServerCnxn@366] - Unable to read additional data
> > from client sessionid 0x400044d531804b6, likely client has closed socket
> > 2019-07-29 15:10:46,062 [myid:1] - WARN
> > [NIOWorkerThread-4:NIOServerCnxn@366] - Unable to read additional data
> > from client sessionid 0x300044d504c0471, likely client has closed socket
> > 2019-07-29 15:11:12,965 [myid:1] - WARN [qtp1789550256-23:URIUtil@396]
> -
> >
> >
> /%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
> > org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> > byte C0 in state 0
> > 2019-07-29 15:11:17,588 [myid:1] - INFO [/177.153.44.152:3888
> > :QuorumCnxManager$Listener@888] - Received connection request /
> > 177.55.55.63:33721
> > java.lang.OutOfMemoryError: Java heap space
> > Dumping heap to java_pid97598.hprof ...
> > 2019-07-29 15:11:17,590 [myid:1] - WARN
> > [RecvWorker:1:QuorumCnxManager$RecvWorker@1176] - Connection broken for
> > id
> > 1, my id = 1, error =
> > java.io.IOException: Received packet with invalid packet: 1966420818
> > at
> >
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> > 2019-07-29 15:11:19,452 [myid:1] - WARN
> > [RecvWorker:1:QuorumCnxManager$RecvWorker@1179] - Interrupting
> SendWorker
> > Heap dump file created [638922790 bytes in 3,132 secs]
> > #
> > # java.lang.OutOfMemoryError: Java heap space
> > # -XX:OnOutOfMemoryError="kill -9 %p"
> > # Executing /bin/sh -c "kill -9 97598"...
> >
> > That's right. Any tips ou tricks to workaround this error? How to
> increase
> > the memory in Zookeeper?
> >
> > Look my zoo.conf.
> >
> > [15:30:09] root@eddison0001:/zoop/conf$ cat zoo.cfg
> > # The number of milliseconds of each tick
> > tickTime=2000
> > # The number of ticks that the initial
> > # synchronization phase can take
> > initLimit=10
> > # The number of ticks that can pass between
> > # sending a request and getting an acknowledgement
> > syncLimit=5
> > # the directory where the snapshot is stored.
> > # do not use /tmp for storage, /tmp here is just
> > # example sakes.
> > dataDir=/zoop/zookeeperdata
> > # the port at which the clients will connect
> > clientPort=2181
> > # the maximum number of client connections.
> > # increase this if you need to handle more clients
> > #maxClientCnxns=60
> > #
> > # Be sure to read the maintenance section of the
> > # administrator guide before turning on autopurge.
> > #
> > #
> >
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> > #
> > # The number of snapshots to retain in dataDir
> > #autopurge.snapRetainCount=3
> > # Purge task interval in hours
> > # Set to "0" to disable auto purge feature
> > #autopurge.purgeInterval=1
> > oom_heap_dump_dir=/tmp
> > zookeeper_server_java_heaps=4G
> >
> > server.1=177.55.55.152:2888:3888
> > server.2=177.55.55.153:2888:3888
> > server.3=177.55.55.154:2888:3888
> > server.4=177.55.55.155:2888:3888
> > server.5=177.55.55.156:2888:3888
> >
> > Any workaround?
> >
> > Regards,
> >
>
Re: [ZOOKEEPER] - Error - HEAP MEMORY
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
I don’t find any documentation about the parameter zookeeper_server_java_heaps
in zoo.cfg.
The way to control java heap size is either the java.env file of the
zookeeper-env.sh file. In zookeeper-env.sh
SERVER_JVMFLAGS="-Xmx=512m"
How many RAM on your server ?
Regards
Dominique
Le lun. 29 juil. 2019 à 20:35, Rodrigo Oliveira <
adamantina.rodrigo@gmail.com> a écrit :
> Hi,
>
> After 3 days running, my zookeeper showing this error.
>
> 2019-07-29 15:10:41,906 [myid:1] - WARN
> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
> Connection broken for id 4332550065071534382, my id = 1, error =
> java.io.IOException: Received packet with invalid packet: 824196618
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> 2019-07-29 15:10:41,906 [myid:1] - WARN
> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
> Interrupting SendWorker
> 2019-07-29 15:10:41,907 [myid:1] - WARN
> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
> Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> 2019-07-29 15:10:41,907 [myid:1] - WARN
> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - Send
> worker leaving thread id 4332550065071534382 my id = 1
> 2019-07-29 15:10:41,917 [myid:1] - INFO [/177.55.55.152:3888
> :QuorumCnxManager$Listener@888] - Received connection request /
> 177.55.55.63:53972
> 2019-07-29 15:10:41,920 [myid:1] - WARN
> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1176] -
> Connection broken for id 4332550065071534382, my id = 1, error =
> java.io.IOException: Received packet with invalid packet: 840973834
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> 2019-07-29 15:10:41,921 [myid:1] - WARN
> [RecvWorker:4332550065071534382:QuorumCnxManager$RecvWorker@1179] -
> Interrupting SendWorker
> 2019-07-29 15:10:41,922 [myid:1] - WARN
> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1092] -
> Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> 2019-07-29 15:10:41,922 [myid:1] - WARN
> [SendWorker:4332550065071534382:QuorumCnxManager$SendWorker@1102] - Send
> worker leaving thread id 4332550065071534382 my id = 1
> 2019-07-29 15:10:41,932 [myid:1] - INFO [/177.55.55.152:3888
> :QuorumCnxManager$Listener@888] - Received connection request /
> 177.55.55.63:38633
> 2019-07-29 15:10:41,933 [myid:1] - WARN
> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1176] -
> Connection broken for id 4332550065071534638, my id = 1, error =
> java.io.IOException: Received packet with invalid packet: 807419402
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> 2019-07-29 15:10:41,933 [myid:1] - WARN
> [RecvWorker:4332550065071534638:QuorumCnxManager$RecvWorker@1179] -
> Interrupting SendWorker
> 2019-07-29 15:10:41,934 [myid:1] - WARN
> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1092] -
> Interrupted while waiting for message on queue
> java.lang.InterruptedException
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1243)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:78)
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1080)
> 2019-07-29 15:10:41,935 [myid:1] - WARN
> [SendWorker:4332550065071534638:QuorumCnxManager$SendWorker@1102] - Send
> worker leaving thread id 4332550065071534638 my id = 1
> 2019-07-29 15:10:41,981 [myid:1] - WARN
> [NIOWorkerThread-6:NIOServerCnxn@370] - Exception causing close of
> session
> 0x0: Len error 1008750420
> 2019-07-29 15:10:41,987 [myid:1] - WARN
> [NIOWorkerThread-7:NIOServerCnxn@370] - Exception causing close of
> session
> 0x0: Len error 1008750420
> 2019-07-29 15:10:41,992 [myid:1] - WARN
> [NIOWorkerThread-1:NIOServerCnxn@370] - Exception causing close of
> session
> 0x0: Len error 1008750420
> 2019-07-29 15:10:42,008 [myid:1] - WARN
> [NIOWorkerThread-3:NIOServerCnxn@370] - Exception causing close of
> session
> 0x0: Len error 1008750420
> 2019-07-29 15:10:45,375 [myid:1] - WARN
> [NIOWorkerThread-11:NIOServerCnxn@366] - Unable to read additional data
> from client sessionid 0x400044d531804b6, likely client has closed socket
> 2019-07-29 15:10:46,062 [myid:1] - WARN
> [NIOWorkerThread-4:NIOServerCnxn@366] - Unable to read additional data
> from client sessionid 0x300044d504c0471, likely client has closed socket
> 2019-07-29 15:11:12,965 [myid:1] - WARN [qtp1789550256-23:URIUtil@396] -
>
> /%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> byte C0 in state 0
> 2019-07-29 15:11:17,588 [myid:1] - INFO [/177.153.44.152:3888
> :QuorumCnxManager$Listener@888] - Received connection request /
> 177.55.55.63:33721
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid97598.hprof ...
> 2019-07-29 15:11:17,590 [myid:1] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@1176] - Connection broken for
> id
> 1, my id = 1, error =
> java.io.IOException: Received packet with invalid packet: 1966420818
> at
>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1163)
> 2019-07-29 15:11:19,452 [myid:1] - WARN
> [RecvWorker:1:QuorumCnxManager$RecvWorker@1179] - Interrupting SendWorker
> Heap dump file created [638922790 bytes in 3,132 secs]
> #
> # java.lang.OutOfMemoryError: Java heap space
> # -XX:OnOutOfMemoryError="kill -9 %p"
> # Executing /bin/sh -c "kill -9 97598"...
>
> That's right. Any tips ou tricks to workaround this error? How to increase
> the memory in Zookeeper?
>
> Look my zoo.conf.
>
> [15:30:09] root@eddison0001:/zoop/conf$ cat zoo.cfg
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=/zoop/zookeeperdata
> # the port at which the clients will connect
> clientPort=2181
> # the maximum number of client connections.
> # increase this if you need to handle more clients
> #maxClientCnxns=60
> #
> # Be sure to read the maintenance section of the
> # administrator guide before turning on autopurge.
> #
> #
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> #
> # The number of snapshots to retain in dataDir
> #autopurge.snapRetainCount=3
> # Purge task interval in hours
> # Set to "0" to disable auto purge feature
> #autopurge.purgeInterval=1
> oom_heap_dump_dir=/tmp
> zookeeper_server_java_heaps=4G
>
> server.1=177.55.55.152:2888:3888
> server.2=177.55.55.153:2888:3888
> server.3=177.55.55.154:2888:3888
> server.4=177.55.55.155:2888:3888
> server.5=177.55.55.156:2888:3888
>
> Any workaround?
>
> Regards,
>