You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/01/21 02:56:27 UTC
[GitHub] [incubator-dolphinscheduler] brucemen711 opened a new issue #4516: [Question] Spark task hanging when zk out of memory
brucemen711 opened a new issue #4516:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4516
I set spark task (spark streaming) to 100 retries if job failed. But i got task hanging when zookeeper OOM. The log is below:
`java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.jute.BinaryOutputArchive.<init>(BinaryOutputArchive.java:33)
at org.apache.jute.BinaryOutputArchive.getArchive(BinaryOutputArchive.java:38)
at org.apache.zookeeper.ClientCnxn$Packet.createBB(ClientCnxn.java:298)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:115)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[WARN] 2021-01-21 03:04:12.958 org.apache.zookeeper.ClientCnxn:[1108] - Client session timed out, have not heard from server in 27345ms for sessionid 0x1b7b1c3b7000d
[INFO] 2021-01-21 03:04:12.959 org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread:[164] - develop mode is: false
[INFO] 2021-01-21 03:04:12.960 org.apache.zookeeper.ClientCnxn:[1156] - Client session timed out, have not heard from server in 27345ms for sessionid 0x1b7b1c3b7000d, closing socket connection and attempting reconnect
[INFO] 2021-01-21 03:04:12.961 org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread:[182] - exec local path: /tmp/dolphinscheduler/exec/process/3/7/283/870 cleared.
[INFO] 2021-01-21 03:04:13.060 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: SUSPENDED
[INFO] 2021-01-21 03:04:13.061 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: SUSPENDED`
**Which version of DolphinScheduler:**
-[1.3.4]
**Additional context**
Add any other context about the problem here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] brucemen711 commented on issue #4516: [Question] Spark task hanging when zk out of memory
Posted by GitBox <gi...@apache.org>.
brucemen711 commented on issue #4516:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4516#issuecomment-766679768
My machine has 32GB but dolphinscheduler has default settings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong commented on issue #4516: [Question] Spark task hanging when zk out of memory
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #4516:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4516#issuecomment-765883434
What is your machine configuration, memory?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] brucemen711 commented on issue #4516: [Question] Spark task hanging when zk out of memory
Posted by GitBox <gi...@apache.org>.
brucemen711 commented on issue #4516:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4516#issuecomment-766679768
My machine has 32GB but dolphinscheduler has default settings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org