You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by "4devwithgit (via GitHub)" <gi...@apache.org> on 2023/04/04 10:35:53 UTC
[GitHub] [mina-sshd] 4devwithgit opened a new issue, #354: OOM error with apache mina client, thousands of threads at sun/misc/Unsafe.park(Native Method)
4devwithgit opened a new issue, #354:
URL: https://github.com/apache/mina-sshd/issues/354
### Version
2.7.0
### Bug description
We are using Apache mina jars v2.7.0 in our application. Our customer on this version, reported the out of memory(OOM) issue with the Apache mina sftp client, and there is a build of 7000 plus threads of sshd-SschClient
***WARNING*** Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to allocate JNIEnv" received
Here is the thread dump, which you use to analyze the issue, https://syncsortinc-my.sharepoint.com/:t:/r/personal/dev_sahu_syncsort_com/Documents/1/_2023/defects/L3REQ-43331_GENPACT/javacore.txt?csf=1&web=1&e=mdPeac
We can see there are 7000+ threads of which majority are for "sshd-SshClient" like,
**sshd-SshClient**[154d0fa0]-timer-thread-1
### Actual behavior
Getting OOM error with apache mina sftp client
### Expected behavior
it should be able to process without OOM.
### Relevant log output
```Shell
From the thread dump
Thread Name sshd-SshClient[154d0fa0]-timer-thread-1
State Runnable
Java Stack at sun/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:226(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2089(Compiled Code))
at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1104(Compiled Code))
at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:820(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1085(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635(Compiled Code))
at java/lang/Thread.run(Thread.java:822(Compiled Code))
Native Stack ZwWaitForSingleObject+0x14 (0x00007FFACF1B0874 [ntdll+0xa0874])
WaitForSingleObjectEx+0x93 (0x00007FFACBBC43B3 [KERNELBASE+0x343b3])
monitor_wait_original+0x917 (omrthread.c:4679, 0x00007FFAB7C47237 [J9THR29+0x7237])
omrthread_monitor_wait+0x43 (omrthread.c:4395, 0x00007FFAB7C42AD3 [J9THR29+0x2ad3])
internalAcquireVMAccessNoMutexWithMask+0x2fc (vmaccess.cpp:384, 0x00007FFAB56E1A6C [j9vm29+0x61a6c])
internalAcquireVMAccessClearStatus+0x98 (vmaccess.cpp:336, 0x00007FFAB56E15D8 [j9vm29+0x615d8])
threadParkImpl+0x19e (threadpark.c:98, 0x00007FFAB5738D3E [j9vm29+0xb8d3e])
(0x00007FFAA1A30609)
(0x0000000000000003)
(0x0000000001AE3D00)
(0x000000064E3BD6A8)
(0x000000016591FB80)
(0x00000000017E9C00)
(0x0000000008952800)
(0x000000064E3BD6A8)
```
### Other information
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
[GitHub] [mina-sshd] 4devwithgit commented on issue #354: OOM error with apache mina client, thousands of threads at sun/misc/Unsafe.park(Native Method)
Posted by "4devwithgit (via GitHub)" <gi...@apache.org>.
4devwithgit commented on issue #354:
URL: https://github.com/apache/mina-sshd/issues/354#issuecomment-1497215083
Okay. I had given open permission, but there may be some issue for the file. Let me know if there is any other way to share the dump, which is 27 MB file.
Wanted to know, what's the expected behavior for the sshd-SshClient threads under heavy load? What's the internal pool size to which it will be restricted, and what are the default time out after which they get killed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
[GitHub] [mina-sshd] tomaswolf commented on issue #354: OOM error with apache mina client, thousands of threads at sun/misc/Unsafe.park(Native Method)
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #354:
URL: https://github.com/apache/mina-sshd/issues/354#issuecomment-1565675788
These threads normally terminate when the `SshClient` is closed. If your code calls `SshClient.setSchedulerExecutor()`, then your code is responsible for shutting it down.
Sounds like the code is using multiple clients, and doesn't close them properly when they're no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
[GitHub] [mina-sshd] tomaswolf commented on issue #354: OOM error with apache mina client, thousands of threads at sun/misc/Unsafe.park(Native Method)
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf commented on issue #354:
URL: https://github.com/apache/mina-sshd/issues/354#issuecomment-1496564870
A private sharepoint link doesn't help. But anyway: there have been fixes since 2.7.0 that should have resolved some of these issues. Try with 2.9.2 or with current master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org
[GitHub] [mina-sshd] tomaswolf closed issue #354: OOM error with apache mina client, thousands of threads at sun/misc/Unsafe.park(Native Method)
Posted by "tomaswolf (via GitHub)" <gi...@apache.org>.
tomaswolf closed issue #354: OOM error with apache mina client, thousands of threads at sun/misc/Unsafe.park(Native Method)
URL: https://github.com/apache/mina-sshd/issues/354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@mina.apache.org
For additional commands, e-mail: dev-help@mina.apache.org