You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/11/05 05:11:00 UTC
[jira] [Updated] (HDDS-5943) Shutdown ResultHandlerExecutorService
for StorageVolumeChecker
[ https://issues.apache.org/jira/browse/HDDS-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-5943:
---------------------------------
Labels: pull-request-available (was: )
> Shutdown ResultHandlerExecutorService for StorageVolumeChecker
> --------------------------------------------------------------
>
> Key: HDDS-5943
> URL: https://issues.apache.org/jira/browse/HDDS-5943
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Mark Gui
> Assignee: Mark Gui
> Priority: Major
> Labels: pull-request-available
>
> During unit test on my laptop, theres OOM during TestStorageVolumeChecker:
> {code:java}
> Exception in thread "VolumeCheck ResultHandler thread 15" java.lang.OutOfMemoryError: unable to create new native threadException in thread "VolumeCheck ResultHandler thread 15" java.lang.OutOfMemoryError: unable to create new native thread2021-11-05 12:55:13,999 [VolumeCheck ResultHandler thread 44] INFO volume.TestStorageVolumeChecker (TestStorageVolumeChecker.java:schedule(289)) - Returning success for volume check at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) at com.google.common.util.concurrent.ImmediateFuture.addListener(ImmediateFuture.java:47) at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1047)2021-11-05 12:55:13,998 [VolumeCheck ResultHandler thread 57] WARN volume.MutableVolumeSet (MutableVolumeSet.java:lambda$checkVolumeAsync$1(283)) - checkVolumeAsync callback got 1 failed volumes: [/var/folders/_s/q95mqw991pj9c2cz4fr8ytnr0000gn/T/junit1980556976380740035/439b9ee5-1037-4830-ba87-c7f7778b387f/hdds] at org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker.checkVolume(StorageVolumeChecker.java:268) at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.checkVolumeAsync(MutableVolumeSet.java:280) at org.apache.hadoop.ozone.container.common.utils.StorageVolumeUtil.onFailure(StorageVolumeUtil.java:41) at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.writeToContainerFile(KeyValueContainer.java:234) at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerFile(KeyValueContainer.java:255) at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.updateContainerData(KeyValueContainer.java:366) at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.markContainerUnhealthy(KeyValueContainer.java:297) at org.apache.hadoop.ozone.container.common.impl.ContainerSet.lambda$handleVolumeFailures$0(ContainerSet.java:129) at java.lang.Iterable.forEach(Iterable.java:75) at org.apache.hadoop.ozone.container.common.impl.ContainerSet.handleVolumeFailures(ContainerSet.java:126) at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.handleVolumeFailures(OzoneContainer.java:341) at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.handleVolumeFailures(MutableVolumeSet.java:267) at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.lambda$checkVolumeAsync$1(MutableVolumeSet.java:288) at org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.invokeCallback(StorageVolumeChecker.java:366) at org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.cleanup(StorageVolumeChecker.java:359) at org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.onSuccess(StorageVolumeChecker.java:337) at org.apache.hadoop.ozone.container.common.volume.StorageVolumeChecker$ResultHandler.onSuccess(StorageVolumeChecker.java:282) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1080) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
> {code}
> We should shutdown the ResultHandlerExecutorService correctly when we shutdown StorageVolumeChecker to prevent this OOM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org