You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2021/06/21 07:56:00 UTC
[jira] [Created] (HDDS-5363) Datanode shutdown due to too many bad
volumes in CI
Attila Doroszlai created HDDS-5363:
--------------------------------------
Summary: Datanode shutdown due to too many bad volumes in CI
Key: HDDS-5363
URL: https://issues.apache.org/jira/browse/HDDS-5363
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 1.2.0
Reporter: Attila Doroszlai
_Acceptance (secure)_ check is frequently failing, usually at S3 tests. The root cause is that datanodes are shut down due to too many "bad" volumes.
{noformat:title=S3 Gateway log}
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. Requested 1 blocks
{noformat}
{noformat:title=SCM log}
Pipeline creation failed due to no sufficient healthy datanodes. Required 3. Found 0.
{noformat}
{noformat:title=Datanode log}
datanode_2 | 2021-06-19 13:26:08,010 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/hdds/hdds
datanode_2 | 2021-06-19 13:36:08,013 [Periodic HDDS volume checker] WARN volume.StorageVolumeChecker: checkAllVolumes timed out after 600000 ms
datanode_2 | 2021-06-19 13:36:08,014 [Periodic HDDS volume checker] WARN volume.MutableVolumeSet: checkAllVolumes got 1 failed volumes - [/data/hdds/hdds]
datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] INFO volume.MutableVolumeSet: Moving Volume : /data/hdds/hdds to failed Volumes
datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] ERROR statemachine.DatanodeStateMachine: DatanodeStateMachine Shutdown due to too many bad volumes, check hdds.datanode.failed.data.volumes.tolerated and hdds.datanode.failed.metadata.volumes.tolerated
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org