You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2021/06/21 10:07:00 UTC
[jira] [Assigned] (HDDS-5363) Datanode shutdown due to too many bad
volumes in CI
[ https://issues.apache.org/jira/browse/HDDS-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Doroszlai reassigned HDDS-5363:
--------------------------------------
Assignee: Attila Doroszlai
> Datanode shutdown due to too many bad volumes in CI
> ---------------------------------------------------
>
> Key: HDDS-5363
> URL: https://issues.apache.org/jira/browse/HDDS-5363
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 1.2.0
> Reporter: Attila Doroszlai
> Assignee: Attila Doroszlai
> Priority: Blocker
>
> _Acceptance (secure)_ check is frequently failing, usually at S3 tests. The root cause is that datanodes are shut down due to too many "bad" volumes.
> {noformat:title=S3 Gateway log}
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Allocated 0 blocks. Requested 1 blocks
> {noformat}
> {noformat:title=SCM log}
> Pipeline creation failed due to no sufficient healthy datanodes. Required 3. Found 0.
> {noformat}
> {noformat:title=Datanode log}
> datanode_2 | 2021-06-19 13:26:08,010 [Periodic HDDS volume checker] INFO volume.StorageVolumeChecker: Scheduled health check for volume /data/hdds/hdds
> datanode_2 | 2021-06-19 13:36:08,013 [Periodic HDDS volume checker] WARN volume.StorageVolumeChecker: checkAllVolumes timed out after 600000 ms
> datanode_2 | 2021-06-19 13:36:08,014 [Periodic HDDS volume checker] WARN volume.MutableVolumeSet: checkAllVolumes got 1 failed volumes - [/data/hdds/hdds]
> datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] INFO volume.MutableVolumeSet: Moving Volume : /data/hdds/hdds to failed Volumes
> datanode_2 | 2021-06-19 13:36:08,016 [Periodic HDDS volume checker] ERROR statemachine.DatanodeStateMachine: DatanodeStateMachine Shutdown due to too many bad volumes, check hdds.datanode.failed.data.volumes.tolerated and hdds.datanode.failed.metadata.volumes.tolerated
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org