You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "adoroszlai (via GitHub)" <gi...@apache.org> on 2023/06/23 14:38:08 UTC

[GitHub] [ozone] adoroszlai opened a new pull request, #4966: HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics

adoroszlai opened a new pull request, #4966:
URL: https://github.com/apache/ozone/pull/4966

   ## What changes were proposed in this pull request?
   
   If failures happen during volume initialization, the volume object is abandoned, and a failed volume created instead.  The original object should be cleaned up.
   
   1. `StorageVolume#initialize` may throw `IOException`.  By this time it may have started background thread for disk usage check.  `HddsVolume` also created and registered metrics objects.
   2. `MutableVolumeSet#initializeVolumeSet` also may throw `IOException` if storage directory does not exist and cannot be created.
   
   To reproduce, simply add `/root` as an additional data or DB directory (assuming non-root user runs datanode):
   
   ```
   OZONE-SITE.XML_hdds.datanode.dir=/root,/data/hdds
   OZONE-SITE.XML_hdds.datanode.container.db.dir=/root,/data/metadata/db
   ```
   
   Failed data volume results in datanode stopping during startup:
   
   ```
   datanode_1  | [main] INFO volume.HddsVolume: Creating HddsVolume: /root/hdds of storage type : DISK capacity : 499596230656
   datanode_1  | [main] ERROR ozone.HddsDatanodeService: Exception in HddsDatanodeService.
   datanode_1  | org.apache.hadoop.metrics2.MetricsException: Metrics source VolumeInfoMetrics-/root already exists!
   datanode_1  | 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
   datanode_1  | 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
   datanode_1  | 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.VolumeInfoMetrics.init(VolumeInfoMetrics.java:50)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.VolumeInfoMetrics.<init>(VolumeInfoMetrics.java:45)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:140)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:73)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolume$Builder.build(HddsVolume.java:115)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolumeFactory.createFailedVolume(HddsVolumeFactory.java:60)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.initializeVolumeSet(MutableVolumeSet.java:187)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.<init>(MutableVolumeSet.java:135)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.<init>(MutableVolumeSet.java:99)
   datanode_1  | 	at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.<init>(OzoneContainer.java:146)
   datanode_1  | 	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.<init>(DatanodeStateMachine.java:173)
   datanode_1  | 	at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:306)
   ```
   
   Failed DB volume only results in leftover thread for the abandoned volume (`/root` here):
   
   ```
   $ jstack $(jps | grep Datanode | awk '{ print $1 }') | grep DiskUsage
   "DiskUsage-/data/hdds-
   "DiskUsage-/data/metadata/ratis-
   "DiskUsage-/root-
   "DiskUsage-/data/metadata/db-
   ```
   
   https://issues.apache.org/jira/browse/HDDS-8914
   
   ## How was this patch tested?
   
   Verified that datanode starts successfully with additional read-only volume, and that `DiskUsage` thread for the abandoned volume is not running.
   
   ```
   $ jstack $(jps | grep Datanode | awk '{ print $1 }') | grep DiskUsage
   "DiskUsage-/data/hdds-
   "DiskUsage-/data/metadata/ratis-
   "DiskUsage-/data/metadata/db-
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] adoroszlai commented on pull request #4966: HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4966:
URL: https://github.com/apache/ozone/pull/4966#issuecomment-1605988289

   @ArafatKhan2198 please take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] adoroszlai merged pull request #4966: HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #4966:
URL: https://github.com/apache/ozone/pull/4966


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] ArafatKhan2198 commented on pull request #4966: HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics

Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on PR #4966:
URL: https://github.com/apache/ozone/pull/4966#issuecomment-1607738710

   Thanks for working on this @adoroszlai it's a good find!
    LGTM +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] adoroszlai commented on pull request #4966: HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4966:
URL: https://github.com/apache/ozone/pull/4966#issuecomment-1608024943

   Thanks @ArafatKhan2198, @duongkame for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org