You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "symious (via GitHub)" <gi...@apache.org> on 2025/02/27 07:13:34 UTC
[PR] HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState [ozone]
symious opened a new pull request, #7977:
URL: https://github.com/apache/ozone/pull/7977
## What changes were proposed in this pull request?
We met the following error.
```
2025-02-10 10:03:19,995 [EventQueue-HealthyReadonlyNodeForHealthyReadOnlyNodeHandler] INFO org.apache.hadoop.hdds.scm.node.HealthyReadOnlyNodeHandler: Datanode 4570a118-82ab-44fe-98c6-fad28dc9f622{ip: 10.169.59.142, host: ip-10-169-59-142.idata-server.shopee.io, ports: [CLIENT_RPC=9864, REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /DC/RACK, certSerialId: null, persistedOpState: ENTERING_MAINTENANCE, persistedOpStateExpiryEpochSec: 0} moved to HEALTHY READONLY state.
2025-02-10 10:03:19,995 [EventQueue-HealthyReadonlyNodeForHealthyReadOnlyNodeHandler] INFO org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Added a new node: /DC/RACK/4570a118-82ab-44fe-98c6-fad28dc9f622
2025-02-10 10:03:20,470 [EventQueue-DeleteBlockStatusForDeletedBlockLogImpl] WARN org.apache.hadoop.hdds.scm.block.DeletedBlockLogImpl: Skip commit transactions since current SCM is not leader.
2025-02-10 10:03:20,756 [IPC Server handler 76 on default port 9861] INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Update the operationalState saved in follower SCM for 4570a118-82ab-44fe-98c6-fad28dc9f622{ip: 10.169.59.142, host: ip-10-169-59-142.idata-server.shopee.io, ports: [CLIENT_RPC=9864, REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0} as the reported value does not match the value stored in SCM (ENTERING_MAINTENANCE, 0)
2025-02-10 10:03:20,756 [EventQueue-HealthyReadonlyNodeForHealthyReadOnlyNodeHandler] INFO org.apache.hadoop.hdds.scm.node.HealthyReadOnlyNodeHandler: Datanode 4570a118-82ab-44fe-98c6-fad28dc9f622{ip: 10.169.59.142, host: ip-10-169-59-142.idata-server.shopee.io, ports: [CLIENT_RPC=9864, REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0} moved to HEALTHY READONLY state.
2025-02-10 10:03:20,756 [EventQueue-HealthyReadonlyNodeForHealthyReadOnlyNodeHandler] ERROR org.apache.hadoop.hdds.server.events.SingleThreadExecutor: Error on execution message 4570a118-82ab-44fe-98c6-fad28dc9f622{ip: 10.169.59.142, host: ip-10-169-59-142.idata-server.shopee.io, ports: [CLIENT_RPC=9864, REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}
org.apache.hadoop.hdds.scm.net.NetworkTopology$InvalidTopologyException: Failed to add /default-rack/ip-10-169-59-142.idata-server.shopee.io: Its path depth is not 4
at org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.add(NetworkTopologyImpl.java:101)
at org.apache.hadoop.hdds.scm.node.HealthyReadOnlyNodeHandler.onMessage(HealthyReadOnlyNodeHandler.java:75)
at org.apache.hadoop.hdds.scm.node.HealthyReadOnlyNodeHandler.onMessage(HealthyReadOnlyNodeHandler.java:39)
at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
The erorr is caused by incorrect Rack information reported by Datanode's heartbeat, instead we should use SCM's view of the Datanode here.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12423
## How was this patch tested?
Existing test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7977:
URL: https://github.com/apache/ozone/pull/7977#issuecomment-3519268758
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7977: HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState
URL: https://github.com/apache/ozone/pull/7977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7977:
URL: https://github.com/apache/ozone/pull/7977#issuecomment-2736175454
@symious , If the topology schema is 4 layers, such as "/DC/RACK/4570a118-82ab-44fe-98c6-fad28dc9f622" in above log, it only supports 4 layers, cannot support other cases, such as 3 layer or 5 layer. This is the excepted behavior.
BTW, datanode which has incorrect network path layer will fail at registration to SCM already. It cannot survive to this HealthyReadOnlyNodeHandler handler. Just curious about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState [ozone]
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #7977:
URL: https://github.com/apache/ozone/pull/7977#issuecomment-2742521096
@ChenSammi As the log shows, 10-169-59-142 has the correct the layer and is added first with location "/DC/RACK/4570a118-82ab-44fe-98c6-fad28dc9f622".
But it's then treated as "/default-rack/ip-10-169-59-142.idata-server.shopee.io".
The topology is correct, only SCM is using different views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #7977:
URL: https://github.com/apache/ozone/pull/7977#issuecomment-2750655106
@symious it looks like first location string "/DC/RACK/4570a118-82ab-44fe-98c6-fad28dc9f622", this DN is presented by a UUID, the second string "/default-rack/ip-10-169-59-142.idata-server.shopee.io", DN is represented by a hostname. DN's network location will be returned by SCM after DN registered to SCM, later this network location will be used in DN's other request to SCM. So the DN request should carry the same network location as the one in SCM, why here DN cannot be recognized by SCM, have this DN changed its hostname or IP after its registration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12423. Use inner view of DatanodeDetail in setNodeOperationalState [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7977:
URL: https://github.com/apache/ozone/pull/7977#issuecomment-3549967402
Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org