You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ivandika3 (via GitHub)" <gi...@apache.org> on 2023/10/10 10:11:58 UTC
[PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
ivandika3 opened a new pull request, #5414:
URL: https://github.com/apache/ozone/pull/5414
## What changes were proposed in this pull request?
In `BlockDataStreamOutput#setupStream`, if pipeline mode is enabled, we pass the Pipeline object to the `RatisHelper#getRoutingTable`.
In `RatisHelper#getRoutingTable`, we pick the first datanode in the pipeline as the primary, which from my understanding should be the Datanode closest to the client. However, I'm unable to find the logic that sort the datanodes in the pipeline.
From what I understand, when we open the stream key and request SCM to allocate the block (`OMKeyRequest#allocateBlock`), we need to also pass the clientMachineString so that so that nearest datanodes will be used instead. This should improve the performance of the stream pipeline mode.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9392
## How was this patch tested?
TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1764902250
cc @swamirishi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1848839297
Thank you for the review @szetszwo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1846798171
Hi @szetszwo, I have marked this MR as ready. This will finish of the works in https://github.com/apache/ozone/pull/5666 by integrating topology awareness to the Streaming Write Pipeline API. PTAL. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #5414:
URL: https://github.com/apache/ozone/pull/5414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1764901767
cc @tanvipenumudy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1825137030
@szetszwo Understood, thank you for the info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1811966959
@ivandika3 , please feel free to let me know when this is ready. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1825070298
Hi @szetszwo , might be slightly unrelated, but I have a question regarding the topology aware read with respect to Ratis pipeline.
From my understanding, the intent for topology aware read is for the client to be able to get the block / chunk from the nearest datanode. For RATIS pipeline, it seems we always use `XceiverClientRatis` (in `XceiverClientManager#getClient` we pick the `XceiverClientSpi` implementation based on the pipeline type). However, since we have not supported Linearizable read from follower for Datanodes, won't the block/chunk read using Ratis always go through the pipeline leader? I was expecting the topology-aware read to use `XceiverClientGrpc`, but was not able to find the logic that uses it.
I probably missed something since when the Ratis pipeline is closed, the Ratis group will be removed for the pipeline. This should mean that any Raft client trying to connect to that group will not be able to read the data using Ratis client (will throw `GroupMismatchException`).
Any clarifications would be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9392. Streaming write pipeline should pick the nearest datanode as primary node [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5414:
URL: https://github.com/apache/ozone/pull/5414#issuecomment-1825115552
@ivandika3 , for read, it will rebuild a pipeline for closed container; see https://github.com/apache/ozone/blob/60bb0604217fbd44dbc6b115b59fc26a297fcaba/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java#L275-L279
(the code is changed recently but the logic remains unchanged.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org