You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/05/20 05:06:02 UTC
[GitHub] [ozone] guihecheng opened a new pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
guihecheng opened a new pull request #2271:
URL: https://github.com/apache/ozone/pull/2271
## What changes were proposed in this pull request?
Pipeline placement policy should filter out those datanodes that do not even have enough space for a single container, otherwise even if the pipeline creation succeeds, a datanode may quickly report that it fails to create a container and there will be successive failures because we will choose that node again and again.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5252
## How was this patch tested?
a new ut.
manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-846978968
> @bshashikant Ah, just now I've a talk with @GlenGeng , he suggested that we could do better here.
>
> We could make the `sizeRequired` to a new config item such as 'pipeline.reserved.space', may be default to 0.
> So, with this new item, we could do the following things:
>
> 1. use default, then we keep the same behavior as before, so users/developers who just want to do small tests or try out ozone will not be annoyed by some warning says that they have no space for pipeline creation.
> 2. config it to some value for space reservation (should better be > container size), so then we could solve the problem here, too.
> Moreover, when we have this 'reserved.space' more than container size, say 20% of the volume capacity, then we can filter out the nodes with all its volumes with usage > 80%, which may not be a good choice for pipeline allocation in production as Glen mentioned.
> It seems like we could got overall controll over the space usage of the cluster with this new config item.
> And this should not be conflict with the datanode config 'du.reserved', since its use case is that we have multiple consumers for a single disk.
I guess, we need to be more precise here. A container creation won't succeed on a datanode unless it has remaining space >= container size as there is hard check there(check RoundRobinVolumeChoosingPolicy#chooseVolume()). Its better to to not choose such datanodes upfront during pipeline allocation in any case.
The pipeline reserved space notion should consider ratis volumes size specifically as well as so as to ensure not allocate any new pipelines on nodes which are very low on ratis log disk space, and should have enough space on all constituent datanodes of the pipeline fo the existing pipelines to get closed as closing a pipeline takes a snapshot and does IO on the raft log disk as well.
This patch itself looks good to me and can be committed and pipeline reserve semantic can be then built on top of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-846832155
@GlenGeng Thanks for your comments, I understand your concern, but I think it is a bit different here.
A datanode with usage > 80% is not a good candidate indeed, and should **have a lower priority** to be chosen, but it should still have a chance to be chosen. There is a possibility that all datanodes have > 80% space in the cluster, then if we filter out them completely then we don't have a writable cluster while actually there are ~20% space left.
Here we try to solve the problem that if there not not even space for a single container then that datanode should not be chosen anyway.
Maybe we could implement a new placement policy for pipelines that have a priority defined according to available space of datanodes to solve the problem you mentioned ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-846942209
@bshashikant Ah, just now I've a talk with @GlenGeng , he suggested that we could do better here.
We could make the `sizeRequired` to a new config item such as 'pipeline.reserved.space', may be default to 0.
So, with this new item, we could do the following things:
1. use default, then we keep the same behavior as before, so users/developers who just want to do small tests or try out ozone will not be annoyed by some warning says that they have no space for pipeline creation.
2. config it to some value for space reservation (should better be > container size), so then we could solve the problem here, too.
Moreover, when we have this 'reserved.space' more than container size, say 20% of the volume capacity, then we can filter out the nodes with all its volumes with usage > 80%, which may not be a good choice for pipeline allocation in production as Glen mentioned.
It seems like we could got overall controll over the space usage of the cluster with this new config item.
And this should not be conflict with the datanode config 'du.reserved', since its use case is that we have multiple consumers for a single disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-846879298
Thanks @guihecheng for the clarificaiton.
One concern about the solution based on `sizeRequired` is, assume there are 3 DN dockers for test, each has 1GB storage space, will scm be able to create a pipeline from them ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GlenGeng commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-846812200
Thanks @guihecheng for the fix!
I agree with the necessity of the solution. My concern here is the `sizeRequired` which is 5GB by default, which seems to be not quite viable for a production cluster.
How about adjust it to base on usage percent of the Datanode? For example, a Datanode with usage great then 80% is not a good candidate for new pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-844767953
@bshashikant @ChenSammi please help review this one, thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant merged pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
bshashikant merged pull request #2271:
URL: https://github.com/apache/ozone/pull/2271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-847014674
> > @bshashikant Ah, just now I've a talk with @GlenGeng , he suggested that we could do better here.
> > We could make the `sizeRequired` to a new config item such as 'pipeline.reserved.space', may be default to 0.
> > So, with this new item, we could do the following things:
> >
> > 1. use default, then we keep the same behavior as before, so users/developers who just want to do small tests or try out ozone will not be annoyed by some warning says that they have no space for pipeline creation.
> > 2. config it to some value for space reservation (should better be > container size), so then we could solve the problem here, too.
> > Moreover, when we have this 'reserved.space' more than container size, say 20% of the volume capacity, then we can filter out the nodes with all its volumes with usage > 80%, which may not be a good choice for pipeline allocation in production as Glen mentioned.
> > It seems like we could got overall controll over the space usage of the cluster with this new config item.
> > And this should not be conflict with the datanode config 'du.reserved', since its use case is that we have multiple consumers for a single disk.
>
> I guess, we need to be more precise here. A container creation won't succeed on a datanode unless it has remaining space >= container size as there is hard check there(check RoundRobinVolumeChoosingPolicy#chooseVolume()). Its better to to not choose such datanodes upfront during pipeline allocation in any case.
Sure, even if we have the proposed config item, we should ensure that if configed value < container size, we should drop it and use container size instead(with a warning log).
> The pipeline reserved space notion should consider ratis volumes size specifically as well as so as to ensure not allocate any new pipelines on nodes which are very low on ratis log disk space, and should have enough space on all constituent datanodes of the pipeline fo the existing pipelines to get closed as closing a pipeline takes a snapshot and does IO on the raft log disk as well.
Yes, ratis log disk should be taken into consideration.
> This patch itself looks good to me and can be committed and pipeline reserve semantic can be then built on top of this.
Ok, then I'm not going to update this but build another pr for the proposed solution.
Thanks very much @bshashikant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng edited a comment on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
guihecheng edited a comment on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-847014674
> > @bshashikant Ah, just now I've a talk with @GlenGeng , he suggested that we could do better here.
> > We could make the `sizeRequired` to a new config item such as 'pipeline.reserved.space', may be default to 0.
> > So, with this new item, we could do the following things:
> >
> > 1. use default, then we keep the same behavior as before, so users/developers who just want to do small tests or try out ozone will not be annoyed by some warning says that they have no space for pipeline creation.
> > 2. config it to some value for space reservation (should better be > container size), so then we could solve the problem here, too.
> > Moreover, when we have this 'reserved.space' more than container size, say 20% of the volume capacity, then we can filter out the nodes with all its volumes with usage > 80%, which may not be a good choice for pipeline allocation in production as Glen mentioned.
> > It seems like we could got overall controll over the space usage of the cluster with this new config item.
> > And this should not be conflict with the datanode config 'du.reserved', since its use case is that we have multiple consumers for a single disk.
>
> I guess, we need to be more precise here. A container creation won't succeed on a datanode unless it has remaining space >= container size as there is hard check there(check RoundRobinVolumeChoosingPolicy#chooseVolume()). Its better to to not choose such datanodes upfront during pipeline allocation in any case.
Sure, even if we have the proposed config item, we should ensure that if configed value < container size, we should drop it and use container size instead(with a warning log).
> The pipeline reserved space notion should consider ratis volumes size specifically as well as so as to ensure not allocate any new pipelines on nodes which are very low on ratis log disk space, and should have enough space on all constituent datanodes of the pipeline fo the existing pipelines to get closed as closing a pipeline takes a snapshot and does IO on the raft log disk as well.
Yes, ratis log disk should be taken into consideration.
> This patch itself looks good to me and can be committed and pipeline reserve semantic can be then built on top of this.
Ok, then I'm not going to update this but build another pr for the proposed solution.
Thanks very much @bshashikant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on pull request #2271: HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on pull request #2271:
URL: https://github.com/apache/ozone/pull/2271#issuecomment-846884120
@GlenGeng oh, I think not, after this one we should have to set a proper container size for test deploy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org