You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by "awdavidson (via GitHub)" <gi...@apache.org> on 2023/05/10 15:02:06 UTC
[GitHub] [incubator-uniffle] awdavidson opened a new issue, #865: [Bug] Adding server LocalStorageChecker HealthChecks fails to initialize
awdavidson opened a new issue, #865:
URL: https://github.com/apache/incubator-uniffle/issues/865
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Enabling health checks and configuring the LocalStorageChecker throws IllegalArgumentException on initialisation. I would like to limit the amount of local disk uniffle can consume, it seems that enabling health checks should help us achieve this, however, local storage health checks are current not working.
### Affects Version(s)
master
### Uniffle Server Log Output
_No response_
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
```yaml
rss.server.health.check.enable true rss.server.health.checker.class.names org.apache.uniffle.server.LocalStorageChecker
```
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #865: [Bug] Adding server LocalStorageChecker HealthChecks fails to initialize
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #865:
URL: https://github.com/apache/incubator-uniffle/issues/865#issuecomment-1560537625
Nice catch. The real bug occurs in our internal enviroment. The hard limit is necessary, especially for the colocation deployment, more details please see #901 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] awdavidson commented on issue #865: [Bug] Adding server LocalStorageChecker HealthChecks fails to initialize
Posted by "awdavidson (via GitHub)" <gi...@apache.org>.
awdavidson commented on issue #865:
URL: https://github.com/apache/incubator-uniffle/issues/865#issuecomment-1546018839
After further investigation, LocalStorageChecker is automatically enabled when using LocalStorage. Enabling via health check and configuring the LocalStorageChecker class is not required.
Out the box it’s not obvious that LocalStorageChecker is automatically enabled, some additional logging when this class is initialised and preventing the HealthCheck from initialising LocalStorageChecker maybe valuable?
NB: From what I can see enabled HealthChecks does not prevent the server from writing to disk. The server metric is labelled as unhealthy but this is not used. Is this intentional? I did find another bug whilst trying to limit the amount of local disk uniffle can consume. The bug is to do with the metadata disk size not being updated when events are processed. I’ll raise a separate issue for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #865: [Bug] Adding server LocalStorageChecker HealthChecks fails to initialize
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #865:
URL: https://github.com/apache/incubator-uniffle/issues/865#issuecomment-1567256032
@awdavidson Should we close this issue after #902 is merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] awdavidson commented on issue #865: [Bug] Adding server LocalStorageChecker HealthChecks fails to initialize
Posted by "awdavidson (via GitHub)" <gi...@apache.org>.
awdavidson commented on issue #865:
URL: https://github.com/apache/incubator-uniffle/issues/865#issuecomment-1547415429
@jerqi Thank you for your response.
As for the hard limit, a design should not be required, uniffle does already have this functionality when the underlying bug to update `LocalStorageMeta` disk size is fixed. I'll raise another issue with information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #865: [Bug] Adding server LocalStorageChecker HealthChecks fails to initialize
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #865:
URL: https://github.com/apache/incubator-uniffle/issues/865#issuecomment-1546543659
> After further investigation, LocalStorageChecker is automatically enabled when using LocalStorage. Enabling via health check and configuring the LocalStorageChecker class is not required.
>
> Out the box it’s not obvious that LocalStorageChecker is automatically enabled, some additional logging when this class is initialised and preventing the HealthCheck from initialising LocalStorageChecker maybe valuable?
>
> NB: From what I can see enabled HealthChecks does not prevent the server from writing to disk. The server metric is labelled as unhealthy but this is not used. Is this intentional? I did find another bug whilst trying to limit the amount of local disk uniffle can consume. The bug is to do with the metadata disk size not being updated when events are processed. I’ll raise a separate issue for this
1. It's ok to log extra information.
2. We don't have hard limit about the disk usage. If you have interest, you can design it and contribute to Uniffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org