You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2025/03/04 08:28:23 UTC
[PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
sarvekshayr opened a new pull request, #8003:
URL: https://github.com/apache/ozone/pull/8003
## What changes were proposed in this pull request?
The following deletion related configurations are now dynamically reconfigurable without requiring a restart.
This allows for easy tuning of deletion in response to logs and metrics.
OM:
- ozone.directory.deleting.service.interval
- ozone.thread.number.dir.deletion
DATANODE:
- ozone.block.deleting.service.interval
- ozone.block.deleting.service.timeout
## What is the link to the Apache JIRA
[HDDS-11513](https://issues.apache.org/jira/browse/HDDS-11513)
## How was this patch tested?
OM:
- List all OM reconfigurable properties
```
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 properties
OM: Node [om:9862] Reconfigurable properties:
ozone.administrators
ozone.directory.deleting.service.interval
ozone.key.deleting.limit.per.task
ozone.om.server.list.max.size
ozone.om.volume.listall.allowed
ozone.readonly.administrators
ozone.thread.number.dir.deletion
```
- Reconfigured `ozone.directory.deleting.service.interval` successfully
```
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 05:51:56 UTC 2025 and finished at Tue Mar 04 05:51:56 UTC 2025.
SUCCESS: Changed property ozone.directory.deleting.service.interval
From: "1m"
To: "2m"
```
- Reconfigured `ozone.thread.number.dir.deletion` successfully
```
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 06:41:19 UTC 2025 and finished at Tue Mar 04 06:41:19 UTC 2025.
SUCCESS: Changed property ozone.thread.number.dir.deletion
From: ""
To: "20"
```
- Error case when `ozone.thread.number.dir.deletion` value is negative.
```
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 07:07:08 UTC 2025 and finished at Tue Mar 04 07:07:08 UTC 2025.
FAILED: Change property ozone.thread.number.dir.deletion
From: ""
To: "-20"
Error: ozone.thread.number.dir.deletion cannot be negative..
```
DATANODE:
- List all DN reconfigurable properties
```
bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 properties
DN: Node [datanode:19864] Reconfigurable properties:
hdds.datanode.block.delete.threads.max
hdds.datanode.block.deleting.limit.per.interval
hdds.datanode.replication.streams.limit
ozone.block.deleting.service.interval
ozone.block.deleting.service.timeout
ozone.block.deleting.service.workers
```
- Reconfigured `ozone.block.deleting.service.interval` and `ozone.block.deleting.service.timeout` successfully
```
bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 start
DN: Started reconfiguration task on node [datanode:19864].
bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 status
DN: Reconfiguring status for node [datanode:19864]: started at Tue Mar 04 05:50:29 UTC 2025 and finished at Tue Mar 04 05:50:29 UTC 2025.
SUCCESS: Changed property ozone.block.deleting.service.interval
From: "1m"
To: "2m"
SUCCESS: Changed property ozone.block.deleting.service.timeout
From: "300000ms"
To: "350000ms"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2697327921
`BlockDeletingService` is created with interval and timeout taken from config:
https://github.com/apache/ozone/blob/10945c88c3a1cfbf197e156ab229a00179262241/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java#L245-L260
They are stored in the base `BackgroundService` class:
https://github.com/apache/ozone/blob/10945c88c3a1cfbf197e156ab229a00179262241/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java#L58-L65
which schedules the task when started:
https://github.com/apache/ozone/blob/10945c88c3a1cfbf197e156ab229a00179262241/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java#L108-L109
Reconfiguring does not change values in `BackgroundService` (they are `final`), and even updating `interval` wouldn't change the task already scheduled with fixed delay (it needs to be rescheduled).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2135570932
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java:
##########
@@ -86,6 +86,11 @@ public void setPoolSize(int size) {
exec.setCorePoolSize(size);
}
+ public synchronized void setServiceTimeoutInNanos(long newTimeout) {
+ LOG.info("{} timeout is {} {}", serviceName, newTimeout, TimeUnit.NANOSECONDS.name().toLowerCase());
+ this.serviceTimeoutInNanos = newTimeout;
Review Comment:
The `serviceTimeoutInNanos` field is written under synchronization but read without synchronization, which can lead to visibility issues. Consider declaring the field as `volatile` or using an `AtomicLong` to ensure thread-safe updates and reads.
```suggestion
this.serviceTimeoutInNanos.set(newTimeout);
```
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -5159,6 +5161,12 @@ private String reconfOzoneDirDeletingServiceInterval(String newVal) {
return newVal;
}
+ private String reconfOzoneThreadNumberDirDeletion(String newVal) {
+ getConfiguration().set(OZONE_THREAD_NUMBER_DIR_DELETION, newVal);
+ getKeyManager().getDirDeletingService().setPoolSize(Integer.parseInt(newVal));
Review Comment:
Parsing the new thread count directly may throw `NumberFormatException` on invalid input and cause the reconfiguration task to fail without a clear error message. Consider validating the string (e.g., with a regex or try/catch) and returning a descriptive error for invalid values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #8003:
URL: https://github.com/apache/ozone/pull/8003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r1979005537
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/BlockDeletingService.java:
##########
@@ -64,6 +65,8 @@ public class BlockDeletingService extends BackgroundService {
private static final int TASK_PRIORITY_DEFAULT = 1;
private final Duration blockDeletingMaxLockHoldingTime;
+ private String blockDeletingServiceInterval;
+ private String blockDeletingServiceTimeout;
Review Comment:
These should not store the raw String config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2698184897
> simply updating these properties without starting the OM again won't be effective here
That's my point. OM can be tweaked to pick up the config without restart, but the current patch is not enough for that yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2144372295
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java:
##########
@@ -92,6 +92,11 @@ public void setPoolSize(int size) {
exec.setCorePoolSize(size);
}
+ public synchronized void setServiceTimeoutInNanos(long newTimeout) {
+ LOG.info("{} timeout is {} {}", serviceName, newTimeout, TimeUnit.NANOSECONDS.name().toLowerCase());
Review Comment:
Can we make this log a bit more clear?
```suggestion
LOG.info("{} timeout is set to {} {}", serviceName, newTimeout, TimeUnit.NANOSECONDS.name().toLowerCase());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2135559751
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -5159,6 +5161,12 @@ private String reconfOzoneDirDeletingServiceInterval(String newVal) {
return newVal;
}
+ private String reconfOzoneThreadNumberDirDeletion(String newVal) {
+ getConfiguration().set(OZONE_THREAD_NUMBER_DIR_DELETION, newVal);
Review Comment:
restart trigger should be done now on 2 param matches, interval, threadCount, but this new addition is not there in registered callback at DirectoryDeletingService
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/BlockDeletingService.java:
##########
@@ -94,12 +98,31 @@ public BlockDeletingService(
dnConf = conf.getObject(DatanodeConfiguration.class);
if (reconfigurationHandler != null) {
reconfigurationHandler.register(dnConf);
+ registerReconfigCallbacks(reconfigurationHandler, (OzoneConfiguration) conf);
}
this.blockDeletingMaxLockHoldingTime =
dnConf.getBlockDeletingMaxLockHoldingTime();
metrics = BlockDeletingServiceMetrics.create();
}
+ public void registerReconfigCallbacks(ReconfigurationHandler handler, OzoneConfiguration ozoneConf) {
+ handler.registerCompleteCallback((changedKeys, newConf) -> {
+ if (changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_INTERVAL)) {
Review Comment:
For timeout also, do need restart the service? it is directly set in HddsDatanodeService, do that setting needs to be done here during updateAndRestart ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2697475115
Thanks for the clarification, @adoroszlai. I checked the code, for these configs to be updated, we need to restart the OM. The "ozone om" command starts the key manager, which then initializes the block deleting and directory deleting service with their intervals. So, simply updating these properties without starting the OM again won't be effective here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2143708735
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/BlockDeletingService.java:
##########
@@ -94,12 +102,41 @@ public BlockDeletingService(
dnConf = conf.getObject(DatanodeConfiguration.class);
if (reconfigurationHandler != null) {
reconfigurationHandler.register(dnConf);
+ registerReconfigCallbacks(reconfigurationHandler, (OzoneConfiguration) conf);
}
this.blockDeletingMaxLockHoldingTime =
dnConf.getBlockDeletingMaxLockHoldingTime();
metrics = BlockDeletingServiceMetrics.create();
}
+ public void registerReconfigCallbacks(ReconfigurationHandler handler, OzoneConfiguration ozoneConf) {
+ handler.registerCompleteCallback((changedKeys, newConf) -> {
+ if (changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_INTERVAL) ||
+ changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_TIMEOUT) ||
+ changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_WORKERS)) {
+ updateAndRestart(ozoneConf);
Review Comment:
IIUC, consider using the newConf object passed into the callback, containing the updated configuration values. This ensures that updateAndRestart() uses the latest values and avoids applying outdated configuration settings.
`updateAndRestart(newConf);`
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java:
##########
@@ -110,7 +112,8 @@ public DirectoryDeletingService(long interval, TimeUnit unit,
public void registerReconfigCallbacks(ReconfigurationHandler handler, OzoneConfiguration conf) {
handler.registerCompleteCallback((changedKeys, newConf) -> {
- if (changedKeys.containsKey(OZONE_DIR_DELETING_SERVICE_INTERVAL)) {
+ if (changedKeys.containsKey(OZONE_DIR_DELETING_SERVICE_INTERVAL) ||
+ changedKeys.containsKey(OZONE_THREAD_NUMBER_DIR_DELETION)) {
updateAndRestart(conf);
Review Comment:
Same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2948954883
@sumitagrawl @aryangupta1998 @Tejaskriya Please review this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]
Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2697227362
@adoroszlai, the dirDeletingServiceInterval(ozone.directory.deleting.service.interval) and blockDeletingServiceInterval(ozone.block.deleting.service.interval) are used by the background services to determine the intervals between their execution. These intervals, apart from being used in tests, are configured for the background services that delete directories and blocks. The background service uses scheduleWithFixedDelay to schedule the tasks with the specified intervals.
`public void start() {
exec.scheduleWithFixedDelay(service, 0, interval, unit);
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org