You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2025/03/04 08:28:23 UTC

[PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

sarvekshayr opened a new pull request, #8003:
URL: https://github.com/apache/ozone/pull/8003

   ## What changes were proposed in this pull request?
   The following deletion related configurations are now dynamically reconfigurable without requiring a restart. 
   This allows for easy tuning of deletion in response to logs and metrics.
   
   OM:
   - ozone.directory.deleting.service.interval
   - ozone.thread.number.dir.deletion 
   
   DATANODE:
   - ozone.block.deleting.service.interval 
   - ozone.block.deleting.service.timeout   
   
   ## What is the link to the Apache JIRA
   [HDDS-11513](https://issues.apache.org/jira/browse/HDDS-11513)
   
   ## How was this patch tested?
   OM:
   - List all OM reconfigurable properties
   ```
   bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 properties
   OM: Node [om:9862] Reconfigurable properties:
   ozone.administrators
   ozone.directory.deleting.service.interval
   ozone.key.deleting.limit.per.task
   ozone.om.server.list.max.size
   ozone.om.volume.listall.allowed
   ozone.readonly.administrators
   ozone.thread.number.dir.deletion
   ```
   - Reconfigured `ozone.directory.deleting.service.interval` successfully
   ```
   bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start     
   OM: Started reconfiguration task on node [om:9862].
   bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
   OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 05:51:56 UTC 2025 and finished at Tue Mar 04 05:51:56 UTC 2025.
   SUCCESS: Changed property ozone.directory.deleting.service.interval
           From: "1m"
           To: "2m"
   ```
   - Reconfigured `ozone.thread.number.dir.deletion` successfully
   ```
   bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start     
   OM: Started reconfiguration task on node [om:9862].
   bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
   OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 06:41:19 UTC 2025 and finished at Tue Mar 04 06:41:19 UTC 2025.
   SUCCESS: Changed property ozone.thread.number.dir.deletion
           From: ""
           To: "20"
   ```
   - Error case when `ozone.thread.number.dir.deletion` value is negative.
   ```
   bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start     
   OM: Started reconfiguration task on node [om:9862].
   bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
   OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 07:07:08 UTC 2025 and finished at Tue Mar 04 07:07:08 UTC 2025.
   FAILED: Change property ozone.thread.number.dir.deletion
           From: ""
           To: "-20"
           Error: ozone.thread.number.dir.deletion cannot be negative..
   ```
   
   DATANODE:
   - List all DN reconfigurable properties
   ```
   bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 properties
   DN: Node [datanode:19864] Reconfigurable properties:
   hdds.datanode.block.delete.threads.max
   hdds.datanode.block.deleting.limit.per.interval
   hdds.datanode.replication.streams.limit
   ozone.block.deleting.service.interval
   ozone.block.deleting.service.timeout
   ozone.block.deleting.service.workers
   ```
   - Reconfigured `ozone.block.deleting.service.interval` and `ozone.block.deleting.service.timeout` successfully
   ```
   bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 start     
   DN: Started reconfiguration task on node [datanode:19864].
   bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 status
   DN: Reconfiguring status for node [datanode:19864]: started at Tue Mar 04 05:50:29 UTC 2025 and finished at Tue Mar 04 05:50:29 UTC 2025.
   SUCCESS: Changed property ozone.block.deleting.service.interval
           From: "1m"
           To: "2m"
   SUCCESS: Changed property ozone.block.deleting.service.timeout
           From: "300000ms"
           To: "350000ms"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2697327921

   `BlockDeletingService` is created with interval and timeout taken from config:
   
   https://github.com/apache/ozone/blob/10945c88c3a1cfbf197e156ab229a00179262241/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java#L245-L260
   
   They are stored in the base `BackgroundService` class:
   
   https://github.com/apache/ozone/blob/10945c88c3a1cfbf197e156ab229a00179262241/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java#L58-L65
   
   which schedules the task when started:
   
   https://github.com/apache/ozone/blob/10945c88c3a1cfbf197e156ab229a00179262241/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java#L108-L109
   
   Reconfiguring does not change values in `BackgroundService` (they are `final`), and even updating `interval` wouldn't change the task already scheduled with fixed delay (it needs to be rescheduled).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2135570932


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java:
##########
@@ -86,6 +86,11 @@ public void setPoolSize(int size) {
     exec.setCorePoolSize(size);
   }
 
+  public synchronized void setServiceTimeoutInNanos(long newTimeout) {
+    LOG.info("{} timeout is {} {}", serviceName, newTimeout, TimeUnit.NANOSECONDS.name().toLowerCase());
+    this.serviceTimeoutInNanos = newTimeout;

Review Comment:
   The `serviceTimeoutInNanos` field is written under synchronization but read without synchronization, which can lead to visibility issues. Consider declaring the field as `volatile` or using an `AtomicLong` to ensure thread-safe updates and reads.
   ```suggestion
       this.serviceTimeoutInNanos.set(newTimeout);
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -5159,6 +5161,12 @@ private String reconfOzoneDirDeletingServiceInterval(String newVal) {
     return newVal;
   }
 
+  private String reconfOzoneThreadNumberDirDeletion(String newVal) {
+    getConfiguration().set(OZONE_THREAD_NUMBER_DIR_DELETION, newVal);
+    getKeyManager().getDirDeletingService().setPoolSize(Integer.parseInt(newVal));

Review Comment:
   Parsing the new thread count directly may throw `NumberFormatException` on invalid input and cause the reconfiguration task to fail without a clear error message. Consider validating the string (e.g., with a regex or try/catch) and returning a descriptive error for invalid values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #8003:
URL: https://github.com/apache/ozone/pull/8003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r1979005537


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/BlockDeletingService.java:
##########
@@ -64,6 +65,8 @@ public class BlockDeletingService extends BackgroundService {
   private static final int TASK_PRIORITY_DEFAULT = 1;
 
   private final Duration blockDeletingMaxLockHoldingTime;
+  private String blockDeletingServiceInterval;
+  private String blockDeletingServiceTimeout;

Review Comment:
   These should not store the raw String config.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2698184897

   > simply updating these properties without starting the OM again won't be effective here
   
   That's my point.  OM can be tweaked to pick up the config without restart, but the current patch is not enough for that yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2144372295


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java:
##########
@@ -92,6 +92,11 @@ public void setPoolSize(int size) {
     exec.setCorePoolSize(size);
   }
 
+  public synchronized void setServiceTimeoutInNanos(long newTimeout) {
+    LOG.info("{} timeout is {} {}", serviceName, newTimeout, TimeUnit.NANOSECONDS.name().toLowerCase());

Review Comment:
   Can we make this log a bit more clear?
   ```suggestion
       LOG.info("{} timeout is set to {} {}", serviceName, newTimeout, TimeUnit.NANOSECONDS.name().toLowerCase());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2135559751


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -5159,6 +5161,12 @@ private String reconfOzoneDirDeletingServiceInterval(String newVal) {
     return newVal;
   }
 
+  private String reconfOzoneThreadNumberDirDeletion(String newVal) {
+    getConfiguration().set(OZONE_THREAD_NUMBER_DIR_DELETION, newVal);

Review Comment:
   restart trigger should be done now on 2 param matches, interval, threadCount, but this new addition is not there in registered callback at DirectoryDeletingService



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/BlockDeletingService.java:
##########
@@ -94,12 +98,31 @@ public BlockDeletingService(
     dnConf = conf.getObject(DatanodeConfiguration.class);
     if (reconfigurationHandler != null) {
       reconfigurationHandler.register(dnConf);
+      registerReconfigCallbacks(reconfigurationHandler, (OzoneConfiguration) conf);
     }
     this.blockDeletingMaxLockHoldingTime =
         dnConf.getBlockDeletingMaxLockHoldingTime();
     metrics = BlockDeletingServiceMetrics.create();
   }
 
+  public void registerReconfigCallbacks(ReconfigurationHandler handler, OzoneConfiguration ozoneConf) {
+    handler.registerCompleteCallback((changedKeys, newConf) -> {
+      if (changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_INTERVAL)) {

Review Comment:
   For timeout also, do need restart the service? it is directly set in HddsDatanodeService, do that setting needs to be done here during updateAndRestart ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2697475115

   Thanks for the clarification, @adoroszlai. I checked the code, for these configs to be updated, we need to restart the OM. The "ozone om" command starts the key manager, which then initializes the block deleting and directory deleting service with their intervals. So, simply updating these properties without starting the OM again won't be effective here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on code in PR #8003:
URL: https://github.com/apache/ozone/pull/8003#discussion_r2143708735


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/BlockDeletingService.java:
##########
@@ -94,12 +102,41 @@ public BlockDeletingService(
     dnConf = conf.getObject(DatanodeConfiguration.class);
     if (reconfigurationHandler != null) {
       reconfigurationHandler.register(dnConf);
+      registerReconfigCallbacks(reconfigurationHandler, (OzoneConfiguration) conf);
     }
     this.blockDeletingMaxLockHoldingTime =
         dnConf.getBlockDeletingMaxLockHoldingTime();
     metrics = BlockDeletingServiceMetrics.create();
   }
 
+  public void registerReconfigCallbacks(ReconfigurationHandler handler, OzoneConfiguration ozoneConf) {
+    handler.registerCompleteCallback((changedKeys, newConf) -> {
+      if (changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_INTERVAL) ||
+          changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_TIMEOUT) ||
+          changedKeys.containsKey(OZONE_BLOCK_DELETING_SERVICE_WORKERS)) {
+        updateAndRestart(ozoneConf);

Review Comment:
   IIUC, consider using the newConf object passed into the callback, containing the updated configuration values. This ensures that updateAndRestart() uses the latest values and avoids applying outdated configuration settings.
   `updateAndRestart(newConf);`



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java:
##########
@@ -110,7 +112,8 @@ public DirectoryDeletingService(long interval, TimeUnit unit,
 
   public void registerReconfigCallbacks(ReconfigurationHandler handler, OzoneConfiguration conf) {
     handler.registerCompleteCallback((changedKeys, newConf) -> {
-      if (changedKeys.containsKey(OZONE_DIR_DELETING_SERVICE_INTERVAL)) {
+      if (changedKeys.containsKey(OZONE_DIR_DELETING_SERVICE_INTERVAL) ||
+          changedKeys.containsKey(OZONE_THREAD_NUMBER_DIR_DELETION)) {
         updateAndRestart(conf);

Review Comment:
   Same here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2948954883

   @sumitagrawl @aryangupta1998 @Tejaskriya Please review this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-11513. All deletion configurations should be configurable without restart [ozone]

Posted by "aryangupta1998 (via GitHub)" <gi...@apache.org>.
aryangupta1998 commented on PR #8003:
URL: https://github.com/apache/ozone/pull/8003#issuecomment-2697227362

   @adoroszlai, the dirDeletingServiceInterval(ozone.directory.deleting.service.interval) and blockDeletingServiceInterval(ozone.block.deleting.service.interval) are used by the background services to determine the intervals between their execution. These intervals, apart from being used in tests, are configured for the background services that delete directories and blocks. The background service uses scheduleWithFixedDelay to schedule the tasks with the specified intervals.
   `public void start() {
       exec.scheduleWithFixedDelay(service, 0, interval, unit);
   }`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org