You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2025/04/30 19:22:48 UTC

[I] Move to a single thread pool for bulk import rename operations [accumulo]

keith-turner opened a new issue, #5524:
URL: https://github.com/apache/accumulo/issues/5524

   **Is your feature request related to a problem? Please describe.**
   
   The property `manager.rename.threadpool.size` is used to control the number of threads used for renaming files for bulk import.  The implementation will create a thread pool of this size per bulk import operation.  If there are F fate threads and this property is R, then can create up to F*R threads in the worst case.  Eventually too many threads will start to cause problems.   Also since the thread pools are per bulk import, can not easily track metrics on the thread pools.
   
   
   **Describe the solution you'd like**
   
   A single thread pool in the manager that all bulk imports use would offer two advantages.
   
    * More predictable control over resources
    * Ability to track metrics related to the rename pool
   
   This single thread pool could hang off of server context.
   
   **Additional context**
   
   Changing the implementation behind this property could negatively impact current systems w/ current settings for this property as they would all of a sudden have less threads for this work in some cases.  So not sure where this change should land if its done.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "arsenalzp (via GitHub)" <gi...@apache.org>.
arsenalzp commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2920040513

   Thank you for your support!
   Do you think singleton would be more appropriate for this solution? 
   Each manager implementation maintains its own Threads pool which is created during instantiation/first call then use it for each call when it needs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo closed issue #5524: Move to a single thread pool for bulk import rename operations
URL: https://github.com/apache/accumulo/issues/5524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "arsenalzp (via GitHub)" <gi...@apache.org>.
arsenalzp commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2849107370

   Hello,
   How complicated this issue is being considered for newcomers?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "arsenalzp (via GitHub)" <gi...@apache.org>.
arsenalzp commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2853942454

   Let me explore it by myself as this project is pretty new for me. 
   If no luck I will ask you for help, it possible.
   Is it OK for you?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-3192359764

   Pretty sure this can be closed via #5592. If that didn't fix this then it can be reopened.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2850841867

   @arsenalzp a lot of the complexity of this change would be related to finding the existing code and knowing where put the new code.  Would you like pointers to the existing code so you can assess it for yourself?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "arsenalzp (via GitHub)" <gi...@apache.org>.
arsenalzp commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2900382355

   /assign


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "arsenalzp (via GitHub)" <gi...@apache.org>.
arsenalzp commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2916158017

   Hello @keith-turner,
   It looks like Threads pool of size `manager.rename.threadpool.size` is created for every bulk rename operation.
   As far as I understand we want to have a single Threads pool for all bulk rename operations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2930965987

   > Do you think singleton would be more appropriate for this solution?
   
   @arsenalzp yes that would be appropriate


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "arsenalzp (via GitHub)" <gi...@apache.org>.
arsenalzp commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2910746081

   Hello,
   I seems bulk import is done in by `org.apache.accumulo.server.fs.VolumeManager` implementation, which is `VolumeManagerImpl`.
   
   ```
   public void bulkRename(Map<Path,Path> oldToNewPathMap, int poolSize, String poolName,
         FateId fateId) throws IOException {
       List<Future<Void>> results = new ArrayList<>();
       ExecutorService workerPool = ThreadPools.getServerThreadPools().getPoolBuilder(poolName)
           .numCoreThreads(poolSize).build();
       oldToNewPathMap.forEach((oldPath, newPath) -> results.add(workerPool.submit(() -> {
           ...
       }
   ```
   As I see each call of `fs.bulkRename()` creates a new `ExecutorService`  instance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-2916848190

   > It seems bulk import is done in by org.apache.accumulo.server.fs.VolumeManager implementation, which is VolumeManagerImpl.
   
   
   That is the code that needs to change.  Could possibly add a method like the following where the executor service is passed in.
   
   ```java
   public void bulkRename(Map<Path,Path> oldToNewPathMap, ExecutorService workerPool,
         FateId fateId) throws IOException {
       List<Future<Void>> results = new ArrayList<>();
       oldToNewPathMap.forEach((oldPath, newPath) -> results.add(workerPool.submit(() -> {
           ...
       }
   ```
   
   > Hello @keith-turner,
   It looks like Threads pool of size manager.rename.threadpool.size is created for every bulk rename operation.
   As far as I understand we want to have a single Threads pool for all bulk rename operations.
   
   
   Correct.  Could use that property to create a single executor service for the manager process and then pass that service to the bulk rename method.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Move to a single thread pool for bulk import rename operations [accumulo]

Posted by "kevinrr888 (via GitHub)" <gi...@apache.org>.
kevinrr888 commented on issue #5524:
URL: https://github.com/apache/accumulo/issues/5524#issuecomment-3128047308

   @keith-turner - can this be closed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org