[GitHub] [accumulo] alerman opened a new issue, #3749: Accumulo not handling compaction failures well, and results in tablet never compacting

["alerman (via GitHub)" <gi...@apache.org>]

alerman opened a new issue, #3749:
URL: https://github.com/apache/accumulo/issues/3749

   **Describe the bug**
   
   We are seeing cases where a compaction fails and as a result the tservers get into a bad state. When this happens, the tablet gets stuck in an infinite loop. I will lay out the progression here. I have pulled out anything that was not related to the tablet in question.
   
   As I was filling in this issue, I noticed that I think the root cause for the failure may have been updates to Compaction planner options configurations.
   
   1. Compacting <tablet id> on i.default.small for SELECTOR from [File name list] size 1MB
   2. Stopped Compaction executor i.default.small
   3. Updated compaction server<redacted> -- change was to up the number of threads
   4. tablet.DatafileManager reports ERROR:Failure updating files after major compaction <tablet ID> Optional[New C file path] [list of file paths]
   5. java.lang.InterruptedException thrown at TabletserverBatchWriter:506
   6. clientImple.TabletServerBatchWriter DEBUG: faile to send mutations to <server hosting metadata tablet>. Interrupted while waiting for IO on channel <channel info> Total timeout is 120000,115355 millis timeout left
   7. tablet.CompactableImple DEBUG: Selected files not in all files [files in list of file paths from 4] [list of tablet files]
   
   7 has been repeating since. I see this as 2 problems. 1) Updating compaction settings should tank the running compactions and 2) Once it has, it shouldnt prevent compactions from occuring on that tablet
   
   
   **Versions (OS, Maven, Java, and others, as appropriate):**
   - 2.1.3-SNAPSHOT
   - Java 11
   - CentOS 7
   
   **To Reproduce**
   Steps to reproduce the behavior (or a link to an example repository that reproduces the problem):
   I think this can be reproduced by changing compaction settings while a compaction is in progress, but not entirely sure
   
   **Expected behavior**
   Chaning compaction settings is either not zookeeper mutable or doesnt break things when settings are changed.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] alerman commented on issue #3749: Accumulo not handling compaction failures well, and results in tablet never compacting

["alerman (via GitHub)" <gi...@apache.org>]

alerman commented on issue #3749:
URL: https://github.com/apache/accumulo/issues/3749#issuecomment-1715690461

   tserver.compaction.major.service.default.planner.opts.executors was changed. In that, the "small" executor definition wa changed to lower the maxSize and maxThreads values. Specific values not inclueded.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] keith-turner closed issue #3749: Accumulo not handling compaction failures well, and results in tablet never compacting

["keith-turner (via GitHub)" <gi...@apache.org>]

keith-turner closed issue #3749: Accumulo not handling compaction failures well, and results in tablet never compacting
URL: https://github.com/apache/accumulo/issues/3749


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org