You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "keith-turner (via GitHub)" <gi...@apache.org> on 2025/07/02 19:14:13 UTC

[PR] fixes bug where external compaction never starts [accumulo]

keith-turner opened a new pull request, #5707:
URL: https://github.com/apache/accumulo/pull/5707

   Ran into an issue when running random walk bulk test where a compaction that needed to happen was never running.  The code fixed in this PR used to change the state to running and then forget about it in some cases. However [this code][1] would still remember it.  That would cause this [this code][2] to filter out planned jobs when there were running compactions in submitted jobs. Hopefully this change will cause [this code][3] to filter this from submitted jobs.
   
   Ran all Compaction ITs w/ this change and they passed.
   
   [1]:https://github.com/apache/accumulo/blob/5bba2f3cf36717d0e2fabbf4abb93c3455c0619e/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java#L84 [2]:https://github.com/apache/accumulo/blob/5bba2f3cf36717d0e2fabbf4abb93c3455c0619e/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java#L186 [3]:https://github.com/apache/accumulo/blob/5bba2f3cf36717d0e2fabbf4abb93c3455c0619e/server/tserver/src/main/java/org/apache/accumulo/tserver/compactions/CompactionService.java#L359-L365


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fixes bug where external compaction never starts [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5707:
URL: https://github.com/apache/accumulo/pull/5707#issuecomment-3029095785

    There are some notes on what I did to track this bug down.  Started w/ running fate list and kept seeing these compactions not moving.
   
   ```
   TABLE_BULK_IMPORT2 txid: 10ca761c03d9876d  status: SUBMITTED          locked: []              locking: [R:4]           op: PrepBulkImport  created: 2025-07-01T22:56:00.758Z
   TABLE_BULK_IMPORT2 txid: 67be5b2616e8163c  status: SUBMITTED          locked: []              locking: [R:4]           op: PrepBulkImport  created: 2025-07-01T22:56:03.157Z
   TABLE_COMPACT   txid: 2f2ff4546d9052e5  status: IN_PROGRESS        locked: [R:+default, R:4] locking: []              op: CompactionDriver created: 2025-07-01T22:55:29.987Z
   TABLE_BULK_IMPORT2 txid: 6102ab68e0ae9b54  status: SUBMITTED          locked: []              locking: [R:4]           op: PrepBulkImport  created: 2025-07-01T22:56:09.768Z
   TABLE_MERGE     txid: 188da098b244da57  status: SUBMITTED          locked: [R:+default]    locking: [W:4]           op: TableRangeOp    created: 2025-07-01T22:55:59.759Z
   TABLE_BULK_IMPORT2 txid: 545753cd7ac69fd2  status: SUBMITTED          locked: []              locking: [R:4]           op: PrepBulkImport  created: 2025-07-01T22:56:03.246Z
   TABLE_BULK_IMPORT2 txid: 1f12560b9db5834c  status: SUBMITTED          locked: []              locking: [R:4]           op: PrepBulkImport  created: 2025-07-01T22:56:01.417Z
   TABLE_COMPACT   txid: 5481217c586dcaf6  status: IN_PROGRESS        locked: [R:+default, R:4] locking: []              op: CompactionDriver created: 2025-07-01T22:55:22.614Z
   ```
   
   Enabled trace logging in the manager and got some info that indicated that both fate ops were waiting on a single tablet.
   
   ```
   2025-07-01T23:28:17,267 [compact.CompactionDriver] TRACE: FATE[2f2ff4546d9052e5] tablets compacted:33/34 servers contacted:1 expected id:49 compaction extent:4;r165f8;r0786c sleepTime:500
   2025-07-01T23:28:17,267 [compact.CompactionDriver] TRACE: FATE[5481217c586dcaf6] tablets compacted:56/57 servers contacted:1 expected id:48 compaction extent:4<;r003f8 sleepTime:500
   ```
   
   Looked in the metadata table and found the tablet with a lower compact id that was in the range.
   
   ```
   4;r13e11 srv:compact []	50
   4;r16d06 srv:compact []	46
   4;r172ae srv:compact []	50
   4;r172da srv:compact []	50
   ```
   
   Enabled trace logging on the tablet server and saw the following that helped get in the neighborhood of the problem. Needed to filter on the tablet.
   
   ```
   2025-07-02T00:29:17,968 [compactions.CompactionService] TRACE: Did not submit compaction plan 4;r16d06;r13e11 id:default files:Files [allFiles=[[C00009ux.rf, 33437 252441], [A00009dw.rf, 3638005 7727960], [C00009uz.rf, 25527 168294], [I00
   009m2.rf, 25286 0], [I00009m6.rf, 18179 0], [I00009lv.rf, 21772 0], [C00009dz.rf, 18750 132363], [I00009ld.rf, 18232 0], [C00009qf.rf, 27948 204357], [I00009p7.rf, 18369 0]], candidates=[[I00009m6.rf, 18179 0], [I00009lv.rf, 21772 0], [A0
   0009dw.rf, 3638005 7727960], [C00009dz.rf, 18750 132363], [I00009ld.rf, 18232 0], [C00009qf.rf, 27948 204357], [I00009m2.rf, 25286 0]], compacting=[], hints={}] plan:jobs: [CompactionJob [priority=18, executor=e.small, files=[[I00009m6.rf
   , 18179 0], [I00009lv.rf, 21772 0], [A00009dw.rf, 3638005 7727960], [C00009dz.rf, 18750 132363], [I00009ld.rf, 18232 0], [C00009qf.rf, 27948 204357], [I00009m2.rf, 25286 0]], kind=USER]]
   ```
   
   Took a heap dump of the tablet server and ran the following OQL query to find the tablet object in the heap dump.  Got the dir name from the metadata table.  Used the dirName field because its a string, would like to use the extent to find the tablet but that is a byte array and did not know how to query that in OQL.  Would be useful to figure that out.
   
   ```
   select t from org.apache.accumulo.tserver.tablet.Tablet t where t.dirName.toString()=="t-00009q0"
   ```
   
   After finding the tablet in the heap dump was able to find a ExternalJob that indirectly referenced the tablet.  This ExternalJob had a state of running and a null ecid.  Also the ExternalJob was only referenced by CompactoinService.submittedJob.  This was the information that lead to this bug fix.
   
   Not 100% this change will fix this  problem, but it seems like it will .  Also not sure how to test this fix as its a race condition.  May try to see if this problem is reproducible w/o this fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fixes bug where external compaction never starts [accumulo]

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on PR #5707:
URL: https://github.com/apache/accumulo/pull/5707#issuecomment-3029103556

   It seems like you found the one's that weren't running, but do we know "why" the weren't running?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fixes bug where external compaction never starts [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner merged PR #5707:
URL: https://github.com/apache/accumulo/pull/5707


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fixes bug where external compaction never starts [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5707:
URL: https://github.com/apache/accumulo/pull/5707#issuecomment-3029216089

   > It seems like you found the one's that weren't running, but do we know "why" the weren't running?
   
   My suspicion is that CompactoinService.submittedJobs had an ExternalJob object that ExternalCompactionExecutor no longer knew of and would never move out of the running state.  This fix addresses one possible place that could happen, but not sure if its the cause of the bug I saw.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fixes bug where external compaction never starts [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5707:
URL: https://github.com/apache/accumulo/pull/5707#issuecomment-3029872932

   Three runs of randomwalk bulk w/o this fix hung. Two runs w/ this fix passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fixes bug where external compaction never starts [accumulo]

Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on PR #5707:
URL: https://github.com/apache/accumulo/pull/5707#issuecomment-3029098591

   These were some settings for running the bulk test w/ external compactions.  Configured cluster yaml to start 4 compactions for small and 4 for large.
   
   ```
   config -s 'tserver.compaction.major.service.default.planner.opts.executors=[{"name":"small","queue":"small","type":"external","maxSize":"128M"},{"name":"large","queue":"large","type":"external"}]'
   config -s compaction.coordinator.tserver.check.interval=3s
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org