You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/03/03 17:51:15 UTC
[GitHub] [accumulo] milleruntime opened a new issue #2544: Compaction Iterator Bug
milleruntime opened a new issue #2544:
URL: https://github.com/apache/accumulo/issues/2544
I am not sure if this is a bug with the `FileCompactor`, the `SlowIterator` or not a bug at all. While testing 2.1 with my changes in #2541, I have Uno setup to run the `SlowIterator` during a compaction of some CI data. The compaction will start up but it won't get registered in ZK under the fate directory. It will never make it past this section of code in `FileCompactor`:
<pre>
while (itr.hasTop() && env.isCompactionEnabled()) {
mfw.append(itr.getTopKey(), itr.getTopValue());
itr.next();
entriesCompacted++;
if (entriesCompacted % 1024 == 0) {
// Periodically update stats, do not want to do this too often since its volatile
entriesWritten.addAndGet(1024);
}
}
</pre>
I assume the source of the `SlowIterator` will always return true for `hasTop()`, hence why it never makes it past this code. It could just be a bug when using the `SlowIterator` with compactions. But if a user creates an iterator that does something similiar, I wonder if it will run indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime closed issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime closed issue #2544:
URL: https://github.com/apache/accumulo/issues/2544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1059441564
> I'm not seeing the link between the code and the compaction not showing up in ZK.
Yeah I don't know how the `FileCompactor` was running without starting a FATE Tx. I am doing some tracing through the code, which is way more complicated in 2.1 and this is as far as I got:
<pre>
InternalCompactionExecutor.submit()
- new InternalJob
- threadPool.execute(internalJob);
InternalJob
- compactable.compact()
CompactableImpl.compact()
- stats = CompactableUtils.compact(tablet, job, cInfo, compactEnv, compactFiles, tmpFileName);
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1061976510
I think I'm still unsure of what you were expecting to see that you are not seeing. If you want to test the user compaction without having to deal with the system compactions, just set the compaction ratio really high.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1059793157
> > I'm not seeing the link between the code and the compaction not showing up in ZK.
>
> Yeah I don't know how the `FileCompactor` was running without starting a FATE Tx. I am doing some tracing through the code, which is way more complicated in 2.1 and this is as far as I got:
I thought FaTE was only used to coordinate/schedule user-initiated compactions. Regular compactions due to a trigger in the compaction strategy inside a tserver shouldn't need FaTE to compact. The tserver can handle those on their own. Did something change in this regard? Or am I misunderstanding how this is supposed to work? Was this apparent bug observed for a user-initiated compaction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1058441984
It is possible that this was due to the number of entries, which was at about 3 million.
<pre>
3 Million * 5ms = 15000000ms
15000000ms = 250 minutes = 4+ hours
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1058395245
Link to part of the code: https://github.com/apache/accumulo/blob/b6a466348807fe0d23203f3776cbc3119d5a037d/server/base/src/main/java/org/apache/accumulo/server/compaction/FileCompactor.java#L374-L382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1059407005
I'm not seeing the link between the code and the compaction not showing up in ZK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1061906696
OK I think I know what is going on. Steps to reproduce (still not sure if there is a bug though)
1. Create CI table with SlowIterator configured on MajC and test jar copied into accumulo lib
2. Ingest some data (enough to trigger system compact)
3. From the shell, run `compact -t ci`
4. Run `listcompactions` to see that there are only system compactions running
5. Run `fate print` to see the fate TX has started for the user compaction
CI table config:
<pre>
tserver.compaction.major.service.cs1.planner.opts.executors=[{"name":"small","type":"internal","maxSize":"16M","numThreads":8},{"name":"medium","type":"internal","maxSize":"128M","numThreads":4},{"name":"large","type":"internal","numThreads":2}]
table | table.iterator.majc.slow .................. | 1,org.apache.accumulo.test.functional.SlowIterator
table | table.iterator.majc.slow.opt.sleepTime .... | 5
</pre>
Each of the 2 tservers has 10 MajC compactions running on each, with 8 queued each (according to the Monitor). But its more like 24 according to `listcompactions` and the logs.
<pre>
10:27:46 {test-jar} ~/workspace/uno/install/logs/accumulo$ grep "tablet.files" *.log | grep "Compacting 1" | grep small | wc -l
16
10:27:59 {test-jar} ~/workspace/uno/install/logs/accumulo$ grep "tablet.files" *.log | grep "Compacting 1" | grep medium | wc -l
8
</pre>
So it looks like the user compaction gets thrown in the pot with all of the waiting system compactions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1061822359
> I thought FaTE was only used to coordinate/schedule user-initiated compactions. Regular compactions due to a trigger in the compaction strategy inside a tserver shouldn't need FaTE to compact. The tserver can handle those on their own. Did something change in this regard? Or am I misunderstanding how this is supposed to work?
No, you are correct.
> Was this apparent bug observed for a user-initiated compaction?
I am pretty sure it was a user compaction, with the testing I was doing. But it is possible it was a system compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1062054623
> I think I'm still unsure of what you were expecting to see that you are not seeing. If you want to test the user compaction without having to deal with the system compactions, just set the compaction ratio really high.
There isn't a bug. It would be nice to have some insight into the queued compactions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1061919955
The user compaction eventually makes it through:
<pre>
root@uno> listcompactions
SERVER | AGE | TYPE | REASON | READ | WROTE | TABLE | TABLET | INPUT | OUTPUT | ITERATORS | ITERATOR OPTIONS
ip-10-113-12-25:9997 | 40m58s | FULL | SYSTEM | 475K | 475K | ci | 1<;79999999999999a5 | 5 | /1/default_tablet/A000001q.rf_tmp | [] | {}
ip-10-113-12-25:9997 | 40m19s | MAJOR | SYSTEM | 469K | 469K | ci | 1;266666666666666a;2000000000000003 | 5 | /1/t-000005m/C000002q.rf_tmp | [] | {}
ip-10-113-12-25:9997 | 40m17s | MAJOR | SYSTEM | 469K | 469K | ci | 1;733333333333333e;6cccccccccccccd7 | 5 | /1/t-000005y/C000002t.rf_tmp | [] | {}
ip-10-113-12-25:9997 | 12s128ms | MAJOR | USER | 2.05K | 2.05K | ci | 1;666666666666667;6000000000000009 | 10 | /1/t-000005w/C00000it.rf_tmp | [] | {}
ip-10-113-12-25:10000 | 2m45s | MAJOR | USER | 31.7K | 31.7K | ci | 1;399999999999999f;3333333333333338 | 10 | /1/t-000005p/C00000fj.rf_tmp | [] | {}
ip-10-113-12-25:10000 | 2m16s | MAJOR | USER | 26.6K | 26.6K | ci | 1;1333333333333335;0cccccccccccccce | 10 | /1/t-000005j/C00000fk.rf_tmp | [] | {}
ip-10-113-12-25:10000 | 2m3s | MAJOR | USER | 23.6K | 23.6K | ci | 1;4000000000000006;399999999999999f | 10 | /1/t-000005q/C00000fl.rf_tmp | [] | {}
ip-10-113-12-25:10000 | 1m43s | MAJOR | USER | 19.5K | 19.5K | ci | 1;59999999999999a2;533333333333333b | 10 | /1/t-000005u/C00000fm.rf_tmp | [] | {}
root@uno> fate print
2022-03-08T10:44:29,015 [conf.SiteConfiguration] INFO : Found Accumulo configuration on classpath at /home/mike/workspace/uno/install/accumulo-2.1.0-SNAPSHOT/conf/accumulo.properties
txid: 7bd4ad2463222699 status: IN_PROGRESS op: CompactRange locked: [R:+default, R:1] locking: [] top: CompactionDriver created: 2022-03-08T15:10:15.416Z
1 transactions
root@uno>
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] Manno15 commented on issue #2544: Compaction Iterator Bug
Posted by GitBox <gi...@apache.org>.
Manno15 commented on issue #2544:
URL: https://github.com/apache/accumulo/issues/2544#issuecomment-1058444327
@milleruntime Only one way to find out. You definitely have to run it for 4+ hours now...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org