multithreaded compaction causes mutation storms?

[Terje Marthinussen <tm...@gmail.com>]

Tested out multithreaded compaction in 0.8 last night.

We had first fed some data with compaction disabled so there was 1000+
sstables on the nodes and I decided to enable multithreaded compaction on
one of them to see how it performed vs. nodes that had no compaction at all.

Since this was sort of to see how it could perform, I set throughput to
128MB/sec (knowing that this was probably a bit more than it could manage)

It quickly generated 24 tmp files for the main CF (24 compaction threads?),
the CPUs got maxed out 90% (2x6 cores) and I started seeing these

 INFO [FlushWriter:1] 2011-04-24 03:07:46,776 Memtable.java (line 238)
Writing Memtable-Test@757483679(23549136/385697094 serialized/live bytes,
32026 ops)
 WARN [ScheduledTasks:1] 2011-04-24 03:07:46,946 MessagingService.java (line
548) Dropped 36506 MUTATION messages in the last 5000ms
 INFO [ScheduledTasks:1] 2011-04-24 03:07:46,947 StatusLogger.java (line 50)
Pool Name                    Active   Pending
 INFO [ScheduledTasks:1] 2011-04-24 03:07:46,947 StatusLogger.java (line 65)
ReadStage                         0         0
 INFO [ScheduledTasks:1] 2011-04-24 03:07:46,948 StatusLogger.java (line 65)
RequestResponseStage              0         3
 INFO [ScheduledTasks:1] 2011-04-24 03:07:46,948 StatusLogger.java (line 65)
ReadRepairStage                   0         0
 INFO [ScheduledTasks:1] 2011-04-24 03:07:46,948 StatusLogger.java (line 65)
MutationStage                    10     39549
 INFO [ScheduledTasks:1] 2011-04-24 03:07:46,949 StatusLogger.java (line 65)
ReplicateOnWriteStage             0         0

That the system is a bit overloaded is not really the question (I wanted to
find out what it could manage), but the curious part is that when checking
tpstats, the Mutation stage was mostly idle, however at seemingly regular
intervals it would get massive amounts of mutations.

Not sure if it could be related, but the log message always showed up just
before the "StatusLogger" printout (but not necessarily before all of them)

Some sort of internal event occuring causing these mutation storms or
something which ends up synchronizing the compaction threads in a way that
causes mutations storms like these?

The messages went away a little while after reducing the throughput
significantly to 6MB/sec...

It does not seem to be a problem normally, just when doing something extreme
like enabling multithreaded compaction when you have hundreds or thousands
of memtables already.

Regards,
Terje