[jira] [Commented] (CASSANDRA-8692) Coalesce intra-cluster network messages

["Benedict (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309535#comment-14309535 ] 

Benedict commented on CASSANDRA-8692:
-------------------------------------

On the whole I like the patch, but I still have a lot of suggestions, and think it should be changed quite a bit before commit.

first, minor suggestions/nits:
# The bug you spotted some time ago with .size() being passed in to drainTo() is still present.
# For clarity, make DISABLE_COALESCING a strategy, instead of a parameter? Fewer conditional statements in the code, and it'slikely the optimizer will eliminate these calls anyway, since the strategy is final and the body will be a no-op.
# VM options might be good to prefix with "cassandra." or similar - we should decide a uniform approach to this
# parkLoop probably shouldn't loop if it's within a few % of the target sleep time if on a second iteration (might explain your double-sleeping?)
# The haveAllSamples() check seems unnecessary - unless our max coalesce window is >= 134 millis, we should never use the dummy samples?

It also seems to me that it is never useful to not drain the queue; it's not like it costs us very much by proportion to anything else, and the current behaviour is that it is always drained anyway (since we always "block", due to the bug mentioned in nit #1). We can simplify the code in this method quite a bit with this assumption.

More substantively:

It seems that we're missing some useful information to help us make better decisions by only providing the first message we drained to our strategy prior to blocking. IMO, we should:

# Immediately drain the queue, and notify of all sample data points
# Then ask the strategy if we want to sleep; and we should provide it the current outstanding message count to help make this decision

The current moving average strategy admittedly would not work well with this setup, because it only bases its calculation on the most recent 16 samples, and so providing it with extra samples upfront most likely won't help, and might even hurt. This seems counterintuitive behaviour-wise though, as sleeping for some period based on the guessed arrival time of the next message when that message is potentially already in our buffer strikes me as odd; as it is that providing more data points might make the decision worse. I suspect that the first and last messages in a batch dominate the behaviour, and it is probably similar to only tracking these and sleeping for some ratio of the delta.

I would like to propose a _slightly_ more involved strategy, that calculates the moving average over time horizons instead of count horizons. For simplicity, let's say we start with a linear discretization of time (i.e. split the horizon into even sized buckets). Let's also assume that the 1s average is a good barometer. So your 16 buckets would each represent the number of messages that arrived in the prior 16 intervals of 62.5ms (with the boundary bucket not utilised until we cross it). The interesting bit is then how we use this information. Let's also assume a uniform message arrival distribution (again, this can be improved later, but won't likely yield much parameter tweaking can't).

This gives us an average delta between messages of Dns. Let's also say we have N messages in our buffer already.

I can think of two complementary logics to apply off the top of my head. There are plenty of other possible variants of this style of approach, especially if we integrate other data (such as network latencies), but these ones spring to mind as simple and likely yielding much of the potential.

Logic 1:
We have N messages waiting to send; there's no point delaying these unless we intend to appreciably increase N. So let's say we want to double the number of messages in N to make it worthwhile sleeping, otherwise we may as well send what we have without delay. So we calculate XD, and sleep for it ONLY if it is less than the max coalesce interval; if it's greater, we just send immediately. This multiplier can be configurable, but 2x seems likely to work pretty well.

Logic 2:
Our average arrival rate says that we should expect these N messages to have arrived over a timespan of NDns; if they actually arrived in less time (by checking the delta between the first/last), we assume this is a burst of activity and just flush immediately since the non-uniformity of arrival is in our favour. If the opposite is true we might be able to estimate the probability that such a cluster of messages will arrive in the next Xns, but this requires some further thought.

These can both be applied, with Logic 2 overriding Logic 1 if it says to short-circuit the waiting, say. We can then follow up with some research into some more advanced analysis of the data, and integration of other metrics to the decision.

Do these descriptions make sense, and sound reasonable to you?

> Coalesce intra-cluster network messages
> ---------------------------------------
>
>                 Key: CASSANDRA-8692
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8692
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 2.1.4
>
>         Attachments: batching-benchmark.png
>
>
> While researching CASSANDRA-8457 we found that it is effective and can be done without introducing additional latency at low concurrency/throughput.
> The patch from that was used and found to be useful in a real life scenario so I propose we implement this in 2.1 in addition to 3.0.
> The change set is a single file and is small enough to be reviewable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)