OutOfMemoryError in ReadStage

[Ian Rose <ia...@fullstory.com>]

Hi all -

I had a nasty streak of OOMs earlier today (several on one node, and a
single OOM on one other node).  I've downloaded a few of the hprof files
for local analysis.  In each case, there is a single ReadStage thread with
a huge (> 7.5GB) org.apache.cassandra.db.ArrayBackedSortedColumns
instance.  I'm trying to understand exactly what this means.

1) Does a ReadStage thread only process one query at a time?  If so, then a
reasonable conclusion (I think) would be I had a single query that produced
a ton of results.  If not (if ReadStage threads can work on multiple
queries concurrently) then this volume of data might have been produced by
a combination of queries.

2) My driver (gocql) does not appear to enable paging by default.  Am I
correct in assuming that this should "solve the problem" (more precisely:
avoid OOMs due to me fetching a ton of rows, assuming that is the problem
and not that I am fetching a small number of very large rows)?

3) Is there any way for me (either from the system.log or from the hprof
dumps) to tell what query was currently executing when the process OOMed?
If I dig down in the object hierarchy, I see: Thread -> MessageDeliveryTask
-> message -> payload, which has the right ksName and cfName.  But the
"key" property is a byte array - is there an easy way for me to map this
onto my column key (which has multiple CQL columns in a composite key).

4) Alternatively, is it possible for me to see how many rows had been read
for that query so far?  That way I can at least validate that the problem
was "too many rows" and not "rows are too big".

Many thanks!
- Ian