You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jedd Rashbrooke <je...@imagini.net> on 2010/09/17 16:41:33 UTC
Dazed and confused with Cassandra on EC2 ...
Howdi,
I've just landed in an experiment to get Cassandra going, and
fed by PHP via Thrift via Hadoop, all running on EC2. I've been
lurking a bit on the list for a couple of weeks, mostly reading any
threads with the word 'performance' in them. Few people have
anything polite to say about EC2, but I want to just throw out
some observations and get some feedback on whether what
I'm seeing is even approaching any kind of normal.
My background is mostly *nix and networking, with half-way
decent understanding of DB's -- but Cassandra, Hadoop, Thrift
and EC2 are all fairly new to me.
We're using a four-node decently-specced (m2.2xlarge, if you're
EC2-aware) cluster - 32GB, 4-core, if you're not :) I'm using
Ubuntu with the Deb packages for Cassandra and Hadoop, and
some fairly conservative tweaks to things like JVM memory
(bumping them up to 4GB, then 16GB).
One of our insert jobs - a mapper only process - was running
pretty fast a few days ago. Somewhere around a million lines
of input, split into a dozen files, inserting via a Hadoop job in
about a half hour. Happy times. This was when the cluster
was modestly sized - 20-50GB. It's now about 200GB, and
performance has dropped by an order of magnitude - perhaps
5-6 hours to do the same amount of work, using the same
codebase and the same input data.
I've read that reads slow down as the DB grows, but had an
expectation that writes would be consistently snappy. How
surprising is this performance drop given the DB growth?
My 4-node cluster started off as a 2-node - and now nodetool
ring suggests the two original nodes are 200GB each, and
the newer two are 40GB. Is this normal? Would a rebalance
likely improve performance substantially? My feeling is that
it would be expensive to perform.
EC2 seems to get a bad rap, and we're feeling quite a bit of
pain, which is sad given the (on paper) spec of the machines,
and the cost - over US$3k/month for the cluster. I've split
Cassandra commitlog, Cassandra data, hadoop(hdfs) and
tmp onto separate 'spindles' - observations so far suggest
late '90's disk IO speed (15MB max sustained writes, one
machine, one disk to another), and consistently inconsistent
performance (identical machine next to it running the same
task at the same time was getting 28MB) over several hours.
Cassandra nodes seems to disappear too easily - even
with just one core (out of four) maxed out with a jsvc task,
minimal disk or network activity, the machine feels very
sluggish. Tailing the cassandra logs hints that it's doing
hinted handoffs and occasionally compaction tasks. I've
never seen this kind of behaviour - and suspect this is
more a feature of EC2.
Gossip now seems to be pining the loss of an older machine
(that I stupidly took offline briefly - EC2 gave it a new IP address
when it came back). There's nothing in the storage-conf to
refer to the old address, all 4 Cassandra daemons have been
re-started several times since, but gossip occasionally (a day
later) says that it is looking for it - and more worrying that
it is 'now part of the cluster'. I'm unsure if this is just an
irritation or part of the underlying problem.
What I'm going to do next is to try importing some data into
a local machine - it's just time-consuming to pull in our S3
data - and see if I can fake up to around the same capacity
and watch for performance degradation.
I'm also toying with the idea of going from 4 to 8 nodes,
but I'm clueless on whether / how much this would help.
As I say, though, I'm keen on anyone else's observations on
my observations - I'm painfully aware that I'm juggling a lot
of unknown factors at the moment.
cheers,
Jedd.
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Peter Schuller <pe...@infidyne.com>.
> Heap *shrinkage* on the other hand is another matter and for both CMS
> and G1, shrinkage never happens except on Full GC, unfortunately. I'm
And to be clear, the implication here is that shrinkage normally
doesn't happen. The implication is *not* that you see fallbacks to
full GC for the purpose of shrinkage.
--
/ Peter Schuller
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Peter Schuller <pe...@infidyne.com>.
> The main reason to set Xms=Xmx is so mlockall can tag the entire heap
> as "don't swap this out" on startup. Secondarily whenever the heap
> resizes upwards the JVM does a stop-the-world gc, but no, not really a
> big deal when your uptime is in days or weeks.
I'm not sure where this is coming from, assuming you mean a *full* stw
GC. I cannot remember ever observing this in "all my hears of heap
growth ;)", and I don't see why it would be the case.
Heap *shrinkage* on the other hand is another matter and for both CMS
and G1, shrinkage never happens except on Full GC, unfortunately. I'm
hoping G1 will eventually improve here since its compacting design
should fundamentally make it feasible to implement efficiently.
Disregarding the mlockall issue which is pretty specific to Cassandra,
the typical reason, as I have understood it, for setting Xms=Xmx on
machine dedicatedly running some particular JVM is that if you have to
reserve Xmx amount of memory anyway, and perhaps even expect that to
be reached, it is generally more efficiency to just let the JVM use it
all from the start - due to the usual effect of GC being more
efficient with larger heap sizes and in order to avoid bad policy
decisions of the GC causing excessive GC activity rather than just
growing the heap which you're fine with anyway.
--
/ Peter Schuller
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Jonathan Ellis <jb...@gmail.com>.
On Fri, Oct 8, 2010 at 4:54 AM, Jedd Rashbrooke
<je...@imagini.net> wrote:
> On 8 October 2010 02:05, Matthew Dennis <md...@riptano.com> wrote:
>> Also, in general, you probably want to set Xms = Xmx (regardless of the
>> value you eventually decide on for that).
>
> Matthew - we'd just about reached that conclusion! Is it as big an
> issue for clusters that are up for days at a time, with maybe Xmx:3G?
> I'd kind of assumed that it'd get to its max pretty quickly. I haven't
> been watching that with jconsole, but might watch it on the next startup.
The main reason to set Xms=Xmx is so mlockall can tag the entire heap
as "don't swap this out" on startup. Secondarily whenever the heap
resizes upwards the JVM does a stop-the-world gc, but no, not really a
big deal when your uptime is in days or weeks.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Jedd Rashbrooke <je...@imagini.net>.
On 7 October 2010 20:49, Peter Schuller <pe...@infidyne.com> wrote:
> ... if you "waste" 10-15 gigs of RAM on the JVM heap for a
> Cassandra instances which could live with e.g. 1 GB, you're actively
> taking away those 10-15 gigs of RAM from the operating system to use
> for the buffer cache. Particularly if you're I/O bound on reads then,
> this could have very detrimental effects (assuming the data set is
Peter - in my defense, I was doing this on 68GB mega-boxes :) But,
yeah, we're seeing far more *predictable* results from the OS's ability
to cache file system access, than Cassandra-with-Java's ability to
manage its own memory. My own sys-admin experience previously
had been on real hardware, usually 16GB, occasionally 32GB boxes,
but with *much* larger file systems than we're using with Cassandra,
hence my earlier optimism at leaving a 'mere' 30-40GB for FS cache.
On 8 October 2010 02:05, Matthew Dennis <md...@riptano.com> wrote:
> Also, in general, you probably want to set Xms = Xmx (regardless of the
> value you eventually decide on for that).
Matthew - we'd just about reached that conclusion! Is it as big an
issue for clusters that are up for days at a time, with maybe Xmx:3G?
I'd kind of assumed that it'd get to its max pretty quickly. I haven't
been watching that with jconsole, but might watch it on the next startup.
As to heap size determination ... the equation offered:
MemtableThroughputInMB * 3 * (number of ColumnFamilies)
+ 1G + (size of internal caches)
-- has (for us, at least) 3 constants and 2 quite fuzzy values.
The experiments continue!
Jedd.
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Matthew Dennis <md...@riptano.com>.
Also, in general, you probably want to set Xms = Xmx (regardless of the
value you eventually decide on for that).
If you set them equal, the JVM will just go ahead and allocate that amount
on startup. If they're different, then when you grow above Xms it has to
allocate more and move a bunch of stuff around. It may have to do this
multiple times. Note that it does this as the worst time possible (i.e.
under heavy load, which is likely what caused you to grow past Xms in the
first place).
On Thu, Oct 7, 2010 at 2:49 PM, Peter Schuller
<pe...@infidyne.com>wrote:
> > There's some words on the 'Net that - the recent pages on
> > Riptano's site in fact - that strongly encourage scaling left
> > and right, rather than beefing up the boxes - and certainly
> > we're seeing far less bother from GC using a much smaller
> > heap - previously we'd been going up to 16GB, or even
> > higher. This is based on my previous positive experiences
> > of getting better performance from memory hog apps (eg.
> > Java) by giving them more memory. In any case, it seems
> > that using large amounts of memory on EC2 is just asking
> > for trouble.
>
> Keep in mind that while GC tends to be more efficient with larger heap
> sizes, that does not always translate into better overall performance
> when other things have to be considered. In particular, in the case of
> Cassandra, if you "waste" 10-15 gigs of RAM on the JVM heap for a
> Cassandra instances which could live with e.g. 1 GB, you're actively
> taking away those 10-15 gigs of RAM from the operating system to use
> for the buffer cache. Particularly if you're I/O bound on reads then,
> this could have very detrimental effects (assuming the data set is
> sufficiently small and locality is such that 15 GB of extra buffer
> cache makes a difference; usually, but not always, this is the case).
>
> So with Cassandra, in the general case, you definitely want to keep
> hour heap size reasonable in relation to the actual live set (amount
> of actually reachable data), rather than just cranking it up as much
> as possible.
>
> (The main issue here is also keeping it high enough to not OOM, given
> that exact memory demands are hard to predict; it would be absolutely
> great if the JVM was better at maintaining a reasonable heap size to
> live set size ratio so that much less tweaking of heap sizes was
> necessary, but this is not the case.)
>
> --
> / Peter Schuller
>
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Peter Schuller <pe...@infidyne.com>.
> There's some words on the 'Net that - the recent pages on
> Riptano's site in fact - that strongly encourage scaling left
> and right, rather than beefing up the boxes - and certainly
> we're seeing far less bother from GC using a much smaller
> heap - previously we'd been going up to 16GB, or even
> higher. This is based on my previous positive experiences
> of getting better performance from memory hog apps (eg.
> Java) by giving them more memory. In any case, it seems
> that using large amounts of memory on EC2 is just asking
> for trouble.
Keep in mind that while GC tends to be more efficient with larger heap
sizes, that does not always translate into better overall performance
when other things have to be considered. In particular, in the case of
Cassandra, if you "waste" 10-15 gigs of RAM on the JVM heap for a
Cassandra instances which could live with e.g. 1 GB, you're actively
taking away those 10-15 gigs of RAM from the operating system to use
for the buffer cache. Particularly if you're I/O bound on reads then,
this could have very detrimental effects (assuming the data set is
sufficiently small and locality is such that 15 GB of extra buffer
cache makes a difference; usually, but not always, this is the case).
So with Cassandra, in the general case, you definitely want to keep
hour heap size reasonable in relation to the actual live set (amount
of actually reachable data), rather than just cranking it up as much
as possible.
(The main issue here is also keeping it high enough to not OOM, given
that exact memory demands are hard to predict; it would be absolutely
great if the JVM was better at maintaining a reasonable heap size to
live set size ratio so that much less tweaking of heap sizes was
necessary, but this is not the case.)
--
/ Peter Schuller
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Jedd Rashbrooke <je...@imagini.net>.
Hi Peter,
Thanks again for your time and thoughts on this problem.
We think we've got a bit ahead of the problem by just
scaling back (quite savagely) on the rate that we try to
hit the cluster. Previously, with a surplus of optimism,
we were throwing very big Hadoop jobs at Cassandra,
including what I understand to be a worst-case usage
(random reads).
Now we're throttling right back on the number of parallel
jobs that we fire from Hadoop, and we're seeing better
performance, in terms of the boxes generally staying up
as far as nodetool and other interactive sessions are
concerned.
As discussed, we've adopted quite a number of different
approaches with GC - at the moment we've returned to:
JVM_OPTS=" \
-ea \
-Xms2G \
-Xmx3G \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:SurvivorRatio=8 \
-XX:MaxTenuringThreshold=1 \
-XX:+HeapDumpOnOutOfMemoryError \
-Dcom.sun.management.jmxremote.port=8080 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
... which is much closer to the default as shipped - notable
change is the heap size, which out of the box comes as 1G.
There's some words on the 'Net that - the recent pages on
Riptano's site in fact - that strongly encourage scaling left
and right, rather than beefing up the boxes - and certainly
we're seeing far less bother from GC using a much smaller
heap - previously we'd been going up to 16GB, or even
higher. This is based on my previous positive experiences
of getting better performance from memory hog apps (eg.
Java) by giving them more memory. In any case, it seems
that using large amounts of memory on EC2 is just asking
for trouble.
And because it's Amazon, more smaller machines generally
works out as the same CPU grunt per dollar, of course ..
although the management costs go up.
To answer your last question there - we'd been using some
pretty beefy EC2 boxes, but now we think we'll head back
to the 2-core 7GB medium-ish sized machines I think.
All IO still runs like a dog no matter how much money you
spend, sadly.
cheers,
Jedd.
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Peter Schuller <pe...@infidyne.com>.
(sorry for the delay in following up on this thread)
> Actually, there's a question - is it 'acceptable' do you think
> for GC to take out a small number of your nodes at a time,
> so long as the bulk (or at least where RF is > nodes gone
> on STW GC) of the nodes are okay? I suspect this is a
> question peculiar to Amazon EC2, as I've never seen a box
> rendered non-communicative by a single core flat-lining.
Well, first of all I still find it very strange that GC takes nodes
down at all, unless one is specifically putting sufficiant CPU load on
the cluster that e.g. concurrent GC causes a problem. But in
particular if you're still seeing those crazy long GC pause times
still, IMO something is severaly wrong and I would not personally
recommend going to production with that unresolved since whatever the
cause is, may suddenly start having other effects. Severely long
ParNew pause times are really not expected; the only two major reasons
I can think of, at least when running on real hardware, and barring
JVM bugs, are (1) swapping, and (2) possibly extreme performance
penalties associated with a very full old generation in which case the
solution is "larger heap". I don't remember whether you indicated any
heap statistics so I'm not sure whether (2) is a possibility. But I
would expect OutOfMemory errors long before a ParNew takes 300+
*seconds*, just out of JVM policies w.r.t. acceptable GC efficiency.
Bottom line: 300+ seconds for a ParNew collections is *way way way*
out there. 300 *milli*-seconds is more along the lines of what one
might expect (usually lower than that). Even if you can seemingly
lessen the impact by using the throughput collector, I wouldn't be
comfortable with shrugging off whatever is happening.
That said, in terms of the effects on the cluster: I have not had much
hands-on experience with this, but I believe you'd expect a definite
visible from the point of view of clients. Cassandra is not optimized
for instantly detecting slow nodes and transparently working around
them with zero impact on clients; I don't think it is recommended to
be running a cluster with nodes regularly bouncing in and our, for
whatever reason, if it can be avoided.
Not sure what to say, other than to strongly recommend getting to the
bottom of this problem which seems non-specific to Cassandra, before
relying on the system in a production setting. The extremity of the
issues you're seeing are far beyond what I would ever expect even
allowing for "who knows what EC2 is doing or what other people are
running on the machine", except for the hypothesis that they
over-commit memory and the extreme latencies are due to swapping. But
if that is what is happening, that just tells me that EC2 is unusable
for this type of thing, but I still think it's far fetched since the
impact should be significant on a great number of customers of theirs.
I forget and I didn't find it by brief sifting through thread history;
were you running on small EC2 instances or larger ones?
--
/ Peter Schuller
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Jedd Rashbrooke <je...@imagini.net>.
Peter - my apologies for the slow response - we had
to divert down a 'Plan B' approach last week involving
MySQL, memcache, redis and various other uglies.
On 20 September 2010 23:11, Peter Schuller <pe...@infidyne.com> wrote:
> Are you running an old JVM by any chance? (Just grasping for straws.)
JVM is Sun's 1.6 - I've been caught out once before with
openjdk's performance challenges, so I'm particularly
careful with this now.
> Hmm. I can see useless spinning decreasing efficiency, but the numbers
> from your log are really extreme. Do you have a URL / bug id or
> anything that one can read up on about this?
We've rebuilt the Cassandra cluster this week, avoiding
Hadoop entirely - partly to reduce the variables in play,
and partly because it looks like we'll only need two 'feeder'
nodes for our jobs with the size of Cassandra cluster that
we're likely going to end up with (10-12 ish). Any ratio
higher than that seems to, on EC2 at least, cause too many
fails on the Cassandra side.
Actually, there's a question - is it 'acceptable' do you think
for GC to take out a small number of your nodes at a time,
so long as the bulk (or at least where RF is > nodes gone
on STW GC) of the nodes are okay? I suspect this is a
question peculiar to Amazon EC2, as I've never seen a box
rendered non-communicative by a single core flat-lining.
By the end of this week we hope to have a better idea (mind,
I've thought that for the past 5 weeks of experimenting). If I'm
back to square one at that point I'll start pastebining some logs
and configs. Increasingly, I'm convinced that many of these
problems would be solved if we hosted our own servers.
cheers,
Jedd.
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Peter Schuller <pe...@infidyne.com>.
> We think we might have cracked the underlying problem
> though, and it might be similar to the 'behind the scenes
> swap thing' (sadly I suspect that such things might actually
> be happening -- plus I thought that memory overcommit wasn't
> possible with Xen - only with VMware - but I guess they
> could have done all kinds of things with Xen by now over
> there.)
I didn't think Xen supported that until very recently either. I was
just speculating that perhaps something like that was happening (it
would certainly make sense from a business perspective to try to
leverage the ability to over-commit from Amazon's perspective). At the
same time I would expect such practices to very easily have very
negative effects so I would be a bit surprised if they did this.
Are you running an old JVM by any chance? (Just grasping for straws.)
> There's a spinlock problem that's been identified elsewhere
> where the JVM mis-detects the number of cores it has
> running - based on the underlying architecture - and so
Hmm. I can see useless spinning decreasing efficiency, but the numbers
from your log are really extreme. Do you have a URL / bug id or
anything that one can read up on about this?
--
/ Peter Schuller
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Jedd Rashbrooke <je...@imagini.net>.
Hi Peter,
We were logging the GC output as per this before, have since
taken it out, but will put it back in I think.
Apropos logging - I've found that with RMI to our boxes at
EC2 I've had to do the ugly thing with this:
-Djava.rmi.server.hostname=<external / public IP address>
.. which then renders nodetool useless, as it can't talk to
the localhost or internal IP address / hostname any more.
The clocks - I'm pretty sure these are very accurate, but
will investigate this tomorrow morning just in case there's
some drift happening.
We think we might have cracked the underlying problem
though, and it might be similar to the 'behind the scenes
swap thing' (sadly I suspect that such things might actually
be happening -- plus I thought that memory overcommit wasn't
possible with Xen - only with VMware - but I guess they
could have done all kinds of things with Xen by now over
there.)
There's a spinlock problem that's been identified elsewhere
where the JVM mis-detects the number of cores it has
running - based on the underlying architecture - and so
we've reverted to parallel GC and forced the number of
threads:
-XX:+UseParallelGC
-XX:MaxGCPauseMillis=100
-XX:ParallelGCThreads=3"
It *seems* to be working a bit better at the moment, but I'll
be more comfortable with feeling optimistic after a night's
worth of jobs have been thrown at it :)
j.
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Peter Schuller <pe...@infidyne.com>.
> Nope - no swap enabled.
Something is seriously weird, unless the system clock is broken... Given:
INFO [GC inspection] 2010-09-20 15:27:42,046 GCInspector.java (line
129) GC for ParNew: 325411 ms, 84284896 reclaimed leaving 640770336
used; max is 25907560448
INFO [GC inspection] 2010-09-20 15:34:15,924 GCInspector.java (line
129) GC for ParNew: 372272 ms, 82471240 reclaimed leaving 671616240
used; max is 25907560448
We have *extremely* slow ParNew:s on a heap that is not even close to
being full. I highly doubt fragmentation is causing this kind of
extremity.
I wonder if it is possible that EC2 instances are over-committed on
memory such that swapping is happening behind the scenes on the
host... but I have always assumed memory was not over-committed on
EC2.
Can you run with -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimestamps?
--
/ Peter Schuller
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Dave Gardner <da...@imagini.net>.
One other question for the list:
I gather GMFD is "gossip stage" - but what does this actually mean? Is it an
issue to have 203 pending operations?
Thanks
Dave
INFO [GC inspection] 2010-09-20 16:56:12,792 GCInspector.java (line 129) GC
for ParNew: 127970 ms, 570382800 reclaimed leaving 460688576 used; max is
6576406528
INFO [GC inspection] 2010-09-20 16:56:12,803 GCInspector.java (line 150)
Pool Name Active Pending
INFO [GC inspection] 2010-09-20 16:56:12,803 GCInspector.java (line 156)
STREAM-STAGE 0 0
INFO [GC inspection] 2010-09-20 16:56:12,803 GCInspector.java (line 156)
RESPONSE-STAGE 2 2
INFO [GC inspection] 2010-09-20 16:56:12,804 GCInspector.java (line 156)
ROW-READ-STAGE 12 12
INFO [GC inspection] 2010-09-20 16:56:12,804 GCInspector.java (line 156)
LB-OPERATIONS 0 0
INFO [GC inspection] 2010-09-20 16:56:12,804 GCInspector.java (line 156)
MISCELLANEOUS-POOL 0 0
INFO [GC inspection] 2010-09-20 16:56:12,804 GCInspector.java (line 156)
GMFD 1 203
INFO [GC inspection] 2010-09-20 16:56:12,805 GCInspector.java (line 156)
CONSISTENCY-MANAGER 0 0
INFO [GC inspection] 2010-09-20 16:56:12,805 GCInspector.java (line 156)
LB-TARGET 0 0
INFO [GC inspection] 2010-09-20 16:56:12,805 GCInspector.java (line 156)
ROW-MUTATION-STAGE 0 0
INFO [GC inspection] 2010-09-20 16:56:12,806 GCInspector.java (line 156)
MESSAGE-STREAMING-POOL 0 0
INFO [GC inspection] 2010-09-20 16:56:12,806 GCInspector.java (line 156)
LOAD-BALANCER-STAGE 0 0
INFO [GC inspection] 2010-09-20 16:56:12,806 GCInspector.java (line 156)
FLUSH-SORTER-POOL 0 0
INFO [GC inspection] 2010-09-20 16:56:12,806 GCInspector.java (line 156)
MEMTABLE-POST-FLUSHER 0 0
INFO [GC inspection] 2010-09-20 16:56:12,807 GCInspector.java (line 156)
AE-SERVICE-STAGE 0 0
INFO [GC inspection] 2010-09-20 16:56:12,807 GCInspector.java (line 156)
FLUSH-WRITER-POOL 0 0
INFO [GC inspection] 2010-09-20 16:56:12,807 GCInspector.java (line 156)
HINTED-HANDOFF-POOL 0 0
INFO [GC inspection] 2010-09-20 16:56:12,808 GCInspector.java (line 161)
CompactionManager n/a 1
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Dave Gardner <da...@imagini.net>.
Nope - no swap enabled.
top - 16:53:14 up 12 days, 6:11, 3 users, load average: 1.99, 2.63, 5.03
Tasks: 133 total, 1 running, 132 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 35840228k total, 33077580k used, 2762648k free, 263388k buffers
Swap: 0k total, 0k used, 0k free, 29156108k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP
COMMAND
3398 root 20 0 203g 1.2g 186m S 0 3.5 5124095h 202g
jsvc
4162 hadoop 20 0 9095m 84m 9544 S 0 0.2 5124095h 8.8g
java
32153 hadoop 20 0 9057m 154m 9416 S 0 0.4 5124095h 8.7g
java
18091 hadoop 20 0 9257m 561m 9460 S 0 1.6 17232821w 8.5g
java
4267 hadoop 20 0 2400m 85m 9404 S 0 0.2 5124095h 2.3g
java
4289 hadoop 20 0 2337m 78m 9348 S 0 0.2 0:01.44 2.2g
java
On 20 September 2010 17:48, Peter Schuller <pe...@infidyne.com>wrote:
> > Can anyone help shed any light on why this might be happening? We've
> tried a
> > variety of JVM settings to alleviate this; currently with no luck.
>
> Extremely long ParNew (young generations) pause times are almost
> always due to swapping. Are you swapping?
>
> --
> / Peter Schuller
>
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Peter Schuller <pe...@infidyne.com>.
> Can anyone help shed any light on why this might be happening? We've tried a
> variety of JVM settings to alleviate this; currently with no luck.
Extremely long ParNew (young generations) pause times are almost
always due to swapping. Are you swapping?
--
/ Peter Schuller
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Dave Gardner <da...@imagini.net>.
As a follow up to this conversation; we are still having issues with our
Cassandra cluster on EC2.
It *looks* to be related to Garbage Collection; however we aren't sure what
the root cause of the problem is. Here is an extract from logs:
INFO [GMFD:1] 2010-09-20 15:22:00,242 Gossiper.java (line 578) InetAddress
/10.102.57.197 is now UP
INFO [HINTED-HANDOFF-POOL:1] 2010-09-20 15:22:00,242
HintedHandOffManager.java (line 165) Started hinted handoff for endPoint /
10.102.57.197
INFO [HINTED-HANDOFF-POOL:1] 2010-09-20 15:22:00,247
HintedHandOffManager.java (line 222) Finished hinted handoff of 0 rows to
endpoint /10.102.57.197
INFO [GC inspection] 2010-09-20 15:27:42,046 GCInspector.java (line 129) GC
for ParNew: 325411 ms, 84284896 reclaimed leaving 640770336 used; max is
25907560448
INFO [WRITE-/10.102.7.187] 2010-09-20 15:27:42,052
OutboundTcpConnection.java (line 105) error writing to /10.102.7.187
INFO [GC inspection] 2010-09-20 15:27:42,082 GCInspector.java (line 150)
Pool Name Active Pending
INFO [GC inspection] 2010-09-20 15:27:42,083 GCInspector.java (line 156)
STREAM-STAGE 0 0
INFO [GC inspection] 2010-09-20 15:27:42,083 GCInspector.java (line 156)
RESPONSE-STAGE 0 3
INFO [GC inspection] 2010-09-20 15:27:42,083 GCInspector.java (line 156)
ROW-READ-STAGE 6 25
INFO [GC inspection] 2010-09-20 15:27:42,084 GCInspector.java (line 156)
LB-OPERATIONS 0 0
INFO [GC inspection] 2010-09-20 15:27:42,084 GCInspector.java (line 156)
MISCELLANEOUS-POOL 0 0
INFO [GC inspection] 2010-09-20 15:27:42,084 GCInspector.java (line 156)
GMFD 1 129
INFO [GC inspection] 2010-09-20 15:27:42,084 GCInspector.java (line 156)
CONSISTENCY-MANAGER 0 0
INFO [GC inspection] 2010-09-20 15:27:42,085 GCInspector.java (line 156)
LB-TARGET 0 0
INFO [GC inspection] 2010-09-20 15:27:42,085 GCInspector.java (line 156)
ROW-MUTATION-STAGE 1 1
INFO [GC inspection] 2010-09-20 15:27:42,085 GCInspector.java (line 156)
MESSAGE-STREAMING-POOL 0 0
INFO [GC inspection] 2010-09-20 15:27:42,086 GCInspector.java (line 156)
LOAD-BALANCER-STAGE 0 0
INFO [GC inspection] 2010-09-20 15:27:42,086 GCInspector.java (line 156)
FLUSH-SORTER-POOL 0 0
INFO [GC inspection] 2010-09-20 15:27:42,086 GCInspector.java (line 156)
MEMTABLE-POST-FLUSHER 0 0
INFO [GC inspection] 2010-09-20 15:27:42,086 GCInspector.java (line 156)
AE-SERVICE-STAGE 0 0
INFO [GC inspection] 2010-09-20 15:27:42,087 GCInspector.java (line 156)
FLUSH-WRITER-POOL 0 0
INFO [GC inspection] 2010-09-20 15:27:42,087 GCInspector.java (line 156)
HINTED-HANDOFF-POOL 0 0
INFO [GC inspection] 2010-09-20 15:27:42,087 GCInspector.java (line 161)
CompactionManager n/a 0
INFO [GMFD:1] 2010-09-20 15:27:42,088 Gossiper.java (line 592) Node /
10.102.7.187 has restarted, now UP again
INFO [GMFD:1] 2010-09-20 15:27:42,089 StorageService.java (line 548) Node /
10.102.7.187 state jump to normal
INFO [GMFD:1] 2010-09-20 15:27:42,089 StorageService.java (line 555) Will
not change my token ownership to /10.102.7.187
INFO [HINTED-HANDOFF-POOL:1] 2010-09-20 15:27:42,089
HintedHandOffManager.java (line 165) Started hinted handoff for endPoint /
10.102.7.187
INFO [HINTED-HANDOFF-POOL:1] 2010-09-20 15:27:42,112
HintedHandOffManager.java (line 222) Finished hinted handoff of 0 rows to
endpoint /10.102.7.187
INFO [WRITE-/10.102.7.187] 2010-09-20 15:27:42,389
OutboundTcpConnection.java (line 105) error writing to /10.102.7.187
INFO [WRITE-/10.102.57.197] 2010-09-20 15:34:15,911
OutboundTcpConnection.java (line 105) error writing to /10.102.57.197
INFO [GC inspection] 2010-09-20 15:34:15,924 GCInspector.java (line 129) GC
for ParNew: 372272 ms, 82471240 reclaimed leaving 671616240 used; max is
25907560448
INFO [GC inspection] 2010-09-20 15:34:15,925 GCInspector.java (line 150)
Pool Name Active Pending
INFO [GC inspection] 2010-09-20 15:34:15,925 GCInspector.java (line 156)
STREAM-STAGE 0 0
INFO [GC inspection] 2010-09-20 15:34:15,926 GCInspector.java (line 156)
RESPONSE-STAGE 0 21
INFO [Timer-0] 2010-09-20 15:34:15,926 Gossiper.java (line 180) InetAddress
/10.102.7.187 is now dead.
INFO [GC inspection] 2010-09-20 15:34:15,926 GCInspector.java (line 156)
ROW-READ-STAGE 1 85
INFO [GC inspection] 2010-09-20 15:34:15,926 GCInspector.java (line 156)
LB-OPERATIONS 0 0
INFO [GC inspection] 2010-09-20 15:34:15,935 GCInspector.java (line 156)
MISCELLANEOUS-POOL 0 0
INFO [GC inspection] 2010-09-20 15:34:15,935 GCInspector.java (line 156)
GMFD 1 548
INFO [GC inspection] 2010-09-20 15:34:15,935 GCInspector.java (line 156)
CONSISTENCY-MANAGER 0 0
INFO [GC inspection] 2010-09-20 15:34:15,936 GCInspector.java (line 156)
LB-TARGET 0 0
INFO [GC inspection] 2010-09-20 15:34:15,936 GCInspector.java (line 156)
ROW-MUTATION-STAGE 0 0
INFO [GC inspection] 2010-09-20 15:34:15,936 GCInspector.java (line 156)
MESSAGE-STREAMING-POOL 0 0
INFO [GC inspection] 2010-09-20 15:34:15,937 GCInspector.java (line 156)
LOAD-BALANCER-STAGE 0 0
INFO [GC inspection] 2010-09-20 15:34:15,937 GCInspector.java (line 156)
FLUSH-SORTER-POOL 0 0
INFO [GC inspection] 2010-09-20 15:34:15,937 GCInspector.java (line 156)
MEMTABLE-POST-FLUSHER 0 0
INFO [GC inspection] 2010-09-20 15:34:15,938 GCInspector.java (line 156)
AE-SERVICE-STAGE 0 0
INFO [GC inspection] 2010-09-20 15:34:15,938 GCInspector.java (line 156)
FLUSH-WRITER-POOL 0 0
INFO [GC inspection] 2010-09-20 15:34:15,938 GCInspector.java (line 156)
HINTED-HANDOFF-POOL 0 0
INFO [GC inspection] 2010-09-20 15:34:15,938 GCInspector.java (line 161)
CompactionManager n/a 0
INFO [WRITE-/10.102.57.197] 2010-09-20 15:34:15,931
OutboundTcpConnection.java (line 105) error writing to /10.102.57.197
INFO [GMFD:1] 2010-09-20 15:34:15,960 Gossiper.java (line 578) InetAddress
/10.102.7.187 is now UP
INFO [GMFD:1] 2010-09-20 15:34:15,961 Gossiper.java (line 592) Node /
10.102.57.197 has restarted, now UP again
INFO [GMFD:1] 2010-09-20 15:34:15,961 StorageService.java (line 548) Node /
10.102.57.197 state jump to normal
INFO [GMFD:1] 2010-09-20 15:34:15,961 StorageService.java (line 555) Will
not change my token ownership to /10.102.57.197
INFO [HINTED-HANDOFF-POOL:1] 2010-09-20 15:34:15,961
HintedHandOffManager.java (line 165) Started hinted handoff for endPoint /
10.102.7.187
Current thoughts - launching a GC for ParNew at 15:27:42, which takes 325411
ms, followed (almost immediately) by another at 15:34:15 which takes 372272
ms is probably sub-optimal.
INFO [GC inspection] 2010-09-20 15:27:42,046 GCInspector.java (line 129) GC
for ParNew: 325411 ms, 84284896 reclaimed leaving 640770336 used; max is
25907560448
INFO [GC inspection] 2010-09-20 15:34:15,924 GCInspector.java (line 129) GC
for ParNew: 372272 ms, 82471240 reclaimed leaving 671616240 used; max is
25907560448
Can anyone help shed any light on why this might be happening? We've tried a
variety of JVM settings to alleviate this; currently with no luck.
Dave
On 17 September 2010 17:04, Jedd Rashbrooke <je...@imagini.net>wrote:
> Hi Rob,
>
> Thanks for your suggestions. I should have been a bit more verbose
> in my platform description -- I'm using 64-bit instances, which I think
> in a Ben Black video I saw led to a sensible default usage of mmap
> when left at auto. Should I look at forcing this setting?
>
> > You don't mention which version of the deb package you're using, but :
>
> I'm using 0.6.5 - the ones bundled by Eric Evans <ee...@apache.org>
>
> Because these are 32GB machines, I've not configured them with
> any swap at all. I've rarely done this in the past - but was aware
> there was this swap-hell scenario with JVM's, and the rationale
> makes sense -- it's better to have the JVM crash ''cleanly" than
> to have it grind the machine to a halt and make it impossible
> to get onto the machine to kill the process.
>
> cheers,
> Jedd.
>
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Jedd Rashbrooke <je...@imagini.net>.
Hi Rob,
Thanks for your suggestions. I should have been a bit more verbose
in my platform description -- I'm using 64-bit instances, which I think
in a Ben Black video I saw led to a sensible default usage of mmap
when left at auto. Should I look at forcing this setting?
> You don't mention which version of the deb package you're using, but :
I'm using 0.6.5 - the ones bundled by Eric Evans <ee...@apache.org>
Because these are 32GB machines, I've not configured them with
any swap at all. I've rarely done this in the past - but was aware
there was this swap-hell scenario with JVM's, and the rationale
makes sense -- it's better to have the JVM crash ''cleanly" than
to have it grind the machine to a halt and make it impossible
to get onto the machine to kill the process.
cheers,
Jedd.
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Robert Coli <rc...@digg.com>.
On 9/17/10 7:41 AM, Jedd Rashbrooke wrote:
> Happy times. This was when the cluster
> was modestly sized - 20-50GB. It's now about 200GB, and
> performance has dropped by an order of magnitude - perhaps
> 5-6 hours to do the same amount of work, using the same
> codebase and the same input data.
You don't mention which version of the deb package you're using, but :
https://issues.apache.org/jira/browse/CASSANDRA-1214
Is a performance hit which occurs as a result of the growth in data
size, and could be your issue.
Have you checked for swapping behavior like the above while your system
is unhappy?
=Rob
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Jedd Rashbrooke <je...@imagini.net>.
Hi Dave,
Thank you for your response.
I can clarify a couple of things here:
> 2. You grew from 2 nodes to 4, but the original 2 nodes have 200GB and the 2
> new ones have 40 GB. What's the recommended practice for rebalancing (i.e.,
> when should you do it), what's the actual procedure, and what's the expected
> impact of it?
+ is it likely to cause a problem in the short term if I don't (ie.
if I just wait
until 'normal activity' to somehow even out the distribution of data).
> 3. Cassandra nodes "disappear". (I'm not quite clear what this means.)
Nodetool reports the node as down. I'm seeing lots of machine-x is DOWN
in the logs. Flapping, actually. I don't have any swap configured (which I've
read somewhere might induce flapping).
The machine also feels like it goes on a hiatus - separately, but typically
observed at the same time. Tail -f on the Cassandra logs delays for several
minutes, pending ssh's to the box also stall until 'something' happens that
releases the machine from its slumber. Typically that something is a
message in the logs that a compaction of a hintedhandoff has completed.
As I say, nmon/top show minimal network & disk activity, and just one
of the four cores flatlining during this time. The machine *should* be
more responsive.
Actually: http://pastebin.com/AeM2VgL3
All the machines referenced in there are ones that are in the cluster now.
> 4. You took a machine offline without decommissioning it from the cluster.
> Now the machine is gone, but the other nodes (in Gossip logs) report that
> they are still looking for it. How do you stop nodes from looking for a
> removed node?
I was attempting to drain the thing first, but that was stalling, so I stopped
Cassandra then stopped the box. The storage and config were on EBS
(persistent disk) so they came back - it's just that the IP address of the
machine changed. I typically use my own assigned hostnames (cass-01,
cass-02, etc, say) but for proper resolution I use the EC2 'internal
hostnames',
which were updated to all four Cassandra boxes, the other three instances
of Cassandra were stopped, and then all four brought back up.
You say you have similar EC2-related thoughts .. have you done much on
the EC2 hardware so far? Are you seeing the same kind of thing?
cheers,
Jedd.
Re: Dazed and confused with Cassandra on EC2 ...
Posted by Dave Viner <da...@pobox.com>.
Hi Jedd,
I'm using Cassandra on EC2 as well - so I'm quite interested.
Just to clarify your post - it sounds like you have 4 questions/issue:
1. Writes have slowed down significantly. What's the logical explanation?
And what is the logical solution/options to solve it?
2. You grew from 2 nodes to 4, but the original 2 nodes have 200GB and the 2
new ones have 40 GB. What's the recommended practice for rebalancing (i.e.,
when should you do it), what's the actual procedure, and what's the expected
impact of it?
3. Cassandra nodes "disappear". (I'm not quite clear what this means.)
4. You took a machine offline without decommissioning it from the cluster.
Now the machine is gone, but the other nodes (in Gossip logs) report that
they are still looking for it. How do you stop nodes from looking for a
removed node?
I'm not trying to put words in your mouth - but I want to make sure that I
understand what you're asking about (because I have similar ec2-related
thoughts). Let me know if this is an accurate summary.
Dave Viner
On Fri, Sep 17, 2010 at 7:41 AM, Jedd Rashbrooke <
jedd.rashbrooke@imagini.net> wrote:
> Howdi,
>
> I've just landed in an experiment to get Cassandra going, and
> fed by PHP via Thrift via Hadoop, all running on EC2. I've been
> lurking a bit on the list for a couple of weeks, mostly reading any
> threads with the word 'performance' in them. Few people have
> anything polite to say about EC2, but I want to just throw out
> some observations and get some feedback on whether what
> I'm seeing is even approaching any kind of normal.
>
> My background is mostly *nix and networking, with half-way
> decent understanding of DB's -- but Cassandra, Hadoop, Thrift
> and EC2 are all fairly new to me.
>
> We're using a four-node decently-specced (m2.2xlarge, if you're
> EC2-aware) cluster - 32GB, 4-core, if you're not :) I'm using
> Ubuntu with the Deb packages for Cassandra and Hadoop, and
> some fairly conservative tweaks to things like JVM memory
> (bumping them up to 4GB, then 16GB).
>
> One of our insert jobs - a mapper only process - was running
> pretty fast a few days ago. Somewhere around a million lines
> of input, split into a dozen files, inserting via a Hadoop job in
> about a half hour. Happy times. This was when the cluster
> was modestly sized - 20-50GB. It's now about 200GB, and
> performance has dropped by an order of magnitude - perhaps
> 5-6 hours to do the same amount of work, using the same
> codebase and the same input data.
>
> I've read that reads slow down as the DB grows, but had an
> expectation that writes would be consistently snappy. How
> surprising is this performance drop given the DB growth?
>
> My 4-node cluster started off as a 2-node - and now nodetool
> ring suggests the two original nodes are 200GB each, and
> the newer two are 40GB. Is this normal? Would a rebalance
> likely improve performance substantially? My feeling is that
> it would be expensive to perform.
>
> EC2 seems to get a bad rap, and we're feeling quite a bit of
> pain, which is sad given the (on paper) spec of the machines,
> and the cost - over US$3k/month for the cluster. I've split
> Cassandra commitlog, Cassandra data, hadoop(hdfs) and
> tmp onto separate 'spindles' - observations so far suggest
> late '90's disk IO speed (15MB max sustained writes, one
> machine, one disk to another), and consistently inconsistent
> performance (identical machine next to it running the same
> task at the same time was getting 28MB) over several hours.
>
> Cassandra nodes seems to disappear too easily - even
> with just one core (out of four) maxed out with a jsvc task,
> minimal disk or network activity, the machine feels very
> sluggish. Tailing the cassandra logs hints that it's doing
> hinted handoffs and occasionally compaction tasks. I've
> never seen this kind of behaviour - and suspect this is
> more a feature of EC2.
>
> Gossip now seems to be pining the loss of an older machine
> (that I stupidly took offline briefly - EC2 gave it a new IP address
> when it came back). There's nothing in the storage-conf to
> refer to the old address, all 4 Cassandra daemons have been
> re-started several times since, but gossip occasionally (a day
> later) says that it is looking for it - and more worrying that
> it is 'now part of the cluster'. I'm unsure if this is just an
> irritation or part of the underlying problem.
>
> What I'm going to do next is to try importing some data into
> a local machine - it's just time-consuming to pull in our S3
> data - and see if I can fake up to around the same capacity
> and watch for performance degradation.
>
> I'm also toying with the idea of going from 4 to 8 nodes,
> but I'm clueless on whether / how much this would help.
>
> As I say, though, I'm keen on anyone else's observations on
> my observations - I'm painfully aware that I'm juggling a lot
> of unknown factors at the moment.
>
> cheers,
> Jedd.
>