You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Marcelo Elias Del Valle <mv...@gmail.com> on 2013/02/01 21:27:13 UTC
Cassandra behavior on single node
Hello,
I am trying to figure out why the following behavior happened. Any
help would be highly appreciated.
This graph shows the server resources allocation of my single
cassandra machine (running at Amazon EC2):
http://mvalle.com/downloads/cassandra_host1.png
I ran a hadoop process that reads a CSV file and writtes data to
Cassandra. For about 1 h, the process ran fine, but taking about 100% of
CPU. After 1 h, my hadoop process started to have its connection attempts
refused by cassandra, as shown bellow.
Since them, it has been taking 100% of the machine IO. It has been 2 h
already since the IO is 100% on the machine running Cassandra.
I am running Cassandra under Amazon EBS, which is slow, but I didn't
think it would be that slow. Just wondering, is it normal for Cassandra to
use a high amount of CPU? I am guessing all the writes were going to the
memtables and when it was time to flush the server went down.
Makes sense? I am still learning Cassandra as it's the first time I
use it in production, so I am not sure if I am missing something really
basic here.
2013-02-01 16:44:43,741 ERROR
com.s1mbi0se.dmp.input.service.InputService (Thread-18):
EXCEPTION:PoolTimeoutException: [host=(10.84.65.108):9160,
latency=5005(5005), attempts=1] Timed out waiting for connection
com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException:
PoolTimeoutException: [host=nosql1.s1mbi0se.com.br(10.84.65.108):9160,
latency=5005(5005), attempts=1] Timed out waiting for connection
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:201)
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:158)
at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:60)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:50)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186)
at com.s1mbi0se.dmp.input.service.InputService.searchUserByKey(InputService.java:700)
...
at com.s1mbi0se.dmp.importer.map.ImporterMapper.map(ImporterMapper.java:20)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper$MapRunner.run(MultithreadedMapper.java:268)
2013-02-01 16:44:43,743 ERROR
com.s1mbi0se.dmp.input.service.InputService (Thread-15):
EXCEPTION:PoolTimeoutException:
Best regards,
--
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr
Re: Cassandra behavior on single node
Posted by Edward Capriolo <ed...@gmail.com>.
You are likely hitting the point where compaction is running all the time
and consuming all the weak cloud io. Ebs is not suggested for performance
you should use the ephermal drives.
On Friday, February 1, 2013, Marcelo Elias Del Valle wrote:
> Hello,
>
> I am trying to figure out why the following behavior happened. Any
> help would be highly appreciated.
> This graph shows the server resources allocation of my single
> cassandra machine (running at Amazon EC2):
> http://mvalle.com/downloads/cassandra_host1.png
> I ran a hadoop process that reads a CSV file and writtes data to
> Cassandra. For about 1 h, the process ran fine, but taking about 100% of
> CPU. After 1 h, my hadoop process started to have its connection attempts
> refused by cassandra, as shown bellow.
> Since them, it has been taking 100% of the machine IO. It has been 2
> h already since the IO is 100% on the machine running Cassandra.
> I am running Cassandra under Amazon EBS, which is slow, but I didn't
> think it would be that slow. Just wondering, is it normal for Cassandra to
> use a high amount of CPU? I am guessing all the writes were going to the
> memtables and when it was time to flush the server went down.
> Makes sense? I am still learning Cassandra as it's the first time I
> use it in production, so I am not sure if I am missing something really
> basic here.
>
> 2013-02-01 16:44:43,741 ERROR com.s1mbi0se.dmp.input.service.InputService (Thread-18): EXCEPTION:PoolTimeoutException: [host=(10.84.65.108):9160, latency=5005(5005), attempts=1] Timed out waiting for connection
> com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=nosql1.s1mbi0se.com.br(10.84.65.108):9160, latency=5005(5005), attempts=1] Timed out waiting for connection
> at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:201)
> at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:158)
> at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:60)
> at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:50)
> at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229)
> at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186)
> at com.s1mbi0se.dmp.input.service.InputService.searchUserByKey(InputService.java:700)
>
> ...
> at com.s1mbi0se.dmp.importer.map.ImporterMapper.map(ImporterMapper.java:20)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper$MapRunner.run(MultithreadedMapper.java:268)
> 2013-02-01 16:44:43,743 ERROR com.s1mbi0se.dmp.input.service.InputService (Thread-15): EXCEPTION:PoolTimeoutException:
>
>
> Best regards,
>
> --
> Marcelo Elias Del Valle
> http://mvalle.com - @mvallebr
>