You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Marcelo Elias Del Valle <mv...@gmail.com> on 2013/02/01 20:53:35 UTC
(Unknown)
Hello,
I am trying to figure out why the following behavior happened. Any
help would be highly appreciated.
This graph shows the server resources allocation of my single
cassandra machine (running at Amazon EC2):
http://mvalle.com/downloads/cassandra_host1.png
I ran a hadoop process that reads a CSV file and writtes data to
Cassandra. For about 1 h, the process ran fine, but taking about 100% of
CPU. After 1 h, my hadoop process started to have its connection attempts
refused by cassandra, as shown bellow.
Since them, it has been taking 100% of the machine IO. It has been 2 h
already since the IO is 100% on the machine running Cassandra.
I am running Cassandra under Amazon EBS, which is slow, but I didn't
think it would be that slow. Just wondering, is it normal for Cassandra to
use a high amount of CPU? I am guessing all the writes were going to the
memtables and when it was time to flush the server went down.
Makes sense? I am still learning Cassandra as it's the first time I
use it in production, so I am not sure if I am missing something really
basic here.
2013-02-01 16:44:43,741 ERROR
com.s1mbi0se.dmp.input.service.InputService (Thread-18):
EXCEPTION:PoolTimeoutException: [host=(10.84.65.108):9160,
latency=5005(5005), attempts=1] Timed out waiting for connection
com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException:
PoolTimeoutException: [host=nosql1.s1mbi0se.com.br(10.84.65.108):9160,
latency=5005(5005), attempts=1] Timed out waiting for connection
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:201)
at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:158)
at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:60)
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:50)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229)
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186)
at com.s1mbi0se.dmp.input.service.InputService.searchUserByKey(InputService.java:700)
...
at com.s1mbi0se.dmp.importer.map.ImporterMapper.map(ImporterMapper.java:20)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper$MapRunner.run(MultithreadedMapper.java:268)
2013-02-01 16:44:43,743 ERROR
com.s1mbi0se.dmp.input.service.InputService (Thread-15):
EXCEPTION:PoolTimeoutException:
Best regards,
--
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr
Re:
Posted by Víctor Hugo Oliveira Molinar <vh...@gmail.com>.
How do you establish the connection?
Are you closing and reopening it?
It's normal for cassandra slowing down after many insertions, but it would
only take more time to process your write, nothing more than that.
On Fri, Feb 1, 2013 at 5:53 PM, Marcelo Elias Del Valle
<mv...@gmail.com>wrote:
> Hello,
>
> I am trying to figure out why the following behavior happened. Any
> help would be highly appreciated.
> This graph shows the server resources allocation of my single
> cassandra machine (running at Amazon EC2):
> http://mvalle.com/downloads/cassandra_host1.png
> I ran a hadoop process that reads a CSV file and writtes data to
> Cassandra. For about 1 h, the process ran fine, but taking about 100% of
> CPU. After 1 h, my hadoop process started to have its connection attempts
> refused by cassandra, as shown bellow.
> Since them, it has been taking 100% of the machine IO. It has been 2
> h already since the IO is 100% on the machine running Cassandra.
> I am running Cassandra under Amazon EBS, which is slow, but I didn't
> think it would be that slow. Just wondering, is it normal for Cassandra to
> use a high amount of CPU? I am guessing all the writes were going to the
> memtables and when it was time to flush the server went down.
> Makes sense? I am still learning Cassandra as it's the first time I
> use it in production, so I am not sure if I am missing something really
> basic here.
>
>
> 2013-02-01 16:44:43,741 ERROR com.s1mbi0se.dmp.input.service.InputService (Thread-18): EXCEPTION:PoolTimeoutException: [host=(10.84.65.108):9160, latency=5005(5005), attempts=1] Timed out waiting for connection
> com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=nosql1.s1mbi0se.com.br(10.84.65.108):9160, latency=5005(5005), attempts=1] Timed out waiting for connection
> at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:201)
> at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:158)
> at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:60)
> at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:50)
> at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229)
> at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186)
> at com.s1mbi0se.dmp.input.service.InputService.searchUserByKey(InputService.java:700)
>
> ...
> at com.s1mbi0se.dmp.importer.map.ImporterMapper.map(ImporterMapper.java:20)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper$MapRunner.run(MultithreadedMapper.java:268)
> 2013-02-01 16:44:43,743 ERROR com.s1mbi0se.dmp.input.service.InputService (Thread-15): EXCEPTION:PoolTimeoutException:
>
>
> Best regards,
> --
> Marcelo Elias Del Valle
> http://mvalle.com - @mvallebr
>