You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by ruslan usifov <ru...@gmail.com> on 2009/11/27 09:55:50 UTC
Casandra insert very slow if i use connection pool
Hello
I use follow python program to insert records into cassandra cluster (i my
test case this is two computers). And i have very very slow inserts (1.4 sec
per 100 records).
lazyboy python lib use round robin to chose which connection need use to
insert.
from lazyboy import *
from lazyboy.key import Key
import time;
# Define your cluster(s)
begin = 0;
btime = time.time();
connection.add_pool('Keyspace1', ['localhost:9090', '192.168.0.220:9160'])
for i in xrange(begin, begin + 500000):
if i!=begin and (i % 10000) == 0:
print time.time() - btime;
btime = time.time();
rc = record.Record({'value': i, 'text': "ruslan text"})
rc.key = Key("Keyspace1", "Standard1", str(i));
rc.save(ZERO);
print time.time() - btime;
If i remove from pool 1 connection, all fine. What i do wrong?
Here is my test one node configuration. Second gave identical except ip
addreses
<ClusterName>Test Cluster</ClusterName>
<Keyspaces>
<Keyspace Name="Keyspace1">
<ColumnFamily CompareWith="BytesType"
Name="Standard1"
FlushPeriodInMinutes="60"/>
<ColumnFamily CompareWith="UTF8Type" Name="Standard2"/>
<ColumnFamily CompareWith="TimeUUIDType" Name="StandardByUUID1"/>
<ColumnFamily ColumnType="Super"
CompareWith="UTF8Type"
CompareSubcolumnsWith="UTF8Type"
Name="Super1"/>
</Keyspace>
</Keyspaces>
<Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>
<InitialToken></InitialToken>
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
<ReplicationFactor>1</ReplicationFactor>
<CommitLogDirectory>/data/cassandra/commitlog</CommitLogDirectory>
<DataFileDirectories>
<DataFileDirectory>/data/cassandra/data</DataFileDirectory>
</DataFileDirectories>
<CalloutLocation>/data/cassandra/callouts</CalloutLocation>
<BootstrapFileDirectory>/data/cassandra/bootstrap</BootstrapFileDirectory>
<StagingFileDirectory>/data/cassandra/staging</StagingFileDirectory>
<Seeds>
<Seed>192.168.0.37</Seed>
</Seeds>
<RpcTimeoutInMillis>5000</RpcTimeoutInMillis>
<CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
<ListenAddress>192.168.0.220</ListenAddress>
<StoragePort>7000</StoragePort>
<ControlPort>7001</ControlPort>
<ThriftAddress>0.0.0.0</ThriftAddress>
<ThriftPort>9160</ThriftPort>
<ThriftFramedTransport>false</ThriftFramedTransport>
<SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
<FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
<FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
<ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
<MemtableSizeInMB>64</MemtableSizeInMB>
<MemtableObjectCountInMillions>0.1</MemtableObjectCountInMillions>
<ConcurrentReads>8</ConcurrentReads>
<ConcurrentWrites>32</ConcurrentWrites>
<CommitLogSync>periodic</CommitLogSync>
<CommitLogSyncPeriodInMS>1000</CommitLogSyncPeriodInMS>
<GCGraceSeconds>864000</GCGraceSeconds>
<FlushMinThreads>1</FlushMinThreads>
<FlushMaxThreads>1</FlushMaxThreads>
<BinaryMemtableSizeInMB>256</BinaryMemtableSizeInMB>
</Storage>
Re: Casandra insert very slow if i use connection pool
Posted by ruslan usifov <ru...@gmail.com>.
My cassandra thrift interface was configured on that port, on one of nodes
Re: Casandra insert very slow if i use connection pool
Posted by Jeff Hodges <je...@somethingsimilar.com>.
Hm, maybe I'm missing something about lazyboy, but it looks like the
port in "localhost:9090" is wrong.
--
Jeff
On Fri, Nov 27, 2009 at 12:55 AM, ruslan usifov <ru...@gmail.com> wrote:
> Hello
>
>
> I use follow python program to insert records into cassandra cluster (i my
> test case this is two computers). And i have very very slow inserts (1.4 sec
> per 100 records).
> lazyboy python lib use round robin to chose which connection need use to
> insert.
>
> from lazyboy import *
> from lazyboy.key import Key
>
> import time;
>
> # Define your cluster(s)
> begin = 0;
> btime = time.time();
> connection.add_pool('Keyspace1', ['localhost:9090', '192.168.0.220:9160'])
>
> for i in xrange(begin, begin + 500000):
> if i!=begin and (i % 10000) == 0:
> print time.time() - btime;
> btime = time.time();
>
> rc = record.Record({'value': i, 'text': "ruslan text"})
> rc.key = Key("Keyspace1", "Standard1", str(i));
> rc.save(ZERO);
>
> print time.time() - btime;
>
>
> If i remove from pool 1 connection, all fine. What i do wrong?
>
>
> Here is my test one node configuration. Second gave identical except ip
> addreses
>
> <ClusterName>Test Cluster</ClusterName>
>
> <Keyspaces>
> <Keyspace Name="Keyspace1">
> <ColumnFamily CompareWith="BytesType"
> Name="Standard1"
> FlushPeriodInMinutes="60"/>
> <ColumnFamily CompareWith="UTF8Type" Name="Standard2"/>
> <ColumnFamily CompareWith="TimeUUIDType" Name="StandardByUUID1"/>
> <ColumnFamily ColumnType="Super"
> CompareWith="UTF8Type"
> CompareSubcolumnsWith="UTF8Type"
> Name="Super1"/>
> </Keyspace>
> </Keyspaces>
>
> <Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>
>
> <InitialToken></InitialToken>
>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>
> <ReplicationFactor>1</ReplicationFactor>
>
> <CommitLogDirectory>/data/cassandra/commitlog</CommitLogDirectory>
> <DataFileDirectories>
> <DataFileDirectory>/data/cassandra/data</DataFileDirectory>
> </DataFileDirectories>
> <CalloutLocation>/data/cassandra/callouts</CalloutLocation>
> <BootstrapFileDirectory>/data/cassandra/bootstrap</BootstrapFileDirectory>
> <StagingFileDirectory>/data/cassandra/staging</StagingFileDirectory>
>
> <Seeds>
> <Seed>192.168.0.37</Seed>
> </Seeds>
>
>
> <RpcTimeoutInMillis>5000</RpcTimeoutInMillis>
> <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
>
>
> <ListenAddress>192.168.0.220</ListenAddress>
> <StoragePort>7000</StoragePort>
> <ControlPort>7001</ControlPort>
>
> <ThriftAddress>0.0.0.0</ThriftAddress>
> <ThriftPort>9160</ThriftPort>
> <ThriftFramedTransport>false</ThriftFramedTransport>
>
>
> <SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
>
> <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
> <FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
>
> <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
>
> <MemtableSizeInMB>64</MemtableSizeInMB>
> <MemtableObjectCountInMillions>0.1</MemtableObjectCountInMillions>
>
> <ConcurrentReads>8</ConcurrentReads>
> <ConcurrentWrites>32</ConcurrentWrites>
>
> <CommitLogSync>periodic</CommitLogSync>
> <CommitLogSyncPeriodInMS>1000</CommitLogSyncPeriodInMS>
> <GCGraceSeconds>864000</GCGraceSeconds>
>
> <FlushMinThreads>1</FlushMinThreads>
> <FlushMaxThreads>1</FlushMaxThreads>
>
> <BinaryMemtableSizeInMB>256</BinaryMemtableSizeInMB>
>
> </Storage>
>
>