You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Sachin PK <sa...@gmail.com> on 2015/05/28 15:15:44 UTC
cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting
for replica nodes' responses]
Hi I'm running Cassandra 2.1.5 ,(single datacenter ,4 node,16GB vps each
node ),I have given my configuration below, I'm using python driver on my
clients ,when i tried to insert 1049067 items I got an error.
cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for
replica nodes' responses] message="Operation timed out - received only 0
responses." info={'received_responses': 0, 'required_responses': 1,
'consistency': 'ONE'}
also I'm getting an error when I check the count of CF from cqlsh
OperationTimedOut: errors={}, last_host=127.0.0.1
I've installed Datastax Ops center for monitoring nodes , it shows about
1000/sec write request even the cluster is idle .
Is there any problem with my cassandra configuration
cluster_name: 'Test Cluster'
num_tokens: 256
hinted_handoff_enabled: true
.
max_hint_window_in_ms: 10800000 # 3 hours
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
disk_failure_policy: stop
commit_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /var/lib/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "10.xx.xx.xx,10.xx.xx.xxx"
concurrent_reads: 45
concurrent_writes: 64
concurrent_counter_writes: 32
memtable_allocation_type: heap_buffers
memtable_flush_writers: 6
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 10.xx.x.xxx
start_native_transport: true
native_transport_port: 9042
rpc_address: 0.0.0.0
rpc_port: 9160
broadcast_rpc_address: 10.xx.x.xxx
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
column_index_size_in_kb: 64
batch_size_warn_threshold_in_kb: 5
compaction_throughput_mb_per_sec: 16
sstable_preemptive_open_interval_in_mb: 50
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
counter_write_request_timeout_in_ms: 5000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
keystore: conf/.keystore
keystore_password: cassandra
truststore: conf/.truststore
truststore_password: cassandra
# enable or disable client/server encryption.
client_encryption_options:
enabled: false
keystore: conf/.keystore
keystore_password: cassandra
internode_compression: all
inter_dc_tcp_nodelay: false
Re: cassandra.WriteTimeout: code=1100 [Coordinator node timed out
waiting for replica nodes' responses]
Posted by Jean Tremblay <je...@zen-innovations.com>.
I have experienced similar results: OperationTimedOut after inserting many millions of records on a 5 nodes cluster, using Cassandra 2.1.5.
I rolled back to 2.1.4 using identically the same configuration as with 2.1.5 and these timeout went away… This is not the solution to your problem but just to say that for me 2.1.5 seems to saturate when bulk inserting many million records.
> On 28 May 2015, at 15:15 , Sachin PK <sa...@gmail.com> wrote:
>
> Hi I'm running Cassandra 2.1.5 ,(single datacenter ,4 node,16GB vps each node ),I have given my configuration below, I'm using python driver on my clients ,when i tried to insert 1049067 items I got an error.
>
> cassandra.WriteTimeout: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
>
> also I'm getting an error when I check the count of CF from cqlsh
>
> OperationTimedOut: errors={}, last_host=127.0.0.1
>
> I've installed Datastax Ops center for monitoring nodes , it shows about 1000/sec write request even the cluster is idle .
>
> Is there any problem with my cassandra configuration
>
>
> cluster_name: 'Test Cluster'
>
> num_tokens: 256
>
>
> hinted_handoff_enabled: true
> .
> max_hint_window_in_ms: 10800000 # 3 hours
>
> hinted_handoff_throttle_in_kb: 1024
>
> max_hints_delivery_threads: 2
>
> batchlog_replay_throttle_in_kb: 1024
>
> authenticator: AllowAllAuthenticator
>
> authorizer: AllowAllAuthorizer
>
> permissions_validity_in_ms: 2000
>
> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>
> data_file_directories:
> - /var/lib/cassandra/data
>
> commitlog_directory: /var/lib/cassandra/commitlog
>
> disk_failure_policy: stop
>
> commit_failure_policy: stop
>
> key_cache_size_in_mb:
>
> key_cache_save_period: 14400
>
> row_cache_size_in_mb: 0
>
>
> row_cache_save_period: 0
>
> counter_cache_size_in_mb:
>
>
> counter_cache_save_period: 7200
>
>
> saved_caches_directory: /var/lib/cassandra/saved_caches
>
> commitlog_sync: periodic
> commitlog_sync_period_in_ms: 10000
>
> commitlog_segment_size_in_mb: 32
>
> seed_provider:
>
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
>
> - seeds: "10.xx.xx.xx,10.xx.xx.xxx"
>
> concurrent_reads: 45
> concurrent_writes: 64
> concurrent_counter_writes: 32
>
> memtable_allocation_type: heap_buffers
>
>
> memtable_flush_writers: 6
>
> index_summary_capacity_in_mb:
>
> index_summary_resize_interval_in_minutes: 60
>
> trickle_fsync: false
> trickle_fsync_interval_in_kb: 10240
>
> storage_port: 7000
>
> ssl_storage_port: 7001
>
> listen_address: 10.xx.x.xxx
>
> start_native_transport: true
>
> native_transport_port: 9042
>
> rpc_address: 0.0.0.0
>
> rpc_port: 9160
>
> broadcast_rpc_address: 10.xx.x.xxx
>
> rpc_keepalive: true
>
> rpc_server_type: sync
>
> thrift_framed_transport_size_in_mb: 15
>
> incremental_backups: false
>
> snapshot_before_compaction: false
>
> auto_snapshot: true
>
> tombstone_warn_threshold: 1000
> tombstone_failure_threshold: 100000
>
> column_index_size_in_kb: 64
>
> batch_size_warn_threshold_in_kb: 5
>
> compaction_throughput_mb_per_sec: 16
>
> sstable_preemptive_open_interval_in_mb: 50
>
> read_request_timeout_in_ms: 5000
>
> range_request_timeout_in_ms: 10000
>
> write_request_timeout_in_ms: 2000
>
> counter_write_request_timeout_in_ms: 5000
>
> cas_contention_timeout_in_ms: 1000
>
> truncate_request_timeout_in_ms: 60000
>
> request_timeout_in_ms: 10000
>
> cross_node_timeout: false
>
> endpoint_snitch: SimpleSnitch
>
>
> dynamic_snitch_update_interval_in_ms: 100
>
> dynamic_snitch_reset_interval_in_ms: 600000
>
> dynamic_snitch_badness_threshold: 0.1
>
> request_scheduler: org.apache.cassandra.scheduler.NoScheduler
>
>
> server_encryption_options:
> internode_encryption: none
> keystore: conf/.keystore
> keystore_password: cassandra
> truststore: conf/.truststore
> truststore_password: cassandra
>
>
> # enable or disable client/server encryption.
> client_encryption_options:
> enabled: false
> keystore: conf/.keystore
> keystore_password: cassandra
>
> internode_compression: all
>
> inter_dc_tcp_nodelay: false
>
>
>
>