You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org> on 2012/06/13 11:35:42 UTC
[jira] [Created] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Patrycjusz Matuszak created CASSANDRA-4337:
----------------------------------------------
Summary: Data insertion fails because of commitlog rename failure
Key: CASSANDRA-4337
URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.1
Environment: - Node 1:
Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
System: Windows Server 2008 R2 x64
Java version: 7 update 4 x64
- Node 2:
Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
System: Windows Server 2008 R2 x64
Java version: 7 update 4 x64
Reporter: Patrycjusz Matuszak
h3. Configuration
Cassandra server configuration:
{noformat}heap size: 4 GB
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
listen_address: xxx.xxx.xxx.10
rpc_address: 0.0.0.0
rpc_port: 9160
rpc_timeout_in_ms: 20000
endpoint_snitch: PropertyFileSnitch{noformat}
cassandra-topology.properties
{noformat}xxx.xxx.xxx.10=datacenter1:rack1
xxx.xxx.xxx.11=datacenter1:rack1
default=datacenter1:rack1{noformat}
Ring configuration:
{noformat}Address DC Rack Status State Load Effective-Ownership Token
85070591730234615865843651857942052864
xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
h3.Problem
I have ctreated keyspace and column family using CLI commands:
{noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
use testks;
create column family testcf;{noformat}
Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
Then, few seconds later Cassandra server on node 2 throws the same exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
{noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10>
Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
at java.lang.Thread.run(Unknown Source)
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrycjusz Matuszak updated CASSANDRA-4337:
-------------------------------------------
Attachment: system-node2-stress-test.log
system-node1-stress-test.log
I've attached logs from test with Stress tool on Java 7 update 5.
I had to run Stress tool twice to reproduce this bug. At first run no exception was thrown.
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Labels: commitlog
> Attachments: system-node1-stress-test.log, system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295314#comment-13295314 ]
Jonathan Ellis commented on CASSANDRA-4337:
-------------------------------------------
Can you reproduce w/ java7 and the Stress tool that ships with Cassandra?
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Labels: commitlog
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4337:
--------------------------------------
Attachment: 4337.txt
It may be the CommitLogSegment mmap'd buffer preventing rename. Can you test the attached patch?
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.3
>
> Attachments: 4337.txt, system-node1-stress-test.log, system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4337:
--------------------------------------
Fix Version/s: 1.1.2
Assignee: Jonathan Ellis
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.2
>
> Attachments: system-node1-stress-test.log, system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrycjusz Matuszak updated CASSANDRA-4337:
-------------------------------------------
Environment:
- Node 1:
Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
System: Windows Server 2008 R2 x64
Java version: 7 update 4 x64
- Node 2:
Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
System: Windows Server 2008 R2 x64
Java version: 7 update 4 x64
was:
- Node 1:
Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
System: Windows Server 2008 R2 x64
Java version: 7 update 4 x64
- Node 2:
Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
System: Windows Server 2008 R2 x64
Java version: 7 update 4 x64
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Hiren Barot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiren Barot updated CASSANDRA-4337:
-----------------------------------
Attachment: Cassandra-Error.txt
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.4
>
> Attachments: 4337-poc.txt, Cassandra-Error.txt, system-node1.log, system-node1-stress-test.log, system-node2.log, system-node2-stress-test.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405929#comment-13405929 ]
Patrycjusz Matuszak commented on CASSANDRA-4337:
------------------------------------------------
I've tested with attached patch and couldn't reproduce this bug.
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.3
>
> Attachments: 4337-poc.txt, system-node1-stress-test.log, system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4337) Data insertion fails
because of commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294294#comment-13294294 ]
Patrycjusz Matuszak edited comment on CASSANDRA-4337 at 6/13/12 9:38 AM:
-------------------------------------------------------------------------
Attached Cassandra server logs
was (Author: patrycjusz):
Cassandra server log from node 1
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Hiren Barot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463110#comment-13463110 ]
Hiren Barot commented on CASSANDRA-4337:
----------------------------------------
I ran into this same problem and I have tried all the solutions mentioned above and I still cant fix the problem.
I am running Cassandra on my windows 7.
Cassandra : 1.1.5 with Java 6,
64 bit version on Windows 7.
ERROR 11:03:01,454 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from C:\DataStax Community\data\commitlog\CommitLog-83930807354964.log to 84059497979959 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocator.java:203)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Rename from C:\DataStax Community\data\commitlog\CommitLog-83930807354964.log to 84059497979959 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 4 more
Any help is greatly appreciated.
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.4
>
> Attachments: 4337-poc.txt, Cassandra-Error.txt, system-node1.log, system-node1-stress-test.log, system-node2.log, system-node2-stress-test.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294834#comment-13294834 ]
Patrycjusz Matuszak commented on CASSANDRA-4337:
------------------------------------------------
I don't think that it has anything to do with permissions. I'm running Cassandra as administrator.
Last night I found a probable cause of the problem. I switched to Java 6 update 33 x64 on both machines and was able to perform all planed tests without any errors.
It seems that this bug is related to Java version I used. Problem doesn't occur when using Java 6.
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Labels: commitlog
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrycjusz Matuszak updated CASSANDRA-4337:
-------------------------------------------
Attachment: system-node1.log
Cassandra server log from node 1
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4337:
--------------------------------------
Attachment: (was: 4337.txt)
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.3
>
> Attachments: system-node1-stress-test.log, system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294515#comment-13294515 ]
Jonathan Ellis commented on CASSANDRA-4337:
-------------------------------------------
Why would commitlog rename fail? Do you have some kind of permissions problem?
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Labels: commitlog
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrycjusz Matuszak updated CASSANDRA-4337:
-------------------------------------------
Attachment: system-node2.log
Cassandra server log from node 2
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrycjusz Matuszak updated CASSANDRA-4337:
-------------------------------------------
Comment: was deleted
(was: Cassandra server log from node 2)
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4337:
--------------------------------------
Attachment: 4337-poc.txt
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.3
>
> Attachments: 4337-poc.txt, system-node1-stress-test.log, system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Patrycjusz Matuszak (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrycjusz Matuszak updated CASSANDRA-4337:
-------------------------------------------
Description:
h3. Configuration
Cassandra server configuration:
{noformat}heap size: 4 GB
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
listen_address: xxx.xxx.xxx.10
rpc_address: 0.0.0.0
rpc_port: 9160
rpc_timeout_in_ms: 20000
endpoint_snitch: PropertyFileSnitch{noformat}
cassandra-topology.properties
{noformat}xxx.xxx.xxx.10=datacenter1:rack1
xxx.xxx.xxx.11=datacenter1:rack1
default=datacenter1:rack1{noformat}
Ring configuration:
{noformat}Address DC Rack Status State Load Effective-Ownership Token
85070591730234615865843651857942052864
xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
h3.Problem
I have ctreated keyspace and column family using CLI commands:
{noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
use testks;
create column family testcf;{noformat}
Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
Then, few seconds later Cassandra server on node 2 throws the same exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
{noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
at java.lang.Thread.run(Unknown Source)
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 8 more{noformat}
was:
h3. Configuration
Cassandra server configuration:
{noformat}heap size: 4 GB
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
listen_address: xxx.xxx.xxx.10
rpc_address: 0.0.0.0
rpc_port: 9160
rpc_timeout_in_ms: 20000
endpoint_snitch: PropertyFileSnitch{noformat}
cassandra-topology.properties
{noformat}xxx.xxx.xxx.10=datacenter1:rack1
xxx.xxx.xxx.11=datacenter1:rack1
default=datacenter1:rack1{noformat}
Ring configuration:
{noformat}Address DC Rack Status State Load Effective-Ownership Token
85070591730234615865843651857942052864
xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
h3.Problem
I have ctreated keyspace and column family using CLI commands:
{noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
use testks;
create column family testcf;{noformat}
Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
Then, few seconds later Cassandra server on node 2 throws the same exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
{noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10>
Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN %Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
at java.lang.Thread.run(Unknown Source)
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 8 more{noformat}
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4337) Data insertion fails because of
commitlog rename failure
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4337.
---------------------------------------
Resolution: Fixed
Reviewer: patrycjusz
I was going to try to make this super robust for JVMs that don't provide an munmap cleaner, but I decided that since (a) it's extremely unlikely that anyone runs such JVMs on windows, and (b) such JVMs work fine anyway on *nix since you can rename w/o unmapping first, and (c) this doesn't make things worse for such JVMs; it just fixes it for OpenJDK / Oracle JDK, I'll just commit this as is.
(In any case, the "right" solution for such a posited JVM is to wrap its munmap funcationality the way we did for Oracle's... such functionality has to exist for the GC to clean up direct buffers.)
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.4
>
> Attachments: 4337-poc.txt, system-node1-stress-test.log, system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load Effective-Ownership Token
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed
> at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Could not fullfill request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira