You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "paul cannon (Created) (JIRA)" <ji...@apache.org> on 2012/02/13 20:30:59 UTC
[jira] [Created] (CASSANDRA-3903) Intermittent unexpected errors:
possibly race condition around CQL parser?
Intermittent unexpected errors: possibly race condition around CQL parser?
--------------------------------------------------------------------------
Key: CASSANDRA-3903
URL: https://issues.apache.org/jira/browse/CASSANDRA-3903
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: Mac OS X 10.7 with Sun/Oracle Java 1.6.0_29
Debian GNU/Linux 6.0.3 (squeeze) with Sun/Oracle Java 1.6.0_26
several recent commits on cassandra-1.1 branch. at least:
0183dc0b36e684082832de43a21b3dc0a9716d48, 3eefbac133c838db46faa6a91ba1f114192557ae, 9a842c7b317e6f1e6e156ccb531e34bb769c979f
Running cassandra under ccm with one node
Reporter: paul cannon
When running multiple simultaneous instances of the test_cql.py piece of the python-cql test suite, I can reliably reproduce intermittent and unpredictable errors in the tests.
The failures often occur at the point of keyspace creation during test setup, with a CQL statement of the form:
{code}
CREATE KEYSPACE 'asnvzpot' WITH strategy_class = SimpleStrategy
AND strategy_options:replication_factor = 1
{code}
An InvalidRequestException is returned to the cql driver, which re-raises it as a cql.ProgrammingError. The message:
{code}
ProgrammingError: Bad Request: line 2:24 no viable alternative at input 'asnvzpot'
{code}
In a few cases, Cassandra threw an ArrayIndexOutOfBoundsException and this traceback, closing the thrift connection:
{code}
ERROR [Thrift:244] 2012-02-10 15:51:46,815 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
java.lang.ArrayIndexOutOfBoundsException: 7
at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
at org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:744)
at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:898)
at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1245)
at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3458)
at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3446)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
{code}
Sometimes I see an ArrayOutOfBoundsError with no traceback:
{code}
ERROR [Thrift:858] 2012-02-13 12:04:01,537 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
java.lang.ArrayIndexOutOfBoundsException
{code}
Sometimes I get this:
{code}
ERROR [MigrationStage:1] 2012-02-13 12:04:46,077 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MigrationStage:1,5,main]
java.lang.IllegalArgumentException: value already present: 1558
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
at org.apache.cassandra.config.Schema.load(Schema.java:392)
at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:284)
at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:209)
at org.apache.cassandra.db.migration.AddColumnFamily.applyImpl(AddColumnFamily.java:49)
at org.apache.cassandra.db.migration.Migration.apply(Migration.java:66)
at org.apache.cassandra.cql.QueryProcessor$1.call(QueryProcessor.java:334)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
Again, around 99% of the instances of this {{CREATE KEYSPACE}} statement work fine, so it's a little hard to git bisect out, but I guess I'll see what I can do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3903) Intermittent unexpected errors:
possibly race condition around CQL parser?
Posted by "paul cannon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
paul cannon updated CASSANDRA-3903:
-----------------------------------
Reviewer: thepaul
Assignee: Sylvain Lebresne
I can't reproduce any of the failure modes above with these patches applied-- not even the "no viable alternative" one, so they must help.
In reviewing the patches, though, I don't think I understand how 0001-Fix-CFS.all-thread-safety.patch helps anything. Isn't that array wholly made and manipulated on the stack?
> Intermittent unexpected errors: possibly race condition around CQL parser?
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-3903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3903
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: Mac OS X 10.7 with Sun/Oracle Java 1.6.0_29
> Debian GNU/Linux 6.0.3 (squeeze) with Sun/Oracle Java 1.6.0_26
> several recent commits on cassandra-1.1 branch. at least:
> 0183dc0b36e684082832de43a21b3dc0a9716d48, 3eefbac133c838db46faa6a91ba1f114192557ae, 9a842c7b317e6f1e6e156ccb531e34bb769c979f
> Running cassandra under ccm with one node
> Reporter: paul cannon
> Assignee: Sylvain Lebresne
> Attachments: 0001-Fix-CFS.all-thread-safety.patch, 0002-Fix-fixCFMaxId.patch
>
>
> When running multiple simultaneous instances of the test_cql.py piece of the python-cql test suite, I can reliably reproduce intermittent and unpredictable errors in the tests.
> The failures often occur at the point of keyspace creation during test setup, with a CQL statement of the form:
> {code}
> CREATE KEYSPACE 'asnvzpot' WITH strategy_class = SimpleStrategy
> AND strategy_options:replication_factor = 1
>
> {code}
> An InvalidRequestException is returned to the cql driver, which re-raises it as a cql.ProgrammingError. The message:
> {code}
> ProgrammingError: Bad Request: line 2:24 no viable alternative at input 'asnvzpot'
> {code}
> In a few cases, Cassandra threw an ArrayIndexOutOfBoundsException and this traceback, closing the thrift connection:
> {code}
> ERROR [Thrift:244] 2012-02-10 15:51:46,815 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException: 7
> at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
> at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
> at org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:744)
> at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:898)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1245)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3458)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3446)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> {code}
> Sometimes I see an ArrayOutOfBoundsError with no traceback:
> {code}
> ERROR [Thrift:858] 2012-02-13 12:04:01,537 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException
> {code}
> Sometimes I get this:
> {code}
> ERROR [MigrationStage:1] 2012-02-13 12:04:46,077 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1558
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
> at org.apache.cassandra.config.Schema.load(Schema.java:392)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:284)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:209)
> at org.apache.cassandra.db.migration.AddColumnFamily.applyImpl(AddColumnFamily.java:49)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:66)
> at org.apache.cassandra.cql.QueryProcessor$1.call(QueryProcessor.java:334)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Again, around 99% of the instances of this {{CREATE KEYSPACE}} statement work fine, so it's a little hard to git bisect out, but I guess I'll see what I can do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3903) Intermittent unexpected errors:
possibly race condition around CQL parser?
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209195#comment-13209195 ]
Sylvain Lebresne commented on CASSANDRA-3903:
---------------------------------------------
bq. Isn't that array wholly made and manipulated on the stack?
It is, the "thread safety" I'm talking about about is that a new keyspace can be added while CFS.all() is running. If so, there can be say 6 keyspace when the array is created, but on the next line, when Table.all() is called, it could return 7 entries.
> Intermittent unexpected errors: possibly race condition around CQL parser?
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-3903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3903
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: Mac OS X 10.7 with Sun/Oracle Java 1.6.0_29
> Debian GNU/Linux 6.0.3 (squeeze) with Sun/Oracle Java 1.6.0_26
> several recent commits on cassandra-1.1 branch. at least:
> 0183dc0b36e684082832de43a21b3dc0a9716d48, 3eefbac133c838db46faa6a91ba1f114192557ae, 9a842c7b317e6f1e6e156ccb531e34bb769c979f
> Running cassandra under ccm with one node
> Reporter: paul cannon
> Assignee: Sylvain Lebresne
> Attachments: 0001-Fix-CFS.all-thread-safety.patch, 0002-Fix-fixCFMaxId.patch
>
>
> When running multiple simultaneous instances of the test_cql.py piece of the python-cql test suite, I can reliably reproduce intermittent and unpredictable errors in the tests.
> The failures often occur at the point of keyspace creation during test setup, with a CQL statement of the form:
> {code}
> CREATE KEYSPACE 'asnvzpot' WITH strategy_class = SimpleStrategy
> AND strategy_options:replication_factor = 1
>
> {code}
> An InvalidRequestException is returned to the cql driver, which re-raises it as a cql.ProgrammingError. The message:
> {code}
> ProgrammingError: Bad Request: line 2:24 no viable alternative at input 'asnvzpot'
> {code}
> In a few cases, Cassandra threw an ArrayIndexOutOfBoundsException and this traceback, closing the thrift connection:
> {code}
> ERROR [Thrift:244] 2012-02-10 15:51:46,815 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException: 7
> at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
> at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
> at org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:744)
> at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:898)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1245)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3458)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3446)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> {code}
> Sometimes I see an ArrayOutOfBoundsError with no traceback:
> {code}
> ERROR [Thrift:858] 2012-02-13 12:04:01,537 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException
> {code}
> Sometimes I get this:
> {code}
> ERROR [MigrationStage:1] 2012-02-13 12:04:46,077 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1558
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
> at org.apache.cassandra.config.Schema.load(Schema.java:392)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:284)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:209)
> at org.apache.cassandra.db.migration.AddColumnFamily.applyImpl(AddColumnFamily.java:49)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:66)
> at org.apache.cassandra.cql.QueryProcessor$1.call(QueryProcessor.java:334)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Again, around 99% of the instances of this {{CREATE KEYSPACE}} statement work fine, so it's a little hard to git bisect out, but I guess I'll see what I can do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3903) Intermittent unexpected errors:
possibly race condition around CQL parser?
Posted by "paul cannon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209585#comment-13209585 ]
paul cannon commented on CASSANDRA-3903:
----------------------------------------
Ohhhh, I see. Derp. Thanks for the explanation.
I +1 these changes.
> Intermittent unexpected errors: possibly race condition around CQL parser?
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-3903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3903
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: Mac OS X 10.7 with Sun/Oracle Java 1.6.0_29
> Debian GNU/Linux 6.0.3 (squeeze) with Sun/Oracle Java 1.6.0_26
> several recent commits on cassandra-1.1 branch. at least:
> 0183dc0b36e684082832de43a21b3dc0a9716d48, 3eefbac133c838db46faa6a91ba1f114192557ae, 9a842c7b317e6f1e6e156ccb531e34bb769c979f
> Running cassandra under ccm with one node
> Reporter: paul cannon
> Assignee: Sylvain Lebresne
> Attachments: 0001-Fix-CFS.all-thread-safety.patch, 0002-Fix-fixCFMaxId.patch
>
>
> When running multiple simultaneous instances of the test_cql.py piece of the python-cql test suite, I can reliably reproduce intermittent and unpredictable errors in the tests.
> The failures often occur at the point of keyspace creation during test setup, with a CQL statement of the form:
> {code}
> CREATE KEYSPACE 'asnvzpot' WITH strategy_class = SimpleStrategy
> AND strategy_options:replication_factor = 1
>
> {code}
> An InvalidRequestException is returned to the cql driver, which re-raises it as a cql.ProgrammingError. The message:
> {code}
> ProgrammingError: Bad Request: line 2:24 no viable alternative at input 'asnvzpot'
> {code}
> In a few cases, Cassandra threw an ArrayIndexOutOfBoundsException and this traceback, closing the thrift connection:
> {code}
> ERROR [Thrift:244] 2012-02-10 15:51:46,815 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException: 7
> at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
> at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
> at org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:744)
> at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:898)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1245)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3458)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3446)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> {code}
> Sometimes I see an ArrayOutOfBoundsError with no traceback:
> {code}
> ERROR [Thrift:858] 2012-02-13 12:04:01,537 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException
> {code}
> Sometimes I get this:
> {code}
> ERROR [MigrationStage:1] 2012-02-13 12:04:46,077 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1558
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
> at org.apache.cassandra.config.Schema.load(Schema.java:392)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:284)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:209)
> at org.apache.cassandra.db.migration.AddColumnFamily.applyImpl(AddColumnFamily.java:49)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:66)
> at org.apache.cassandra.cql.QueryProcessor$1.call(QueryProcessor.java:334)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Again, around 99% of the instances of this {{CREATE KEYSPACE}} statement work fine, so it's a little hard to git bisect out, but I guess I'll see what I can do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3903) Intermittent unexpected errors:
possibly race condition around CQL parser?
Posted by "Sylvain Lebresne (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne resolved CASSANDRA-3903.
-----------------------------------------
Resolution: Fixed
Fix Version/s: 1.1.0
Committed, thanks
> Intermittent unexpected errors: possibly race condition around CQL parser?
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-3903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3903
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: Mac OS X 10.7 with Sun/Oracle Java 1.6.0_29
> Debian GNU/Linux 6.0.3 (squeeze) with Sun/Oracle Java 1.6.0_26
> several recent commits on cassandra-1.1 branch. at least:
> 0183dc0b36e684082832de43a21b3dc0a9716d48, 3eefbac133c838db46faa6a91ba1f114192557ae, 9a842c7b317e6f1e6e156ccb531e34bb769c979f
> Running cassandra under ccm with one node
> Reporter: paul cannon
> Assignee: Sylvain Lebresne
> Fix For: 1.1.0
>
> Attachments: 0001-Fix-CFS.all-thread-safety.patch, 0002-Fix-fixCFMaxId.patch
>
>
> When running multiple simultaneous instances of the test_cql.py piece of the python-cql test suite, I can reliably reproduce intermittent and unpredictable errors in the tests.
> The failures often occur at the point of keyspace creation during test setup, with a CQL statement of the form:
> {code}
> CREATE KEYSPACE 'asnvzpot' WITH strategy_class = SimpleStrategy
> AND strategy_options:replication_factor = 1
>
> {code}
> An InvalidRequestException is returned to the cql driver, which re-raises it as a cql.ProgrammingError. The message:
> {code}
> ProgrammingError: Bad Request: line 2:24 no viable alternative at input 'asnvzpot'
> {code}
> In a few cases, Cassandra threw an ArrayIndexOutOfBoundsException and this traceback, closing the thrift connection:
> {code}
> ERROR [Thrift:244] 2012-02-10 15:51:46,815 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException: 7
> at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
> at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
> at org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:744)
> at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:898)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1245)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3458)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3446)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> {code}
> Sometimes I see an ArrayOutOfBoundsError with no traceback:
> {code}
> ERROR [Thrift:858] 2012-02-13 12:04:01,537 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException
> {code}
> Sometimes I get this:
> {code}
> ERROR [MigrationStage:1] 2012-02-13 12:04:46,077 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1558
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
> at org.apache.cassandra.config.Schema.load(Schema.java:392)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:284)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:209)
> at org.apache.cassandra.db.migration.AddColumnFamily.applyImpl(AddColumnFamily.java:49)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:66)
> at org.apache.cassandra.cql.QueryProcessor$1.call(QueryProcessor.java:334)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Again, around 99% of the instances of this {{CREATE KEYSPACE}} statement work fine, so it's a little hard to git bisect out, but I guess I'll see what I can do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3903) Intermittent unexpected errors:
possibly race condition around CQL parser?
Posted by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3903:
----------------------------------------
Attachment: 0002-Fix-fixCFMaxId.patch
0001-Fix-CFS.all-thread-safety.patch
For
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 7
at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
{noformat}
that's because CFS.all() is not threadSafe as a new keyspace can be added between the allocation of the array and the addition of the column family stores. Attaching patch that use an ArrayList instead of a plain array so that it can grow when that happens (It still use the same "estimate" for the initial size of the ArrayList as 99% of the time this will be the right size, but the point is that it doesn't crash if there is a concurrent modification).
For
{noformat}
java.lang.IllegalArgumentException: value already present: 1558
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
at org.apache.cassandra.config.Schema.load(Schema.java:392)
{noformat}
I believe that's because Schema.fixCFMaxId() may reset cfIdGen to a smaller value since it doesn't check the current value of cfIdGen. Patch attached for that too.
Not sure what's wrong with the ArrayOutOfBoundsError without stacktrace though.
I'm also not sure at all that this will fix the 'no viable alternative at input' error, as I don't think any of those error should trigger that (if only because both happens after the parsing). Not sure how we could have a race in the parser actually, since in theory the parsing of each request should be fully mono-threaded.
> Intermittent unexpected errors: possibly race condition around CQL parser?
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-3903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3903
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: Mac OS X 10.7 with Sun/Oracle Java 1.6.0_29
> Debian GNU/Linux 6.0.3 (squeeze) with Sun/Oracle Java 1.6.0_26
> several recent commits on cassandra-1.1 branch. at least:
> 0183dc0b36e684082832de43a21b3dc0a9716d48, 3eefbac133c838db46faa6a91ba1f114192557ae, 9a842c7b317e6f1e6e156ccb531e34bb769c979f
> Running cassandra under ccm with one node
> Reporter: paul cannon
> Attachments: 0001-Fix-CFS.all-thread-safety.patch, 0002-Fix-fixCFMaxId.patch
>
>
> When running multiple simultaneous instances of the test_cql.py piece of the python-cql test suite, I can reliably reproduce intermittent and unpredictable errors in the tests.
> The failures often occur at the point of keyspace creation during test setup, with a CQL statement of the form:
> {code}
> CREATE KEYSPACE 'asnvzpot' WITH strategy_class = SimpleStrategy
> AND strategy_options:replication_factor = 1
>
> {code}
> An InvalidRequestException is returned to the cql driver, which re-raises it as a cql.ProgrammingError. The message:
> {code}
> ProgrammingError: Bad Request: line 2:24 no viable alternative at input 'asnvzpot'
> {code}
> In a few cases, Cassandra threw an ArrayIndexOutOfBoundsException and this traceback, closing the thrift connection:
> {code}
> ERROR [Thrift:244] 2012-02-10 15:51:46,815 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException: 7
> at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
> at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
> at org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:744)
> at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:898)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1245)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3458)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3446)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> {code}
> Sometimes I see an ArrayOutOfBoundsError with no traceback:
> {code}
> ERROR [Thrift:858] 2012-02-13 12:04:01,537 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException
> {code}
> Sometimes I get this:
> {code}
> ERROR [MigrationStage:1] 2012-02-13 12:04:46,077 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1558
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
> at org.apache.cassandra.config.Schema.load(Schema.java:392)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:284)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:209)
> at org.apache.cassandra.db.migration.AddColumnFamily.applyImpl(AddColumnFamily.java:49)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:66)
> at org.apache.cassandra.cql.QueryProcessor$1.call(QueryProcessor.java:334)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Again, around 99% of the instances of this {{CREATE KEYSPACE}} statement work fine, so it's a little hard to git bisect out, but I guess I'll see what I can do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3903) Intermittent unexpected errors:
possibly race condition around CQL parser?
Posted by "paul cannon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207115#comment-13207115 ]
paul cannon commented on CASSANDRA-3903:
----------------------------------------
I should mention that I adjusted the python-cql tests to be able to run cleanly in parallel, in the parallel-tests branch.
> Intermittent unexpected errors: possibly race condition around CQL parser?
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-3903
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3903
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: Mac OS X 10.7 with Sun/Oracle Java 1.6.0_29
> Debian GNU/Linux 6.0.3 (squeeze) with Sun/Oracle Java 1.6.0_26
> several recent commits on cassandra-1.1 branch. at least:
> 0183dc0b36e684082832de43a21b3dc0a9716d48, 3eefbac133c838db46faa6a91ba1f114192557ae, 9a842c7b317e6f1e6e156ccb531e34bb769c979f
> Running cassandra under ccm with one node
> Reporter: paul cannon
>
> When running multiple simultaneous instances of the test_cql.py piece of the python-cql test suite, I can reliably reproduce intermittent and unpredictable errors in the tests.
> The failures often occur at the point of keyspace creation during test setup, with a CQL statement of the form:
> {code}
> CREATE KEYSPACE 'asnvzpot' WITH strategy_class = SimpleStrategy
> AND strategy_options:replication_factor = 1
>
> {code}
> An InvalidRequestException is returned to the cql driver, which re-raises it as a cql.ProgrammingError. The message:
> {code}
> ProgrammingError: Bad Request: line 2:24 no viable alternative at input 'asnvzpot'
> {code}
> In a few cases, Cassandra threw an ArrayIndexOutOfBoundsException and this traceback, closing the thrift connection:
> {code}
> ERROR [Thrift:244] 2012-02-10 15:51:46,815 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException: 7
> at org.apache.cassandra.db.ColumnFamilyStore.all(ColumnFamilyStore.java:1520)
> at org.apache.cassandra.thrift.ThriftValidation.validateCfDef(ThriftValidation.java:634)
> at org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:744)
> at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:898)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1245)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3458)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3446)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> {code}
> Sometimes I see an ArrayOutOfBoundsError with no traceback:
> {code}
> ERROR [Thrift:858] 2012-02-13 12:04:01,537 CustomTThreadPoolServer.java (line 205) Error occurred during processing of message.
> java.lang.ArrayIndexOutOfBoundsException
> {code}
> Sometimes I get this:
> {code}
> ERROR [MigrationStage:1] 2012-02-13 12:04:46,077 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1558
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
> at org.apache.cassandra.config.Schema.load(Schema.java:392)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:284)
> at org.apache.cassandra.db.migration.MigrationHelper.addColumnFamily(MigrationHelper.java:209)
> at org.apache.cassandra.db.migration.AddColumnFamily.applyImpl(AddColumnFamily.java:49)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:66)
> at org.apache.cassandra.cql.QueryProcessor$1.call(QueryProcessor.java:334)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> Again, around 99% of the instances of this {{CREATE KEYSPACE}} statement work fine, so it's a little hard to git bisect out, but I guess I'll see what I can do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira