You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Voytek Jarnot <vo...@gmail.com> on 2016/11/04 20:58:09 UTC
SASI index creation assertion error
Wondering if anyone has encountered the same...
Full story and stacktraces below, short version is that creating a SASI
index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
the index on an empty table doesn't fail; however, soon after I start
pumping data into the table similar problems occur.
I created CASSANDRA-12877 for this, but am beginning to suspect it might be
related to CASSANDRA-11990. The thing that's throwing me is that I can't
seem to duplicate this with a simple test table.
Background:
Ended up building/loading a 3.10-SNAPSHOT to try to get past
CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
2) created keyspace (SimpleStrategy, RF 1)
3) created table: (simplified version below, many more valX columns present)
CREATE TABLE test_table (
id1 text,
id2 text,
id3 date,
id4 timestamp,
id5 text,
val1 text,
val2 text,
val3 text,
task_id text,
val4 text,
val5 text,
PRIMARY KEY ((id1, id2), id3, id4, id5)
) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
4) created materialized view:
CREATE MATERIALIZED VIEW test_table_by_task_id AS
SELECT *
FROM test_table
WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND id4
IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5 ASC)
5) inserted 27 million "rows" (i.e., unique values for id5)
6) create index attempt
create custom index idx_test_table_val5 on test_table(val5) using
'org.apache.cassandra.index.sasi.SASIIndex';
7) no error in cqlsh, but system.log shows many of the following:
INFO [SASI-General:1] 2016-11-04 13:46:47,578
PerSSTableIndexWriter.java:277 - Flushed index segment
/mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-149-big-SI_idx_test_table_val5.db_0,
took 869 ms.
ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229 -
Exception in thread Thread[SASI-General:1,5,main]
java.lang.AssertionError: cannot have more than 8 overflow collisions per
leaf, but had: 12
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createOverflowEntry(AbstractTokenTreeBuilder.java:357)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.write(AbstractTokenTreeBuilder.java:90)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.flush(OnDiskIndexBuilder.java:446)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.add(OnDiskIndexBuilder.java:433)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.addTerm(OnDiskIndexBuilder.java:207)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:293)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:258)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:241)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
As well as some of these:
ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142 DataTracker.java:168 -
Can't open index file at
/mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-300-big-SI_idx_test_table_val5.db,
skipping.
java.lang.IllegalArgumentException: position: 3472329188772431788, limit:
8180
at
org.apache.cassandra.index.sasi.utils.MappedBuffer.position(MappedBuffer.java:106)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.conf.DataTracker.getIndexes(DataTracker.java:150)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSSTable(SASIIndexBuilder.java:156)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1583)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_101]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Re: SASI index creation assertion error
Posted by Voytek Jarnot <vo...@gmail.com>.
Yep, already done: https://issues.apache.org/jira/browse/CASSANDRA-12877
On Fri, Nov 4, 2016 at 11:15 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Can you file a Jira for this? Would be good to make sure 3.10 doesn't get
> released with this bug.
> On Fri, Nov 4, 2016 at 6:11 PM Voytek Jarnot <vo...@gmail.com>
> wrote:
>
>> Thought I'd follow-up to myself, in case anyone else comes across this
>> problem. I found a reasonably easy test case to reproduce the problem:
>>
>> This works in 3.9, but doesn't work in 3.10-snapshot:
>>
>> CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy',
>> 'replication_factor': '1'};
>> use vjtest ;
>> create table tester(id1 text, id2 text, id3 text, val1 text, primary
>> key((id1, id2), id3));
>> create custom index tester_idx_val1 on tester(val1) using
>> 'org.apache.cassandra.index.sasi.SASIIndex';
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
>>
>> That's it - when Cassandra tries to flush all hell breaks loose (well,
>> maaybe not, but an unhandled error gets logged). Also, the index doesn't
>> actually work subsequently.
>>
>> On Fri, Nov 4, 2016 at 3:58 PM, Voytek Jarnot <vo...@gmail.com>
>> wrote:
>>
>> Wondering if anyone has encountered the same...
>>
>> Full story and stacktraces below, short version is that creating a SASI
>> index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
>> the index on an empty table doesn't fail; however, soon after I start
>> pumping data into the table similar problems occur.
>>
>> I created CASSANDRA-12877 for this, but am beginning to suspect it might
>> be related to CASSANDRA-11990. The thing that's throwing me is that I
>> can't seem to duplicate this with a simple test table.
>>
>> Background:
>>
>> Ended up building/loading a 3.10-SNAPSHOT to try to get past
>> CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
>>
>> 1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
>> 2) created keyspace (SimpleStrategy, RF 1)
>> 3) created table: (simplified version below, many more valX columns
>> present)
>>
>> CREATE TABLE test_table (
>> id1 text,
>> id2 text,
>> id3 date,
>> id4 timestamp,
>> id5 text,
>> val1 text,
>> val2 text,
>> val3 text,
>> task_id text,
>> val4 text,
>> val5 text,
>> PRIMARY KEY ((id1, id2), id3, id4, id5)
>> ) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
>>
>> 4) created materialized view:
>>
>> CREATE MATERIALIZED VIEW test_table_by_task_id AS
>> SELECT *
>> FROM test_table
>> WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND id4
>> IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
>> PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
>> WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5
>> ASC)
>>
>> 5) inserted 27 million "rows" (i.e., unique values for id5)
>> 6) create index attempt
>>
>> create custom index idx_test_table_val5 on test_table(val5) using
>> 'org.apache.cassandra.index.sasi.SASIIndex';
>>
>> 7) no error in cqlsh, but system.log shows many of the following:
>>
>> INFO [SASI-General:1] 2016-11-04 13:46:47,578
>> PerSSTableIndexWriter.java:277 - Flushed index segment
>> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/
>> data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06
>> f0/mc-149-big-SI_idx_test_table_val5.db_0, took 869 ms.
>> ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229 -
>> Exception in thread Thread[SASI-General:1,5,main]
>> java.lang.AssertionError: cannot have more than 8 overflow collisions per
>> leaf, but had: 12
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.createOverflowEntry(
>> AbstractTokenTreeBuilder.java:357) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$
>> DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.
>> write(AbstractTokenTreeBuilder.java:90) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableLevel.flush(OnDiskIndexBuilder.java:446) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableLevel.add(OnDiskIndexBuilder.java:433) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> addTerm(OnDiskIndexBuilder.java:207) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:293) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:258) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:241) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.
>> lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>
>> As well as some of these:
>>
>> ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142 DataTracker.java:168
>> - Can't open index file at /mydir/cassandra/apache-
>> cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-
>> 133dd090a2b411e6b1bf6df2a1af06f0/mc-300-big-SI_idx_test_table_val5.db,
>> skipping.
>> java.lang.IllegalArgumentException: position: 3472329188772431788,
>> limit: 8180
>> at org.apache.cassandra.index.sasi.utils.MappedBuffer.
>> position(MappedBuffer.java:106) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.conf.DataTracker.
>> getIndexes(DataTracker.java:150) [apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSSTable(SASIIndexBuilder.java:156)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.db.compaction.CompactionManager$
>> 14.run(CompactionManager.java:1583) [apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_101]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_101]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_101]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>
>>
>>
Re: SASI index creation assertion error
Posted by Voytek Jarnot <vo...@gmail.com>.
Indeed. I did throw a comment on 11990 - not sure if that triggers emails
to those participants, but was hoping someone would take a look.
On Sat, Nov 5, 2016 at 2:26 AM, DuyHai Doan <do...@gmail.com> wrote:
> So from code review, the error message you get from the log is coming from
> the CASSANDRA-11990: https://github.com/ifesdjeen/cassandra/commit/
> dc4ae57f452e19adbe5a6a2c85f8a4b5a24d4103#diff-
> eae81aa3b81f9b1e07b109c446447a50R357
>
> Now, it's just the consequence of the problem (throwing an assertion
> error), we have to dig further to understand why we fall into this situation
>
> On Sat, Nov 5, 2016 at 5:15 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
>> Can you file a Jira for this? Would be good to make sure 3.10 doesn't get
>> released with this bug.
>> On Fri, Nov 4, 2016 at 6:11 PM Voytek Jarnot <vo...@gmail.com>
>> wrote:
>>
>>> Thought I'd follow-up to myself, in case anyone else comes across this
>>> problem. I found a reasonably easy test case to reproduce the problem:
>>>
>>> This works in 3.9, but doesn't work in 3.10-snapshot:
>>>
>>> CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '1'};
>>> use vjtest ;
>>> create table tester(id1 text, id2 text, id3 text, val1 text, primary
>>> key((id1, id2), id3));
>>> create custom index tester_idx_val1 on tester(val1) using '
>>> org.apache.cassandra.index.sasi.SASIIndex';
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
>>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
>>>
>>> That's it - when Cassandra tries to flush all hell breaks loose (well,
>>> maaybe not, but an unhandled error gets logged). Also, the index doesn't
>>> actually work subsequently.
>>>
>>> On Fri, Nov 4, 2016 at 3:58 PM, Voytek Jarnot <vo...@gmail.com>
>>> wrote:
>>>
>>> Wondering if anyone has encountered the same...
>>>
>>> Full story and stacktraces below, short version is that creating a SASI
>>> index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
>>> the index on an empty table doesn't fail; however, soon after I start
>>> pumping data into the table similar problems occur.
>>>
>>> I created CASSANDRA-12877 for this, but am beginning to suspect it might
>>> be related to CASSANDRA-11990. The thing that's throwing me is that I
>>> can't seem to duplicate this with a simple test table.
>>>
>>> Background:
>>>
>>> Ended up building/loading a 3.10-SNAPSHOT to try to get past
>>> CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
>>>
>>> 1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
>>> 2) created keyspace (SimpleStrategy, RF 1)
>>> 3) created table: (simplified version below, many more valX columns
>>> present)
>>>
>>> CREATE TABLE test_table (
>>> id1 text,
>>> id2 text,
>>> id3 date,
>>> id4 timestamp,
>>> id5 text,
>>> val1 text,
>>> val2 text,
>>> val3 text,
>>> task_id text,
>>> val4 text,
>>> val5 text,
>>> PRIMARY KEY ((id1, id2), id3, id4, id5)
>>> ) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
>>>
>>> 4) created materialized view:
>>>
>>> CREATE MATERIALIZED VIEW test_table_by_task_id AS
>>> SELECT *
>>> FROM test_table
>>> WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND
>>> id4 IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
>>> PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
>>> WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5
>>> ASC)
>>>
>>> 5) inserted 27 million "rows" (i.e., unique values for id5)
>>> 6) create index attempt
>>>
>>> create custom index idx_test_table_val5 on test_table(val5) using '
>>> org.apache.cassandra.index.sasi.SASIIndex';
>>>
>>> 7) no error in cqlsh, but system.log shows many of the following:
>>>
>>> INFO [SASI-General:1] 2016-11-04 13:46:47,578
>>> PerSSTableIndexWriter.java:277 - Flushed index segment
>>> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/
>>> mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-
>>> 149-big-SI_idx_test_table_val5.db_0, took 869 ms.
>>> ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229
>>> - Exception in thread Thread[SASI-General:1,5,main]
>>> java.lang.AssertionError: cannot have more than 8 overflow collisions
>>> per leaf, but had: 12
>>> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r$Leaf.createOverflowEntry(AbstractTokenTreeBuilder.java:357)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder
>>> $DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilde
>>> r.write(AbstractTokenTreeBuilder.java:90) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$Muta
>>> bleDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$Muta
>>> bleLevel.flush(OnDiskIndexBuilder.java:446)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$Muta
>>> bleLevel.add(OnDiskIndexBuilder.java:433) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.addT
>>> erm(OnDiskIndexBuilder.java:207) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.fini
>>> sh(OnDiskIndexBuilder.java:293) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.fini
>>> sh(OnDiskIndexBuilder.java:258) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.fini
>>> sh(OnDiskIndexBuilder.java:241) ~[apache-cassandra-3.10-SNAPSH
>>> OT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$
>>> Index.lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> ~[na:1.8.0_101]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> ~[na:1.8.0_101]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_101]
>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>
>>> As well as some of these:
>>>
>>> ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142
>>> DataTracker.java:168 - Can't open index file at
>>> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/
>>> mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-
>>> 300-big-SI_idx_test_table_val5.db, skipping.
>>> java.lang.IllegalArgumentException: position: 3472329188772431788,
>>> limit: 8180
>>> at org.apache.cassandra.index.sasi.utils.MappedBuffer.position(MappedBuffer.java:106)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
>>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.conf.DataTracker.getIndexes(DataTracker.java:150)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSST
>>> able(SASIIndexBuilder.java:156) [apache-cassandra-3.10-SNAPSHO
>>> T.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1583)
>>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_101]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_101]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [na:1.8.0_101]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> [na:1.8.0_101]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> [na:1.8.0_101]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> [na:1.8.0_101]
>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>
>>>
>>>
>
Re: SASI index creation assertion error
Posted by DuyHai Doan <do...@gmail.com>.
So from code review, the error message you get from the log is coming from
the CASSANDRA-11990:
https://github.com/ifesdjeen/cassandra/commit/dc4ae57f452e19adbe5a6a2c85f8a4b5a24d4103#diff-eae81aa3b81f9b1e07b109c446447a50R357
Now, it's just the consequence of the problem (throwing an assertion
error), we have to dig further to understand why we fall into this situation
On Sat, Nov 5, 2016 at 5:15 AM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Can you file a Jira for this? Would be good to make sure 3.10 doesn't get
> released with this bug.
> On Fri, Nov 4, 2016 at 6:11 PM Voytek Jarnot <vo...@gmail.com>
> wrote:
>
>> Thought I'd follow-up to myself, in case anyone else comes across this
>> problem. I found a reasonably easy test case to reproduce the problem:
>>
>> This works in 3.9, but doesn't work in 3.10-snapshot:
>>
>> CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy',
>> 'replication_factor': '1'};
>> use vjtest ;
>> create table tester(id1 text, id2 text, id3 text, val1 text, primary
>> key((id1, id2), id3));
>> create custom index tester_idx_val1 on tester(val1) using
>> 'org.apache.cassandra.index.sasi.SASIIndex';
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
>>
>> That's it - when Cassandra tries to flush all hell breaks loose (well,
>> maaybe not, but an unhandled error gets logged). Also, the index doesn't
>> actually work subsequently.
>>
>> On Fri, Nov 4, 2016 at 3:58 PM, Voytek Jarnot <vo...@gmail.com>
>> wrote:
>>
>> Wondering if anyone has encountered the same...
>>
>> Full story and stacktraces below, short version is that creating a SASI
>> index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
>> the index on an empty table doesn't fail; however, soon after I start
>> pumping data into the table similar problems occur.
>>
>> I created CASSANDRA-12877 for this, but am beginning to suspect it might
>> be related to CASSANDRA-11990. The thing that's throwing me is that I
>> can't seem to duplicate this with a simple test table.
>>
>> Background:
>>
>> Ended up building/loading a 3.10-SNAPSHOT to try to get past
>> CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
>>
>> 1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
>> 2) created keyspace (SimpleStrategy, RF 1)
>> 3) created table: (simplified version below, many more valX columns
>> present)
>>
>> CREATE TABLE test_table (
>> id1 text,
>> id2 text,
>> id3 date,
>> id4 timestamp,
>> id5 text,
>> val1 text,
>> val2 text,
>> val3 text,
>> task_id text,
>> val4 text,
>> val5 text,
>> PRIMARY KEY ((id1, id2), id3, id4, id5)
>> ) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
>>
>> 4) created materialized view:
>>
>> CREATE MATERIALIZED VIEW test_table_by_task_id AS
>> SELECT *
>> FROM test_table
>> WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND id4
>> IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
>> PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
>> WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5
>> ASC)
>>
>> 5) inserted 27 million "rows" (i.e., unique values for id5)
>> 6) create index attempt
>>
>> create custom index idx_test_table_val5 on test_table(val5) using
>> 'org.apache.cassandra.index.sasi.SASIIndex';
>>
>> 7) no error in cqlsh, but system.log shows many of the following:
>>
>> INFO [SASI-General:1] 2016-11-04 13:46:47,578
>> PerSSTableIndexWriter.java:277 - Flushed index segment
>> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/
>> data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06
>> f0/mc-149-big-SI_idx_test_table_val5.db_0, took 869 ms.
>> ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229 -
>> Exception in thread Thread[SASI-General:1,5,main]
>> java.lang.AssertionError: cannot have more than 8 overflow collisions per
>> leaf, but had: 12
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.createOverflowEntry(
>> AbstractTokenTreeBuilder.java:357) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$
>> DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.
>> write(AbstractTokenTreeBuilder.java:90) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableLevel.flush(OnDiskIndexBuilder.java:446) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableLevel.add(OnDiskIndexBuilder.java:433) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> addTerm(OnDiskIndexBuilder.java:207) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:293) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:258) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:241) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.
>> lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>
>> As well as some of these:
>>
>> ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142 DataTracker.java:168
>> - Can't open index file at /mydir/cassandra/apache-
>> cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-
>> 133dd090a2b411e6b1bf6df2a1af06f0/mc-300-big-SI_idx_test_table_val5.db,
>> skipping.
>> java.lang.IllegalArgumentException: position: 3472329188772431788,
>> limit: 8180
>> at org.apache.cassandra.index.sasi.utils.MappedBuffer.
>> position(MappedBuffer.java:106) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.conf.DataTracker.
>> getIndexes(DataTracker.java:150) [apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSSTable(SASIIndexBuilder.java:156)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.db.compaction.CompactionManager$
>> 14.run(CompactionManager.java:1583) [apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_101]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_101]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_101]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_101]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>
>>
>>
Re: SASI index creation assertion error
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Can you file a Jira for this? Would be good to make sure 3.10 doesn't get
released with this bug.
On Fri, Nov 4, 2016 at 6:11 PM Voytek Jarnot <vo...@gmail.com>
wrote:
> Thought I'd follow-up to myself, in case anyone else comes across this
> problem. I found a reasonably easy test case to reproduce the problem:
>
> This works in 3.9, but doesn't work in 3.10-snapshot:
>
> CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '1'};
> use vjtest ;
> create table tester(id1 text, id2 text, id3 text, val1 text, primary
> key((id1, id2), id3));
> create custom index tester_idx_val1 on tester(val1) using
> 'org.apache.cassandra.index.sasi.SASIIndex';
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
>
> That's it - when Cassandra tries to flush all hell breaks loose (well,
> maaybe not, but an unhandled error gets logged). Also, the index doesn't
> actually work subsequently.
>
> On Fri, Nov 4, 2016 at 3:58 PM, Voytek Jarnot <vo...@gmail.com>
> wrote:
>
> Wondering if anyone has encountered the same...
>
> Full story and stacktraces below, short version is that creating a SASI
> index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
> the index on an empty table doesn't fail; however, soon after I start
> pumping data into the table similar problems occur.
>
> I created CASSANDRA-12877 for this, but am beginning to suspect it might
> be related to CASSANDRA-11990. The thing that's throwing me is that I
> can't seem to duplicate this with a simple test table.
>
> Background:
>
> Ended up building/loading a 3.10-SNAPSHOT to try to get past
> CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
>
> 1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
> 2) created keyspace (SimpleStrategy, RF 1)
> 3) created table: (simplified version below, many more valX columns
> present)
>
> CREATE TABLE test_table (
> id1 text,
> id2 text,
> id3 date,
> id4 timestamp,
> id5 text,
> val1 text,
> val2 text,
> val3 text,
> task_id text,
> val4 text,
> val5 text,
> PRIMARY KEY ((id1, id2), id3, id4, id5)
> ) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
>
> 4) created materialized view:
>
> CREATE MATERIALIZED VIEW test_table_by_task_id AS
> SELECT *
> FROM test_table
> WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND id4
> IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
> PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
> WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5
> ASC)
>
> 5) inserted 27 million "rows" (i.e., unique values for id5)
> 6) create index attempt
>
> create custom index idx_test_table_val5 on test_table(val5) using
> 'org.apache.cassandra.index.sasi.SASIIndex';
>
> 7) no error in cqlsh, but system.log shows many of the following:
>
> INFO [SASI-General:1] 2016-11-04 13:46:47,578
> PerSSTableIndexWriter.java:277 - Flushed index segment
> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-149-big-SI_idx_test_table_val5.db_0,
> took 869 ms.
> ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229 -
> Exception in thread Thread[SASI-General:1,5,main]
> java.lang.AssertionError: cannot have more than 8 overflow collisions per
> leaf, but had: 12
> at
> org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createOverflowEntry(AbstractTokenTreeBuilder.java:357)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.write(AbstractTokenTreeBuilder.java:90)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.flush(OnDiskIndexBuilder.java:446)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.add(OnDiskIndexBuilder.java:433)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.addTerm(OnDiskIndexBuilder.java:207)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:293)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:258)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:241)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>
> As well as some of these:
>
> ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142 DataTracker.java:168
> - Can't open index file at
> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-300-big-SI_idx_test_table_val5.db,
> skipping.
> java.lang.IllegalArgumentException: position: 3472329188772431788, limit:
> 8180
> at
> org.apache.cassandra.index.sasi.utils.MappedBuffer.position(MappedBuffer.java:106)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.conf.DataTracker.getIndexes(DataTracker.java:150)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSSTable(SASIIndexBuilder.java:156)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1583)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>
>
>
Re: SASI index creation assertion error
Posted by Voytek Jarnot <vo...@gmail.com>.
Thought I'd follow-up to myself, in case anyone else comes across this
problem. I found a reasonably easy test case to reproduce the problem:
This works in 3.9, but doesn't work in 3.10-snapshot:
CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '1'};
use vjtest ;
create table tester(id1 text, id2 text, id3 text, val1 text, primary
key((id1, id2), id3));
create custom index tester_idx_val1 on tester(val1) using
'org.apache.cassandra.index.sasi.SASIIndex';
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
That's it - when Cassandra tries to flush all hell breaks loose (well,
maaybe not, but an unhandled error gets logged). Also, the index doesn't
actually work subsequently.
On Fri, Nov 4, 2016 at 3:58 PM, Voytek Jarnot <vo...@gmail.com>
wrote:
> Wondering if anyone has encountered the same...
>
> Full story and stacktraces below, short version is that creating a SASI
> index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
> the index on an empty table doesn't fail; however, soon after I start
> pumping data into the table similar problems occur.
>
> I created CASSANDRA-12877 for this, but am beginning to suspect it might
> be related to CASSANDRA-11990. The thing that's throwing me is that I
> can't seem to duplicate this with a simple test table.
>
> Background:
>
> Ended up building/loading a 3.10-SNAPSHOT to try to get past
> CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
>
> 1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
> 2) created keyspace (SimpleStrategy, RF 1)
> 3) created table: (simplified version below, many more valX columns
> present)
>
> CREATE TABLE test_table (
> id1 text,
> id2 text,
> id3 date,
> id4 timestamp,
> id5 text,
> val1 text,
> val2 text,
> val3 text,
> task_id text,
> val4 text,
> val5 text,
> PRIMARY KEY ((id1, id2), id3, id4, id5)
> ) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
>
> 4) created materialized view:
>
> CREATE MATERIALIZED VIEW test_table_by_task_id AS
> SELECT *
> FROM test_table
> WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND id4
> IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
> PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
> WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5
> ASC)
>
> 5) inserted 27 million "rows" (i.e., unique values for id5)
> 6) create index attempt
>
> create custom index idx_test_table_val5 on test_table(val5) using
> 'org.apache.cassandra.index.sasi.SASIIndex';
>
> 7) no error in cqlsh, but system.log shows many of the following:
>
> INFO [SASI-General:1] 2016-11-04 13:46:47,578
> PerSSTableIndexWriter.java:277 - Flushed index segment
> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/
> data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06
> f0/mc-149-big-SI_idx_test_table_val5.db_0, took 869 ms.
> ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229 -
> Exception in thread Thread[SASI-General:1,5,main]
> java.lang.AssertionError: cannot have more than 8 overflow collisions per
> leaf, but had: 12
> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.
> createOverflowEntry(AbstractTokenTreeBuilder.java:357)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.
> createEntry(AbstractTokenTreeBuilder.java:346) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$
> DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.
> serialize(AbstractTokenTreeBuilder.java:306) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.
> write(AbstractTokenTreeBuilder.java:90) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
> MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
> MutableLevel.flush(OnDiskIndexBuilder.java:446) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
> MutableLevel.add(OnDiskIndexBuilder.java:433) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
> addTerm(OnDiskIndexBuilder.java:207) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
> finish(OnDiskIndexBuilder.java:293) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
> finish(OnDiskIndexBuilder.java:258) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
> finish(OnDiskIndexBuilder.java:241) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.
> lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>
> As well as some of these:
>
> ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142 DataTracker.java:168
> - Can't open index file at /mydir/cassandra/apache-
> cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-
> 133dd090a2b411e6b1bf6df2a1af06f0/mc-300-big-SI_idx_test_table_val5.db,
> skipping.
> java.lang.IllegalArgumentException: position: 3472329188772431788, limit:
> 8180
> at org.apache.cassandra.index.sasi.utils.MappedBuffer.
> position(MappedBuffer.java:106) ~[apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.conf.DataTracker.
> getIndexes(DataTracker.java:150) [apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSSTable(SASIIndexBuilder.java:156)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
> at org.apache.cassandra.db.compaction.CompactionManager$
> 14.run(CompactionManager.java:1583) [apache-cassandra-3.10-
> SNAPSHOT.jar:3.10-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>
>