You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "mck (JIRA)" <ji...@apache.org> on 2018/02/25 21:31:00 UTC
[jira] [Comment Edited] (CASSANDRA-14247) SASI tokenizer for simple
delimiter based entries
[ https://issues.apache.org/jira/browse/CASSANDRA-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376253#comment-16376253 ]
mck edited comment on CASSANDRA-14247 at 2/25/18 9:30 PM:
----------------------------------------------------------
[~mkjellman], have forced pushed the branch again. (let me know if you want to be adding checkpoint commits rather than overwriting the existing commit.)
This adds the test file {{test/resources/tokenization/world_cities_a.csv}}, and a unit test to match. The other unit test methods have been updated to use different delimiters as appropriate for the existing test data files.
Example corridor testing…
{code:java}
create table test ( one text, two int, three text, PRIMARY KEY (one,two) );
# insert a new row, with the contents of test/resources/tokenization/world_cities_a.csv going into column 'three'.
create CUSTOM INDEX on test (three) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = { 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 'delimiter': ',', 'mode': 'prefix', 'analyzed': 'true'};
select one,two from test where three LIKE 'azzazl' ALLOW FILTERING;
{code}
Aside: this tokenizer raises the need for a "exact" mode. Querying a csv inside a column like this is one example where the user may never require wildcarding LIKE clause (using %) and an 'exact' mode would be significantly more performant and use less disk.
was (Author: michaelsembwever):
[~mkjellman], have forced pushed the branch again. (let me know if you want to be adding checkpoint commits rather than overwriting the existing commit.)
This adds the test file {{test/resources/tokenization/world_cities_a.csv}}, and a unit test to match. The other unit test methods have been updated to use different delimiters as appropriate for the existing test data files.
Example corridor testing…
{code:java}
create table test ( one text, two int, three text, PRIMARY KEY (one,two) );
# insert a new row, with the contents of test/resources/tokenization/world_cities_a.csv going into column 'three'.
create CUSTOM INDEX on test (three) USING 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = { 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer', 'delimiter': ',', 'mode': 'prefix', 'analyzed': 'true'};
select one,two from test where three LIKE 'azzazl' ALLOW FILTERING;
{code}
> SASI tokenizer for simple delimiter based entries
> -------------------------------------------------
>
> Key: CASSANDRA-14247
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14247
> Project: Cassandra
> Issue Type: Improvement
> Components: sasi
> Reporter: mck
> Assignee: mck
> Priority: Major
> Fix For: 4.0, 3.11.x
>
>
> Currently SASI offers only two tokenizer options:
> - NonTokenizerAnalyser
> - StandardAnalyzer
> The latter is built upon Snowball, powerful for human languages but overkill for simple tokenization.
> A simple tokenizer is proposed here. The need for this arose as a workaround of CASSANDRA-11182, and to avoid the disk usage explosion when having to resort to {{CONTAINS}}. See https://github.com/openzipkin/zipkin/issues/1861
> Example use of this would be:
> {code}
> CREATE CUSTOM INDEX span_annotation_query_idx
> ON zipkin2.span (annotation_query) USING 'org.apache.cassandra.index.sasi.SASIIndex'
> WITH OPTIONS = {
> 'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.DelimiterAnalyzer',
> 'delimiter': '░',
> 'case_sensitive': 'true',
> 'mode': 'prefix',
> 'analyzed': 'true'};
> {code}
> Original credit for this work goes to https://github.com/zuochangan
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org