You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Ryan McKinley (JIRA)" <ji...@apache.org> on 2009/12/30 19:43:36 UTC
[jira] Created: (SOLR-1690) JSONKeyValueTokenizerFactory -- JSON
Tokenizer
JSONKeyValueTokenizerFactory -- JSON Tokenizer
----------------------------------------------
Key: SOLR-1690
URL: https://issues.apache.org/jira/browse/SOLR-1690
Project: Solr
Issue Type: New Feature
Components: Schema and Analysis
Reporter: Ryan McKinley
Priority: Minor
Sometimes it is nice to group structured data into a single field.
This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
For example, the text:
{code}
{ "hello": "world", "rank":5 }
{code}
gets indexed as two tokens:
|| term position | 1 | 2 |
|| term text | hello:world | rank:5 |
|| term type | word | word |
|| source start,end | 12,17 | 27,28 |
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1690) JSONKeyValueTokenizerFactory -- JSON
Tokenizer
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795737#action_12795737 ]
Ryan McKinley commented on SOLR-1690:
-------------------------------------
I have been using it to have structured data stored in a single field. Kind of like a less cryptic version of:
http://wiki.apache.org/solr/UserTagDesign
I'm not sure it belongs in /trunk, but wanted to post it here so that others could use it if they want...
> JSONKeyValueTokenizerFactory -- JSON Tokenizer
> ----------------------------------------------
>
> Key: SOLR-1690
> URL: https://issues.apache.org/jira/browse/SOLR-1690
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Priority: Minor
> Attachments: noggit-1.0-A1.jar, SOLR-1690-JSONKeyValueTokenizerFactory.patch
>
>
> Sometimes it is nice to group structured data into a single field.
> This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
> {code:xml|title=schema.xml}
> <!-- JSON Field Type -->
> <fieldtype name="json" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
> <analyzer type="index">
> <tokenizer class="solr.JSONKeyValueTokenizerFactory" keepArray="true" hierarchicalKey="false"/>
> <filter class="solr.TrimFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.TrimFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldtype>
> {code}
> Given text:
> {code}
> { "hello": "world", "rank":5 }
> {code}
> indexed as two tokens:
> || term position | 1 | 2 |
> || term text | hello:world | rank:5 |
> || term type | word | word |
> || source start,end | 12,17 | 27,28 |
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1690) JSONKeyValueTokenizerFactory -- JSON
Tokenizer
Posted by "Hayder Marzouk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797313#action_12797313 ]
Hayder Marzouk commented on SOLR-1690:
--------------------------------------
Hi Ryan,
Great solution. It's what i am looking for.
Can u attach the full code please.
> JSONKeyValueTokenizerFactory -- JSON Tokenizer
> ----------------------------------------------
>
> Key: SOLR-1690
> URL: https://issues.apache.org/jira/browse/SOLR-1690
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Priority: Minor
> Attachments: noggit-1.0-A1.jar, SOLR-1690-JSONKeyValueTokenizerFactory.patch
>
>
> Sometimes it is nice to group structured data into a single field.
> This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
> {code:xml|title=schema.xml}
> <!-- JSON Field Type -->
> <fieldtype name="json" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
> <analyzer type="index">
> <tokenizer class="solr.JSONKeyValueTokenizerFactory" keepArray="true" hierarchicalKey="false"/>
> <filter class="solr.TrimFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.TrimFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldtype>
> {code}
> Given text:
> {code}
> { "hello": "world", "rank":5 }
> {code}
> indexed as two tokens:
> || term position | 1 | 2 |
> || term text | hello:world | rank:5 |
> || term type | word | word |
> || source start,end | 12,17 | 27,28 |
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1690) JSONKeyValueTokenizerFactory -- JSON
Tokenizer
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley updated SOLR-1690:
--------------------------------
Attachment: SOLR-1690-JSONKeyValueTokenizerFactory.patch
Here is a simple JSON tokenizer. No tests, but a good place to start if you are looking to do something similar.
> JSONKeyValueTokenizerFactory -- JSON Tokenizer
> ----------------------------------------------
>
> Key: SOLR-1690
> URL: https://issues.apache.org/jira/browse/SOLR-1690
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Priority: Minor
> Attachments: SOLR-1690-JSONKeyValueTokenizerFactory.patch
>
>
> Sometimes it is nice to group structured data into a single field.
> This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
> For example, the text:
> {code}
> { "hello": "world", "rank":5 }
> {code}
> gets indexed as two tokens:
> || term position | 1 | 2 |
> || term text | hello:world | rank:5 |
> || term type | word | word |
> || source start,end | 12,17 | 27,28 |
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1690) JSONKeyValueTokenizerFactory -- JSON
Tokenizer
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley updated SOLR-1690:
--------------------------------
Attachment: noggit-1.0-A1.jar
This tokenizer uses noggit
http://svn.apache.org/repos/asf/labs/noggit/
> JSONKeyValueTokenizerFactory -- JSON Tokenizer
> ----------------------------------------------
>
> Key: SOLR-1690
> URL: https://issues.apache.org/jira/browse/SOLR-1690
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Priority: Minor
> Attachments: noggit-1.0-A1.jar, SOLR-1690-JSONKeyValueTokenizerFactory.patch
>
>
> Sometimes it is nice to group structured data into a single field.
> This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
> For example, the text:
> {code}
> { "hello": "world", "rank":5 }
> {code}
> gets indexed as two tokens:
> || term position | 1 | 2 |
> || term text | hello:world | rank:5 |
> || term type | word | word |
> || source start,end | 12,17 | 27,28 |
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1690) JSONKeyValueTokenizerFactory -- JSON
Tokenizer
Posted by "Ryan McKinley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ryan McKinley updated SOLR-1690:
--------------------------------
Description:
Sometimes it is nice to group structured data into a single field.
This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
{code:xml|title=schema.xml}
<!-- JSON Field Type -->
<fieldtype name="json" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
<analyzer type="index">
<tokenizer class="solr.JSONKeyValueTokenizerFactory" keepArray="true" hierarchicalKey="false"/>
<filter class="solr.TrimFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.TrimFilterFactory" />
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldtype>
{code}
Given text:
{code}
{ "hello": "world", "rank":5 }
{code}
indexed as two tokens:
|| term position | 1 | 2 |
|| term text | hello:world | rank:5 |
|| term type | word | word |
|| source start,end | 12,17 | 27,28 |
was:
Sometimes it is nice to group structured data into a single field.
This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
For example, the text:
{code}
{ "hello": "world", "rank":5 }
{code}
gets indexed as two tokens:
|| term position | 1 | 2 |
|| term text | hello:world | rank:5 |
|| term type | word | word |
|| source start,end | 12,17 | 27,28 |
> JSONKeyValueTokenizerFactory -- JSON Tokenizer
> ----------------------------------------------
>
> Key: SOLR-1690
> URL: https://issues.apache.org/jira/browse/SOLR-1690
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Priority: Minor
> Attachments: noggit-1.0-A1.jar, SOLR-1690-JSONKeyValueTokenizerFactory.patch
>
>
> Sometimes it is nice to group structured data into a single field.
> This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
> {code:xml|title=schema.xml}
> <!-- JSON Field Type -->
> <fieldtype name="json" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
> <analyzer type="index">
> <tokenizer class="solr.JSONKeyValueTokenizerFactory" keepArray="true" hierarchicalKey="false"/>
> <filter class="solr.TrimFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.TrimFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldtype>
> {code}
> Given text:
> {code}
> { "hello": "world", "rank":5 }
> {code}
> indexed as two tokens:
> || term position | 1 | 2 |
> || term text | hello:world | rank:5 |
> || term type | word | word |
> || source start,end | 12,17 | 27,28 |
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1690) JSONKeyValueTokenizerFactory -- JSON
Tokenizer
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795729#action_12795729 ]
Hoss Man commented on SOLR-1690:
--------------------------------
FWIW: I'm finding it hard to imagine use cases that this would be useful for ... so as a result i have no feedback/suggestions on the patch/usage.
> JSONKeyValueTokenizerFactory -- JSON Tokenizer
> ----------------------------------------------
>
> Key: SOLR-1690
> URL: https://issues.apache.org/jira/browse/SOLR-1690
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: Ryan McKinley
> Priority: Minor
> Attachments: noggit-1.0-A1.jar, SOLR-1690-JSONKeyValueTokenizerFactory.patch
>
>
> Sometimes it is nice to group structured data into a single field.
> This (rough) patch, takes JSON input and indexes tokens based on the key values pairs in the json.
> {code:xml|title=schema.xml}
> <!-- JSON Field Type -->
> <fieldtype name="json" class="solr.TextField" positionIncrementGap="100" omitNorms="true">
> <analyzer type="index">
> <tokenizer class="solr.JSONKeyValueTokenizerFactory" keepArray="true" hierarchicalKey="false"/>
> <filter class="solr.TrimFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.KeywordTokenizerFactory"/>
> <filter class="solr.TrimFilterFactory" />
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldtype>
> {code}
> Given text:
> {code}
> { "hello": "world", "rank":5 }
> {code}
> indexed as two tokens:
> || term position | 1 | 2 |
> || term text | hello:world | rank:5 |
> || term type | word | word |
> || source start,end | 12,17 | 27,28 |
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.