You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "chenlica (via GitHub)" <gi...@apache.org> on 2025/10/21 05:33:45 UTC
[D] Query Rewriter (from old wiki) [texera]
GitHub user chenlica created a discussion: Query Rewriter (from old wiki)
From the page https://github.com/apache/texera/wiki/Query-Rewriter (may be dangling)
=====
Authors:
[Shiladitya Sen](http://github.com/shiladityasen), [Kishore Narendran](http://github.com/kishore-narendran)
Reviewer: Chen Li (**DONE**)
## Synopsis
The purpose of the "QueryRewriter" operator is to correct errors of missing spaces in a query that can lead to incorrect tokenization. For instance, a query "newyork" can be rewritten by this operator to "new york". The operator is be used to return:
* The most likely rewritten query found using a word-frequency dictionary; or
* A set of valid rewritten queries.
## Status
As of 6/3/2016: **COMPLETED**
## Modules
`edu.uci.ics.texera.dataflow.queryrewriter`
## Related Issues
Design: Query Rewriter Issue - https://github.com/Texera/texera/issues/29
## Description
The operator inserts spaces to a query string to find likely words in order to rewrite the query. It has two implementations:
* A dynamic programming algorithm that utilizes a word-frequency dictionary to find the most likely tokenization. This algorithm was adopted from the Chinese characters tokenization performed in the [Srch2 Chinese Tokenization] module (https://github.com/SRCH2/srch2-ngn/blob/master/src/core/analyzer/ChineseTokenizer.cpp#L197). The word-frequency dictionary was derived from [Google unigrams](http://storage.googleapis.com/books/ngrams/books/datasetsv2.html) and the [NLTK English dictionary](http://www.nltk.org/book/ch02.html). The score for each word used for the algorithm is a reciprocal of frequency.
* A recursive algorithm that uses an English dictionary (possibly without word frequencies) to find all combinations of valid tokenizations in a search string. This algorithm that can be found [here](https://github.com/Texera/texera/blob/master/texera/texera-dataflow/src/main/java/edu/uci/ics/texera/dataflow/queryrewriter/QuerySegmenter.java#L143)
## Presentation
[Query Rewriter Dynamic Programming Algorithm](https://docs.google.com/presentation/d/1-Ufi_1G2JYYdHCOWeSRxxchhKYXAP2AMHhBDOvTQyVw/pub?start=true&loop=true&delayms=10000)
GitHub link: https://github.com/apache/texera/discussions/3981
----
This is an automatically sent email for notifications@texera.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@texera.apache.org
Re: [D] Query Rewriter (from old wiki) [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
GitHub user chenlica closed a discussion: Query Rewriter (from old wiki)
From the page https://github.com/apache/texera/wiki/Query-Rewriter (may be dangling)
=====
Authors:
[Shiladitya Sen](http://github.com/shiladityasen), [Kishore Narendran](http://github.com/kishore-narendran)
Reviewer: Chen Li (**DONE**)
## Synopsis
The purpose of the "QueryRewriter" operator is to correct errors of missing spaces in a query that can lead to incorrect tokenization. For instance, a query "newyork" can be rewritten by this operator to "new york". The operator is be used to return:
* The most likely rewritten query found using a word-frequency dictionary; or
* A set of valid rewritten queries.
## Status
As of 6/3/2016: **COMPLETED**
## Modules
`edu.uci.ics.texera.dataflow.queryrewriter`
## Related Issues
Design: Query Rewriter Issue - https://github.com/Texera/texera/issues/29
## Description
The operator inserts spaces to a query string to find likely words in order to rewrite the query. It has two implementations:
* A dynamic programming algorithm that utilizes a word-frequency dictionary to find the most likely tokenization. This algorithm was adopted from the Chinese characters tokenization performed in the [Srch2 Chinese Tokenization] module (https://github.com/SRCH2/srch2-ngn/blob/master/src/core/analyzer/ChineseTokenizer.cpp#L197). The word-frequency dictionary was derived from [Google unigrams](http://storage.googleapis.com/books/ngrams/books/datasetsv2.html) and the [NLTK English dictionary](http://www.nltk.org/book/ch02.html). The score for each word used for the algorithm is a reciprocal of frequency.
* A recursive algorithm that uses an English dictionary (possibly without word frequencies) to find all combinations of valid tokenizations in a search string. This algorithm that can be found [here](https://github.com/Texera/texera/blob/master/texera/texera-dataflow/src/main/java/edu/uci/ics/texera/dataflow/queryrewriter/QuerySegmenter.java#L143)
## Presentation
[Query Rewriter Dynamic Programming Algorithm](https://docs.google.com/presentation/d/1-Ufi_1G2JYYdHCOWeSRxxchhKYXAP2AMHhBDOvTQyVw/pub?start=true&loop=true&delayms=10000)
GitHub link: https://github.com/apache/texera/discussions/3981
----
This is an automatically sent email for notifications@texera.apache.org.
To unsubscribe, please send an email to: notifications-unsubscribe@texera.apache.org