You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Tricia Williams <pg...@uwaterloo.ca> on 2007/11/17 00:03:08 UTC

Payloads, Tokenizers, and Filters. Oh My!

Hi All,

    I'll explain what I'm working on, and then I'll ask my two questions.

    I'm working on the issue 
https://issues.apache.org/jira/browse/SOLR-380 which is a feature 
request that allows one to index a "Structured Document" which is 
anything that can be represented by XML in order to provide more context 
to hits in the result set.  This allows us to do things like query the 
index for "Canada" and be able to not only say that that query matched a 
document titled "Some Nonsense" but also that the query term appeared on 
page 7 of chapter 1.  We can then take this one step further and 
markup/highlight the image of this page based on our OCR and position hit. 

For example:

<book title='Some Nonsense'><chapter title='One'><page name='1'>Some 
text from page one of a book.</page><page name='7'>Some more text from 
page seven of a book. Oh and I'm from Canada.</page></chapter></book>

    I accomplished this by creating a custom Tokenizer which strips the 
xml elements and stores them as a Payload at each of the Tokens created 
from the character data in the input.  The payload is the string that 
describes the XPath at that location.  So for <Canada> the payload is 
"/book[title='Some Nonsense']/chapter[title='One']/page[name='7']"

    The other part of this work is the SolrHighlighter which is less 
important to this list.  I retrieve the TermPositions for the Query's 
Terms and use the TermPosition functionality to get back the payload for 
the hits and build output which shows hit positions categorized by the 
payload they are associated with.

QUESTION 1:  Applying TokenFilters to my Tokenizer creates some strange 
(in my opinion) behavior.  First of all the TermPositions change and 
second the Payload is removed.  Is this the expected behavior, or is 
this a bug?  With the Payload being an "experimental feature" I can 
understand if this persistence just hasn't been implemented yet.  But is 
it, or will it be?

In the following example I will denote a token by {pos,<term 
text>,<payload>}:

input: <class name='mammalia'>Dog, and Cat</class>

XmlPayloadTokenizer:
{1,<Dog,>,</class[name='mammalia'][startPos='0']>},{2,<and>,</class[name='mammalia'][startPos='0']>},{3,<Cat>,</class[name='mammalia'][startPos='0']>}
StopFilter:
{1,<Dog,>,</class[name='mammalia'][startPos='0']>},{2,<Cat>,</class[name='mammalia'][startPos='0']>}
WordDelimiterFilter:
{1,<Dog>,<>} {2,<Cat>,</class[name='mammalia'][startPos='0']>}
LowerCaseFilter:
{1,<dog>,<>} {2,<cat>,</class[name='mammalia'][startPos='0']>}


QUESTION 2:  As I explained I'm storing the String representing the 
XPath of the token as the Payload (well the ByteArray of the String) of 
each token.   Is there a more efficient way to do this?  Is this 
exploiting Payload functionality and will it turn around and bite me 
when I get to indexing hundreds of thousands of documents?  Perhaps I 
shouldn't be relying on the Payload functionality before it is deemed 
not experimental?

I feel these questions are both related to Lucene proper rather than 
Solr, which is why I've posted here.  If you think solr-user is a better 
place to post my questions let me know.

Thanks for your input!
Tricia

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Payloads, Tokenizers, and Filters. Oh My!

Posted by Tricia Williams <pg...@uwaterloo.ca>.
Hi Grant,

    Thanks for your response!

    Taking a closer look at the TokenFilter(s) that causes my problem 
with the Payload are all from org.apache.solr.analysis rather than 
org.apache.lucene.analysis.  I had originally thought that all the 
TokenFilters available through Solr's TokenFilterFactory(s) were part of 
Lucene.  But I guess there are TokenFilters specific to Solr, such as 
the WordDelimiterFilter, that aren't aware of Payloads.  Thanks for 
saying exactly the right thing to make me realize that.
> I guess you just want to be careful about how big your payloads get.
    Erik Hatcher suggested storing the bulky XPath strings in a table of 
contents field and just storing a smaller representation of the 
information at each token with the intention of doing a lookup to get 
the bulky stuff at query time. 
>
> One of the original use cases for payloads was for doing XPath queries.
>
    Has anyone actually completed anything with XPath queries and Payloads?
> Also, the only thing experimental about Payloads is the actual 
> signature of the methods, not the need for them.  If anything, I think 
> you will see an expansion of payload capability in the future.  Also 
> note, that you will probably be interested in adding more Payload 
> querying capability.  And also note, I am in the process of adding the 
> ability to get payloads from Spans, but I am not sure if this gets 
> into 2.3 or not.
>
    I look forward to seeing more of Payloads!  I can already see how 
they can be extremely useful.

Thanks,
Tricia

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: urgent

Posted by rohit saini <mr...@gmail.com>.
Hi,
I think u may need to use escape function to escape the words which does not
come in searching....

Rohit


On 11/17/07, Shakti_Sareen <Sh...@satyam.com> wrote:
>
> Hi
>
>
>
> I am facing problem in searching the word containing forward slash (/).
>
> My index file contains more then one documents.
>
> On searching for the word "U/S" in the claims field I am getting hits.
>
> But the word "U/S" is actually not there in the document.
>
>
>
> Below is the code I am using for searching.
>
> Analyzer analyzer = null;
>
>                                analyzer = new StandardAnalyzer();
>
>                                IndexSearcher searcher = new
> IndexSearcher(fIndexFolder.getAbsolutePath());
>
>                                QueryParser parser = new
> QueryParser("Claim",analyzer);
>
>                                Query query = parser.parse("U/S");
>
>
>                                Hits hits = searcher.search(query);
>
>                                LOGGER.info("Hits of word U/S in claims
> ::::::::  " + hits.length());
>
>                                for (int i = 0; i < hits.length(); i++)
> {
>
>                                                docum = hits.doc(i);
>
>                                                LOGGER.info("file NUMBER
> >>>>   " + docum.get("PatentNumber"));
>
>                                }
>
>
>
> Regards
> SHAKTI SAREEN
>
>
> DISCLAIMER:
> This email (including any attachments) is intended for the sole use of the
> intended recipient/s and may contain material that is CONFIDENTIAL AND
> PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or
> distribution or forwarding of any or all of the contents in this message is
> STRICTLY PROHIBITED. If you are not the intended recipient, please contact
> the sender by email and delete all copies; your cooperation in this regard
> is appreciated.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
VANDE - MATRAM

Re: urgent

Posted by John Wang <jo...@gmail.com>.
I think "U/S" is tokenized to 'U' and 'S'.

Sicne the query parser is using the same analyzer/tokenizer, the
search for "U/S" is actually the same as the phrase query: "U S".
Which explains why it is a hit but not in the index.

-John

On Nov 17, 2007 4:58 AM, Grant Ingersoll <gs...@apache.org> wrote:
> http://wiki.apache.org/lucene-java/LuceneFAQ#head-3558e5121806fb4fce80fc022d889484a9248b71
>
> Have a look at what the StandardTokenizer is doing (inside the
> StandardAnalyzer).
>
>
>
> On Nov 17, 2007, at 7:46 AM, Shakti_Sareen wrote:
>
> > Hi
> >
> >
> >
> > I am facing problem in searching the word containing forward slash
> > (/).
> >
> > My index file contains more then one documents.
> >
> > On searching for the word "U/S" in the claims field I am getting hits.
> >
> > But the word "U/S" is actually not there in the document.
> >
> >
> >
> > Below is the code I am using for searching.
> >
> > Analyzer analyzer = null;
> >
> >                                analyzer = new StandardAnalyzer();
> >
> >                                IndexSearcher searcher = new
> > IndexSearcher(fIndexFolder.getAbsolutePath());
> >
> >                                QueryParser parser = new
> > QueryParser("Claim",analyzer);
> >
> >                                Query query = parser.parse("U/S");
> >
> >
> >                                Hits hits = searcher.search(query);
> >
> >                                LOGGER.info("Hits of word U/S in claims
> > ::::::::  " + hits.length());
> >
> >                                for (int i = 0; i < hits.length(); i++)
> > {
> >
> >                                                docum = hits.doc(i);
> >
> >                                                LOGGER.info("file
> > NUMBER
> >>>>>  " + docum.get("PatentNumber"));
> >
> >                                }
> >
> >
> >
> > Regards
> > SHAKTI SAREEN
> >
> >
> > DISCLAIMER:
> > This email (including any attachments) is intended for the sole use
> > of the intended recipient/s and may contain material that is
> > CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance
> > by others or copying or distribution or forwarding of any or all of
> > the contents in this message is STRICTLY PROHIBITED. If you are not
> > the intended recipient, please contact the sender by email and
> > delete all copies; your cooperation in this regard is appreciated.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: urgent

Posted by Grant Ingersoll <gs...@apache.org>.
http://wiki.apache.org/lucene-java/LuceneFAQ#head-3558e5121806fb4fce80fc022d889484a9248b71

Have a look at what the StandardTokenizer is doing (inside the  
StandardAnalyzer).


On Nov 17, 2007, at 7:46 AM, Shakti_Sareen wrote:

> Hi
>
>
>
> I am facing problem in searching the word containing forward slash  
> (/).
>
> My index file contains more then one documents.
>
> On searching for the word "U/S" in the claims field I am getting hits.
>
> But the word "U/S" is actually not there in the document.
>
>
>
> Below is the code I am using for searching.
>
> Analyzer analyzer = null;
>
>                                analyzer = new StandardAnalyzer();
>
>                                IndexSearcher searcher = new
> IndexSearcher(fIndexFolder.getAbsolutePath());
>
>                                QueryParser parser = new
> QueryParser("Claim",analyzer);
>
>                                Query query = parser.parse("U/S");
>
>
>                                Hits hits = searcher.search(query);
>
>                                LOGGER.info("Hits of word U/S in claims
> ::::::::  " + hits.length());
>
>                                for (int i = 0; i < hits.length(); i++)
> {
>
>                                                docum = hits.doc(i);
>
>                                                LOGGER.info("file  
> NUMBER
>>>>>  " + docum.get("PatentNumber"));
>
>                                }
>
>
>
> Regards
> SHAKTI SAREEN
>
>
> DISCLAIMER:
> This email (including any attachments) is intended for the sole use  
> of the intended recipient/s and may contain material that is  
> CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance  
> by others or copying or distribution or forwarding of any or all of  
> the contents in this message is STRICTLY PROHIBITED. If you are not  
> the intended recipient, please contact the sender by email and  
> delete all copies; your cooperation in this regard is appreciated.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


urgent

Posted by Shakti_Sareen <Sh...@satyam.com>.
Hi 

 

I am facing problem in searching the word containing forward slash (/).

My index file contains more then one documents. 

On searching for the word "U/S" in the claims field I am getting hits.

But the word "U/S" is actually not there in the document.

 

Below is the code I am using for searching.

Analyzer analyzer = null;

                                analyzer = new StandardAnalyzer();

                                IndexSearcher searcher = new
IndexSearcher(fIndexFolder.getAbsolutePath());

                                QueryParser parser = new
QueryParser("Claim",analyzer);

                                Query query = parser.parse("U/S");


                                Hits hits = searcher.search(query);

                                LOGGER.info("Hits of word U/S in claims
::::::::  " + hits.length());

                                for (int i = 0; i < hits.length(); i++)
{

                                                docum = hits.doc(i);

                                                LOGGER.info("file NUMBER
>>>>   " + docum.get("PatentNumber"));

                                }

 

Regards
SHAKTI SAREEN


DISCLAIMER:
This email (including any attachments) is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you are not the intended recipient, please contact the sender by email and delete all copies; your cooperation in this regard is appreciated.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Payloads, Tokenizers, and Filters. Oh My!

Posted by Grant Ingersoll <gs...@apache.org>.
Inline below

On Nov 16, 2007, at 6:03 PM, Tricia Williams wrote:

> Hi All,
>
>   I'll explain what I'm working on, and then I'll ask my two  
> questions.
>
>   I'm working on the issue https://issues.apache.org/jira/browse/SOLR-380 
>  which is a feature request that allows one to index a "Structured  
> Document" which is anything that can be represented by XML in order  
> to provide more context to hits in the result set.  This allows us  
> to do things like query the index for "Canada" and be able to not  
> only say that that query matched a document titled "Some Nonsense"  
> but also that the query term appeared on page 7 of chapter 1.  We  
> can then take this one step further and markup/highlight the image  
> of this page based on our OCR and position hit.
> For example:
>
> <book title='Some Nonsense'><chapter title='One'><page name='1'>Some  
> text from page one of a book.</page><page name='7'>Some more text  
> from page seven of a book. Oh and I'm from Canada.</page></chapter></ 
> book>
>
>   I accomplished this by creating a custom Tokenizer which strips  
> the xml elements and stores them as a Payload at each of the Tokens  
> created from the character data in the input.  The payload is the  
> string that describes the XPath at that location.  So for <Canada>  
> the payload is "/book[title='Some Nonsense']/chapter[title='One']/ 
> page[name='7']"
>
>   The other part of this work is the SolrHighlighter which is less  
> important to this list.  I retrieve the TermPositions for the  
> Query's Terms and use the TermPosition functionality to get back the  
> payload for the hits and build output which shows hit positions  
> categorized by the payload they are associated with.
>
> QUESTION 1:  Applying TokenFilters to my Tokenizer creates some  
> strange (in my opinion) behavior.  First of all the TermPositions  
> change and second the Payload is removed.  Is this the expected  
> behavior, or is this a bug?  With the Payload being an "experimental  
> feature" I can understand if this persistence just hasn't been  
> implemented yet.  But is it, or will it be?
>

Do you have other TokenFilters in your Analyzer?  Are you reusing the  
same Token or creating a new one in your TokenFilters?  If creating a  
new one, you will have to set the payload as it won't be copied down.   
Perhaps we should add a constructor that takes a payload.  On the  
other hand, I think we are going to remove the Payload object in favor  
of just using the byte array.


> In the following example I will denote a token by {pos,<term  
> text>,<payload>}:
>
> input: <class name='mammalia'>Dog, and Cat</class>
>
> XmlPayloadTokenizer:
> {1,<Dog,>,</class[name='mammalia'][startPos='0']>},{2,<and>,</ 
> class[name='mammalia'][startPos='0']>},{3,<Cat>,</ 
> class[name='mammalia'][startPos='0']>}
> StopFilter:
> {1,<Dog,>,</class[name='mammalia'][startPos='0']>},{2,<Cat>,</ 
> class[name='mammalia'][startPos='0']>}
> WordDelimiterFilter:
> {1,<Dog>,<>} {2,<Cat>,</class[name='mammalia'][startPos='0']>}
> LowerCaseFilter:
> {1,<dog>,<>} {2,<cat>,</class[name='mammalia'][startPos='0']>}
>
>
> QUESTION 2:  As I explained I'm storing the String representing the  
> XPath of the token as the Payload (well the ByteArray of the String)  
> of each token.   Is there a more efficient way to do this?  Is this  
> exploiting Payload functionality and will it turn around and bite me  
> when I get to indexing hundreds of thousands of documents?  Perhaps  
> I shouldn't be relying on the Payload functionality before it is  
> deemed not experimental?
>

I think this is reasonable.  Micheal Busch had a nice talk at  
ApacheCon on payloads that you can find at http://people.apache.org/~buschmi/apachecon/AdvancedIndexingLuceneAtlanta07.ppt

I guess you just want to be careful about how big your payloads get.

One of the original use cases for payloads was for doing XPath queries.

Also, the only thing experimental about Payloads is the actual  
signature of the methods, not the need for them.  If anything, I think  
you will see an expansion of payload capability in the future.  Also  
note, that you will probably be interested in adding more Payload  
querying capability.  And also note, I am in the process of adding the  
ability to get payloads from Spans, but I am not sure if this gets  
into 2.3 or not.

Cheers,
Grant

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org