You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Pinky Iyer <pi...@yahoo.com> on 2003/04/03 00:13:55 UTC
JSP files
Hi all!
Is there any seperate parser for jsp files. Any other option other than modifying indexHTML.java class is appreciated. I already tried modifying this class, html parsing is fine, but jsp parsing yields all the jsp tags as well in the summary...
Thanks!
Pinky
---------------------------------
Do you Yahoo!?
Yahoo! Tax Center - File online, calculators, forms, and more
RE: Analyzer Incorrect?
Posted by Rob Outar <ro...@ideorlando.org>.
Yeah it has been a bad week. I don't think Query parser is not lowercasing
my fields, maybe it is something I am doing wrong:
public synchronized String[] queryIndex(String query) throws
ParseException,
IOException {
checkForIndexChange();
QueryParser p = new QueryParser("",
new RepositoryIndexAnalyzer());
this.query = p.parse(query);
Hits hits = this.searcher.search(this.query);
return buildReturnArray(hits);
}
When I create Querypaser I do not want it to have default field since
clients can query on whatever field they want. I use my Analyzer which I do
not think is lowercasing the fields because I have tested querying with all
lowercase (got results) with mixed case (no results) so I think my code or
my analyzer is hosed.
Thanks,
Rob
-----Original Message-----
From: Tatu Saloranta [mailto:tatu@hypermall.net]
Sent: Friday, April 04, 2003 9:09 AM
To: Lucene Users List
Subject: Re: Analyzer Incorrect?
On Friday 04 April 2003 05:24, Rob Outar wrote:
> Hi all,
>
> Sorry for the flood of questions this week, clients finally started using
> the search engine I wrote which uses Lucene. When I first started
Yup... that's the root of all evil. :-)
(I'm in similar situation, going through user acceptance test as we speak...
and getting ready to do second version that'll have more advanced metadata
based search using Lucene).
> developing with Lucene the Analyzers it came with did some odd things so I
> decided to implement my own but it is not working the way I expect it to.
> First and foremost I would like to like to have case insensitive searches
> and I do not want to tokenize the fields. No field will ever have a space
If you don't need to tokenize a field, you don't need an analyzer either.
However, to get case insensitive search, you should lower-case field
contents
before adding them to document. QueryParser will do lower casing for search
terms automatically (if you are using it), so matching should work fine
then.
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Analyzer Incorrect?
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Friday 04 April 2003 05:24, Rob Outar wrote:
> Hi all,
>
> Sorry for the flood of questions this week, clients finally started using
> the search engine I wrote which uses Lucene. When I first started
Yup... that's the root of all evil. :-)
(I'm in similar situation, going through user acceptance test as we speak...
and getting ready to do second version that'll have more advanced metadata
based search using Lucene).
> developing with Lucene the Analyzers it came with did some odd things so I
> decided to implement my own but it is not working the way I expect it to.
> First and foremost I would like to like to have case insensitive searches
> and I do not want to tokenize the fields. No field will ever have a space
If you don't need to tokenize a field, you don't need an analyzer either.
However, to get case insensitive search, you should lower-case field contents
before adding them to document. QueryParser will do lower casing for search
terms automatically (if you are using it), so matching should work fine then.
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Analyzer Incorrect?
Posted by Rob Outar <ro...@ideorlando.org>.
Hi all,
Sorry for the flood of questions this week, clients finally started using
the search engine I wrote which uses Lucene. When I first started
developing with Lucene the Analyzers it came with did some odd things so I
decided to implement my own but it is not working the way I expect it to.
First and foremost I would like to like to have case insensitive searches
and I do not want to tokenize the fields. No field will ever have a space
in it so therefore there is no need to tokenize it. I came up with this
Analyzer but case still seems to be an issue:
public TokenStream tokenStream(String field, final Reader reader) {
// do not tokenize any field
TokenStream t = new CharTokenizer(reader) {
protected boolean isTokenChar(char c) {
return true;
}
};
//case insensitive search
t = new LowerCaseFilter(t);
return t;
}
Is there anything I am doing wrong in the Analyzer I have written?
Thanks,
Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Querying Question
Posted by Rob Outar <ro...@ideorlando.org>.
I do not tokenize the file name, the file name is what should be returned
from the query, I think Eric caught my issue, I do a query.ToLowerCase()
which turns AND to and which explains why I got back 4 results, but when I
added the "+" it forced the match. I posted my analyzer a second ago it
should handle the case problem I am experiencing.
Thanks,
Rob
-----Original Message-----
From: Aviran Mordo [mailto:amordo@infosciences.com]
Sent: Thursday, April 03, 2003 5:34 PM
To: 'Lucene Users List'
Subject: RE: Querying Question
You should not tokenize the file name instead you should use
doc.add(new Field(name, value,
true, true, true));
Or
Doc.add(Field.keyword(name,value));
Aviran
-----Original Message-----
From: Rob Outar [mailto:routar@ideorlando.org]
Sent: Thursday, April 03, 2003 5:27 PM
To: Lucene Users List
Subject: RE: Querying Question
Use the following type of Field:
doc.add(new Field(name, value,
true, true, true));
Thanks,
Rob
-----Original Message-----
From: Aviran Mordo [mailto:amordo@infosciences.com]
Sent: Thursday, April 03, 2003 5:19 PM
To: 'Lucene Users List'
Subject: RE: Querying Question
Did you index the value field as a keyword?
Aviran
-----Original Message-----
From: Rob Outar [mailto:routar@ideorlando.org]
Sent: Thursday, April 03, 2003 5:11 PM
To: Lucene Users List
Subject: Querying Question
Importance: High
Hi all,
I am a little fuzzy on complex querying using AND, OR, etc.. For
example:
I have the following name/value pairs
file 1 = name = "checkpoint" value = "filename_1"
file 2 = name = "checkpoint" value = "filename_2"
file 3 = name = "checkpoint" value = "filename_3"
file 4 = name = "checkpoint" value = "filename_4"
I ran the following Query:
name:\"checkpoint\" AND value:\"filenane_1\"
Instead of getting back file 1, I got back all four files?
Then after trying different things I did:
+("name:\"checkpoint\") AND +(value:\"filenane_1\")
it then returned file 1.
Our project queries solely on name value pairs and we need the ability
to query using AND, OR, NOTS, etc.. What the correct syntax for such
queries?
The code I use is :
QueryParser p = new QueryParser("",
new RepositoryIndexAnalyzer());
this.query = p.parse(query.toLowerCase());
Hits hits = this.searcher.search(this.query);
Thanks as always,
Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Querying Question
Posted by Aviran Mordo <am...@infosciences.com>.
You should not tokenize the file name instead you should use
doc.add(new Field(name, value,
true, true, true));
Or
Doc.add(Field.keyword(name,value));
Aviran
-----Original Message-----
From: Rob Outar [mailto:routar@ideorlando.org]
Sent: Thursday, April 03, 2003 5:27 PM
To: Lucene Users List
Subject: RE: Querying Question
Use the following type of Field:
doc.add(new Field(name, value,
true, true, true));
Thanks,
Rob
-----Original Message-----
From: Aviran Mordo [mailto:amordo@infosciences.com]
Sent: Thursday, April 03, 2003 5:19 PM
To: 'Lucene Users List'
Subject: RE: Querying Question
Did you index the value field as a keyword?
Aviran
-----Original Message-----
From: Rob Outar [mailto:routar@ideorlando.org]
Sent: Thursday, April 03, 2003 5:11 PM
To: Lucene Users List
Subject: Querying Question
Importance: High
Hi all,
I am a little fuzzy on complex querying using AND, OR, etc.. For
example:
I have the following name/value pairs
file 1 = name = "checkpoint" value = "filename_1"
file 2 = name = "checkpoint" value = "filename_2"
file 3 = name = "checkpoint" value = "filename_3"
file 4 = name = "checkpoint" value = "filename_4"
I ran the following Query:
name:\"checkpoint\" AND value:\"filenane_1\"
Instead of getting back file 1, I got back all four files?
Then after trying different things I did:
+("name:\"checkpoint\") AND +(value:\"filenane_1\")
it then returned file 1.
Our project queries solely on name value pairs and we need the ability
to query using AND, OR, NOTS, etc.. What the correct syntax for such
queries?
The code I use is :
QueryParser p = new QueryParser("",
new RepositoryIndexAnalyzer());
this.query = p.parse(query.toLowerCase());
Hits hits = this.searcher.search(this.query);
Thanks as always,
Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Querying Question
Posted by Rob Outar <ro...@ideorlando.org>.
Use the following type of Field:
doc.add(new Field(name, value,
true, true, true));
Thanks,
Rob
-----Original Message-----
From: Aviran Mordo [mailto:amordo@infosciences.com]
Sent: Thursday, April 03, 2003 5:19 PM
To: 'Lucene Users List'
Subject: RE: Querying Question
Did you index the value field as a keyword?
Aviran
-----Original Message-----
From: Rob Outar [mailto:routar@ideorlando.org]
Sent: Thursday, April 03, 2003 5:11 PM
To: Lucene Users List
Subject: Querying Question
Importance: High
Hi all,
I am a little fuzzy on complex querying using AND, OR, etc..
For example:
I have the following name/value pairs
file 1 = name = "checkpoint" value = "filename_1"
file 2 = name = "checkpoint" value = "filename_2"
file 3 = name = "checkpoint" value = "filename_3"
file 4 = name = "checkpoint" value = "filename_4"
I ran the following Query:
name:\"checkpoint\" AND value:\"filenane_1\"
Instead of getting back file 1, I got back all four files?
Then after trying different things I did:
+("name:\"checkpoint\") AND +(value:\"filenane_1\")
it then returned file 1.
Our project queries solely on name value pairs and we need the ability
to query using AND, OR, NOTS, etc.. What the correct syntax for such
queries?
The code I use is :
QueryParser p = new QueryParser("",
new RepositoryIndexAnalyzer());
this.query = p.parse(query.toLowerCase());
Hits hits = this.searcher.search(this.query);
Thanks as always,
Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Querying Question
Posted by Aviran Mordo <am...@infosciences.com>.
Did you index the value field as a keyword?
Aviran
-----Original Message-----
From: Rob Outar [mailto:routar@ideorlando.org]
Sent: Thursday, April 03, 2003 5:11 PM
To: Lucene Users List
Subject: Querying Question
Importance: High
Hi all,
I am a little fuzzy on complex querying using AND, OR, etc..
For example:
I have the following name/value pairs
file 1 = name = "checkpoint" value = "filename_1"
file 2 = name = "checkpoint" value = "filename_2"
file 3 = name = "checkpoint" value = "filename_3"
file 4 = name = "checkpoint" value = "filename_4"
I ran the following Query:
name:\"checkpoint\" AND value:\"filenane_1\"
Instead of getting back file 1, I got back all four files?
Then after trying different things I did:
+("name:\"checkpoint\") AND +(value:\"filenane_1\")
it then returned file 1.
Our project queries solely on name value pairs and we need the ability
to query using AND, OR, NOTS, etc.. What the correct syntax for such
queries?
The code I use is :
QueryParser p = new QueryParser("",
new RepositoryIndexAnalyzer());
this.query = p.parse(query.toLowerCase());
Hits hits = this.searcher.search(this.query);
Thanks as always,
Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Querying Question
Posted by Rob Outar <ro...@ideorlando.org>.
Hi all,
I am a little fuzzy on complex querying using AND, OR, etc.. For example:
I have the following name/value pairs
file 1 = name = "checkpoint" value = "filename_1"
file 2 = name = "checkpoint" value = "filename_2"
file 3 = name = "checkpoint" value = "filename_3"
file 4 = name = "checkpoint" value = "filename_4"
I ran the following Query:
name:\"checkpoint\" AND value:\"filenane_1\"
Instead of getting back file 1, I got back all four files?
Then after trying different things I did:
+("name:\"checkpoint\") AND +(value:\"filenane_1\")
it then returned file 1.
Our project queries solely on name value pairs and we need the ability to
query using AND, OR, NOTS, etc.. What the correct syntax for such queries?
The code I use is :
QueryParser p = new QueryParser("",
new RepositoryIndexAnalyzer());
this.query = p.parse(query.toLowerCase());
Hits hits = this.searcher.search(this.query);
Thanks as always,
Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: JSP files
Posted by John Bresnik <jb...@auditintegrity.com>.
Additionally - you can use a crawler to crawl your site, then index the
resulting files. Lucene comes with a crawler called "LARM" but the current
make file doesnt build it properly. I ended using a different crawler called
Spinx :
http://www-2.cs.cmu.edu/~rcm/websphinx/
> Pinky,
>
> You don't want to index the jsp directly, as you would
> be missing the content inserted by the server when the
> pages are accessed. Typically indexing dynamic pages
> is problematic since the content will change
> freqently... That being said, the java.io library
> provides classes for retrieving the content of a URL
> as an input stream. You can write a class to traverse
> your site downloading the URLS and indexing them. It
> will be slower of course than reading HTML from disk
> files.
>
> -Tom
>
> --- Pinky Iyer <pi...@yahoo.com> wrote:
> >
> > Hi all!
> > Is there any seperate parser for jsp files. Any
> > other option other than modifying indexHTML.java
> > class is appreciated. I already tried modifying this
> > class, html parsing is fine, but jsp parsing yields
> > all the jsp tags as well in the summary...
> > Thanks!
> > Pinky
> >
> >
> >
> > ---------------------------------
> > Do you Yahoo!?
> > Yahoo! Tax Center - File online, calculators, forms,
> > and more
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Tax Center - File online, calculators, forms, and more
> http://tax.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: JSP files
Posted by Tomcat Programmer <tc...@yahoo.com>.
Pinky,
You don't want to index the jsp directly, as you would
be missing the content inserted by the server when the
pages are accessed. Typically indexing dynamic pages
is problematic since the content will change
freqently... That being said, the java.io library
provides classes for retrieving the content of a URL
as an input stream. You can write a class to traverse
your site downloading the URLS and indexing them. It
will be slower of course than reading HTML from disk
files.
-Tom
--- Pinky Iyer <pi...@yahoo.com> wrote:
>
> Hi all!
> Is there any seperate parser for jsp files. Any
> other option other than modifying indexHTML.java
> class is appreciated. I already tried modifying this
> class, html parsing is fine, but jsp parsing yields
> all the jsp tags as well in the summary...
> Thanks!
> Pinky
>
>
>
> ---------------------------------
> Do you Yahoo!?
> Yahoo! Tax Center - File online, calculators, forms,
> and more
__________________________________________________
Do you Yahoo!?
Yahoo! Tax Center - File online, calculators, forms, and more
http://tax.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org