You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by bib_lucene bib <bi...@yahoo.com> on 2005/11/11 18:54:36 UTC
Document as Paramter (Rephrased)
Hi Erik & All
My requirement is that I do a search, the results of the search are displayed. I am displaying results by using getbestfragmets and highlighting searched text.
So basically the user can search and see what documents matched his search with snippets of text shown in the result of search.
Now the user needs to select one of the results and view the document completely.
What I did is along with indexing I stored the document in the ROOT webapplication and gave a link to the file. This worked fine.
However, now I need to display the full document with search terms highlighted.
As I understand now I can do this...
1. Read the whole document again and apply some css to highlight terms.
2. Create a new Lucene Document -> Read the file -> Add file Content to Document -> Use nullfragmenter -> Apply CSS -> Display result.
I have many types of documents as word, pdf etc. I am already extracting text and putting in Index. ( Using your example in chapter7)
Question is , is there a way I can just show the content from index rather then parsing the document again, when the user wants to see full document from search results page.
I am stuck here...
Thanks for your help.
bib
---------------------------------
Yahoo! FareChase - Search multiple travel sites in one click.
Re: Document as Paramter (Rephrased)
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On 11 Nov 2005, at 20:32, bib_lucene bib wrote:
> -- Text I want to highlight is stored in the file system and index
> -- I can search and highlight the searched terms in results page
> ( just snippets)
> -- I have given a download link next to snippets ( which will
> point to file I stored in ROOT webapp of tomcat)
>
> I understood the concept of NullFragmenter, sorry for repeating
> myself...
> It is something like a google search. In google if I enter search
> term "highlight" and click on search, I get back search results
> with word "highlight" in bold. ( I can do that)
If you're using NullFragmenter (which I just committed to contrib/
highlighter) then you're not getting snippets, you're getting the
full text. But this is unrelated to your main question.
> Now when I click on the links (Ex: GNU Source-highlight 2.2) which
> is http://www.gnu.org/software/src-highlite/source-highlight.html I
> want the term 'highlight" in bold when the page is displayed (This
> I do not know how to do)
>
> I will index docs like html, pdf, word etc.
> As I have already extracted text using textminer etc, question is
> when I click on "show full document" link in search results page
> which I will give below the highlighted search snippet how can i
> pull out full text of the document from index. As there is no
> unique identifier for each doc in the index (?)
Well, as I said, from here on out is really in the domain of your
application and not Lucene and the Highlighter. Perhaps your index
should have a unique identifying key field (Field.Keyword works well
for this). And then your links to the full text should be augmented
to have that key in them, and that link needs to be to something
dynamic, like a servlet, rather than a static link, that passes in
the id of the document to be highlighted and the query to use with
Highlighter.
> Or do I just have to extract the text from the file stored in file
> system and pass it to the highlighter.
Perhaps. I cannot decide this for you as it is, again, your
application business logic that determines where the text of a
document is. You already mentioned the text is in the Lucene index
in your case, so you could pass the document id (or the path, or some
unique key) to the highlighting servlet, retrieve that document from
Lucene, highlight the text using the NullFragmenter, and serve up
that highlighted text.
> Here is the code snippet.
> LuceneHitHighlighter highlighter = new LuceneHitHighlighter
> (queryStr, "snippet", "body");
> for (int i = 0; i < hits.size(); i++) {
> Document doc = (Document) hits.get(i);
> highlighter.doHighlight(doc);
> out.println("SNIPPET: " + doc.get("snippet"));
> out.println("<hr>");
You're adding a field to the document in doHighlight - this is not
something I'd recommend. It is not persistent, just so you know. I
recommend simply returning the highlighted String from that method
rather than adding a field.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Document as Paramter (Rephrased)
Posted by bib_lucene bib <bi...@yahoo.com>.
Thanks for your time.
-- Text I want to highlight is stored in the file system and index
-- I can search and highlight the searched terms in results page ( just snippets)
-- I have given a download link next to snippets ( which will point to file I stored in ROOT webapp of tomcat)
I understood the concept of NullFragmenter, sorry for repeating myself...
It is something like a google search. In google if I enter search term "highlight" and click on search, I get back search results with word "highlight" in bold. ( I can do that)
Now when I click on the links (Ex: GNU Source-highlight 2.2) which is http://www.gnu.org/software/src-highlite/source-highlight.html I want the term 'highlight" in bold when the page is displayed (This I do not know how to do)
I will index docs like html, pdf, word etc.
As I have already extracted text using textminer etc, question is when I click on "show full document" link in search results page which I will give below the highlighted search snippet how can i pull out full text of the document from index. As there is no unique identifier for each doc in the index (?)
Or do I just have to extract the text from the file stored in file system and pass it to the highlighter. My only concern here would be if like 300 users do this operation it might fail with an error like maximum files open limit exceeded etc.
Here is the code snippet.
LuceneHitHighlighter highlighter = new LuceneHitHighlighter(queryStr, "snippet", "body");
for (int i = 0; i < hits.size(); i++) {
Document doc = (Document) hits.get(i);
highlighter.doHighlight(doc);
out.println("SNIPPET: " + doc.get("snippet"));
out.println("<hr>");
Using Standard Analyzer
public LuceneHitHighlighter(String queryText, String highlightFieldName, String textFieldName)
throws ParseException
{
Query query = QueryParser.parse(queryText, textFieldName, _analyzer);
_highlighter = new Highlighter(_formatter, new QueryScorer(query));
_highlighter.setTextFragmenter(new SimpleFragmenter(100));
_maxNumFragmentsRequired = 3;
_textFieldName = textFieldName;
_highlightFieldName = highlightFieldName;
}
public void doHighlight(Document doc)
{
String highlightText = "";
String text = doc.get(_textFieldName);
if (text != null) {
TokenStream tokenStream = _analyzer.tokenStream(_textFieldName, new StringReader(text));
try {
highlightText = _highlighter.getBestFragments(tokenStream, text, _maxNumFragmentsRequired, "...");
// just store highlight text into document, nothing else
doc.add(new Field(_highlightFieldName, highlightText, true, false, false));
}
}
}
Erik Hatcher <er...@ehatchersolutions.com> wrote:
On 11 Nov 2005, at 12:54, bib_lucene bib wrote:
> My requirement is that I do a search, the results of the search are
> displayed. I am displaying results by using getbestfragmets and
> highlighting searched text.
>
> So basically the user can search and see what documents matched his
> search with snippets of text shown in the result of search.
>
> Now the user needs to select one of the results and view the
> document completely.
>
> What I did is along with indexing I stored the document in the ROOT
> webapplication and gave a link to the file. This worked fine.
>
> However, now I need to display the full document with search terms
> highlighted.
>
> As I understand now I can do this...
>
> 1. Read the whole document again and apply some css to highlight
> terms.
>
> 2. Create a new Lucene Document -> Read the file -> Add file
> Content to Document -> Use nullfragmenter -> Apply CSS -> Display
> result.
There is no need to create a new Lucene Document just to highlight
text. The text you highlight does not have to be the text you
indexed even. The Highlighter takes a String to highlight and a
Query to extract the terms to highlight from.
> Question is , is there a way I can just show the content from index
> rather then parsing the document again, when the user wants to see
> full document from search results page.
Well, the question really comes back to you.... where is the text you
want to highlight? You can store it in a Lucene index, in a
database, on the filesystem, wherever. Neither Lucene nor the
Highlighter are related to that decision whatsoever. So using a
NullFragmenter, which I will commit to contrib/highlighter soon, use
the Highlighter on the full text.
If you're still having issues, send us a (brief, please!) piece of
code you're using to highlight.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------
Yahoo! FareChase - Search multiple travel sites in one click.
Re: Document as Paramter (Rephrased)
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On 11 Nov 2005, at 12:54, bib_lucene bib wrote:
> My requirement is that I do a search, the results of the search are
> displayed. I am displaying results by using getbestfragmets and
> highlighting searched text.
>
> So basically the user can search and see what documents matched his
> search with snippets of text shown in the result of search.
>
> Now the user needs to select one of the results and view the
> document completely.
>
> What I did is along with indexing I stored the document in the ROOT
> webapplication and gave a link to the file. This worked fine.
>
> However, now I need to display the full document with search terms
> highlighted.
>
> As I understand now I can do this...
>
> 1. Read the whole document again and apply some css to highlight
> terms.
>
> 2. Create a new Lucene Document -> Read the file -> Add file
> Content to Document -> Use nullfragmenter -> Apply CSS -> Display
> result.
There is no need to create a new Lucene Document just to highlight
text. The text you highlight does not have to be the text you
indexed even. The Highlighter takes a String to highlight and a
Query to extract the terms to highlight from.
> Question is , is there a way I can just show the content from index
> rather then parsing the document again, when the user wants to see
> full document from search results page.
Well, the question really comes back to you.... where is the text you
want to highlight? You can store it in a Lucene index, in a
database, on the filesystem, wherever. Neither Lucene nor the
Highlighter are related to that decision whatsoever. So using a
NullFragmenter, which I will commit to contrib/highlighter soon, use
the Highlighter on the full text.
If you're still having issues, send us a (brief, please!) piece of
code you're using to highlight.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org