You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Илья Казначеев <il...@office.uw.ru> on 2008/04/03 14:24:15 UTC
Implementing CMS search function using Lucene
Hello.
We've designing a CMS in Java, and I've trying to implement site search
function using lucene.
The basic conception is that:
- Site features numerous objects that we'd like to throw into index: pages,
various text blocks on those pages, descriptions and keyword lists of those
pages, static bits of html, goods sections with goods inside them, etc, etc.
- There would be a search form that would be occasionally used by site
visitors.
- Visitors are highly unlikely to use advanced queries. I assume 95% queries
would be either a few keywords or a phrase to search. We have to find the
best matches for such queries.
- The thing I want to introduce is "phrase in quotes" to search for exact
phrase. Also, most our sites are in Russian, so some, even if rudimentary,
support for Russian morphology is a plus.
I've dug into examples and have a following set of questions:
- Our objects are fairly structured, so I would like to introduce a lot of
fields, something like five different for each object type.
But, as far as I see, all Queries are going to search only one field.
This is certainly bad because users surely want to search *all* the fields at
once. The aren't going to bother with queries.
Maybe I can add queries over every field joined by 'or' operation, but
wouldn't that be too slow? I don't want it to work more than half second on
reasonable sized index. Also, I don't want to hard-code exact list of fields,
I might add them as I develop the system. Is this doable, would that work? Or
I'll have to stuff all text content from object into one blob-field and query
that? Which way is more reasonable?
- Our objects have their hierarchy, e.g., blocks belong to page. Is there a
way to make Lucene govern parent-child relation, somehow summing hits in all
childs to find the best-matching parent? I assume, no, then is there a way
for me to go thru matching documents list, reducing it by 'adding' blocks'
scores to find the best matching page?
- Is there a way to set weights for different fields? Let's say, content have
a weight of 1, title have a weight of 5 and picture subscribe have a weight
of 0.5. If no, can I do that by hand?
- Is there something to support Russian morphology (it's all like "the last n
letters of a word might change, we should match all forms") for either
indexer or searcher? Maybe "inexact match", QueryParser's ~ operator, would
be enough? I heard Nutch project have something like that, but I wonder if I
would be able to reuse parts of Nutch, and I surely can't use Nutch as a
whole.
If there are another considerations, they're welcome.
Thanks for your probable replies.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Implementing CMS search function using Lucene
Posted by Илья Казначеев <il...@office.uw.ru>.
В сообщении от Sunday 13 April 2008 14:20:01 Grant Ingersoll написал(а):
Thanks for your reply!
> > I don't want it to work more than half second on
> > reasonable sized index. Also, I don't want to hard-code exact list
> > of fields,
> > I might add them as I develop the system. Is this doable, would that
> > work? Or
> > I'll have to stuff all text content from object into one blob-field
> > and query
> > that? Which way is more reasonable?
> I'd probably do both, then you can handle generic queries as well as
> field specific ones.
There wouldn't be field specific queries, but I plan to have different boosts.
> > - Is there something to support Russian morphology (it's all like
> > "the last n
> > letters of a word might change, we should match all forms") for either
> > indexer or searcher?
>
> Check contrib/analyzers, I see some Russian analyzers in there, but I
> can't speak to the quality.
Thank you, I'll look there.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Implementing CMS search function using Lucene
Posted by Grant Ingersoll <gs...@apache.org>.
For starters, you might have a look at Jackrabbit (Content Repo. built
on Lucene) as I know it powers several CMS systems.
More below.
On Apr 3, 2008, at 8:24 AM, Илья Казначеев wrote:
> Hello.
>
> We've designing a CMS in Java, and I've trying to implement site
> search
> function using lucene.
>
> The basic conception is that:
> - Site features numerous objects that we'd like to throw into index:
> pages,
> various text blocks on those pages, descriptions and keyword lists
> of those
> pages, static bits of html, goods sections with goods inside them,
> etc, etc.
> - There would be a search form that would be occasionally used by site
> visitors.
> - Visitors are highly unlikely to use advanced queries. I assume 95%
> queries
> would be either a few keywords or a phrase to search. We have to
> find the
> best matches for such queries.
> - The thing I want to introduce is "phrase in quotes" to search for
> exact
> phrase. Also, most our sites are in Russian, so some, even if
> rudimentary,
> support for Russian morphology is a plus.
>
> I've dug into examples and have a following set of questions:
> - Our objects are fairly structured, so I would like to introduce a
> lot of
> fields, something like five different for each object type.
> But, as far as I see, all Queries are going to search only one field.
> This is certainly bad because users surely want to search *all* the
> fields at
> once. The aren't going to bother with queries.
You can search whatever fields you want. It's all a question of how
you generate your queries. Typically, one has an "all" field as well.
>
> Maybe I can add queries over every field joined by 'or' operation, but
> wouldn't that be too slow?
Probably not. I suppose if you are talking hundreds of fields
> I don't want it to work more than half second on
> reasonable sized index. Also, I don't want to hard-code exact list
> of fields,
> I might add them as I develop the system. Is this doable, would that
> work? Or
> I'll have to stuff all text content from object into one blob-field
> and query
> that? Which way is more reasonable?
I'd probably do both, then you can handle generic queries as well as
field specific ones.
>
> - Our objects have their hierarchy, e.g., blocks belong to page. Is
> there a
> way to make Lucene govern parent-child relation, somehow summing
> hits in all
> childs to find the best-matching parent? I assume, no, then is there
> a way
> for me to go thru matching documents list, reducing it by 'adding'
> blocks'
> scores to find the best matching page?
You have to manage parent-child yourself. I am pretty sure Jackrabbit
does this.
>
> - Is there a way to set weights for different fields? Let's say,
> content have
> a weight of 1, title have a weight of 5 and picture subscribe have a
> weight
> of 0.5. If no, can I do that by hand?
Field.setBoost()
>
> - Is there something to support Russian morphology (it's all like
> "the last n
> letters of a word might change, we should match all forms") for either
> indexer or searcher?
Check contrib/analyzers, I see some Russian analyzers in there, but I
can't speak to the quality.
> Maybe "inexact match", QueryParser's ~ operator, would
> be enough? I heard Nutch project have something like that, but I
> wonder if I
> would be able to reuse parts of Nutch, and I surely can't use Nutch
> as a
> whole.
Not sure.
>
>
> If there are another considerations, they're welcome.
>
> Thanks for your probable replies.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Implementing CMS search function using Lucene
Posted by Matthew Hall <mh...@informatics.jax.org>.
You could try something like this, which use when I put my own documents
together:
public Document getDocument(){
Document doc = new Document();
doc.add(new Field("db_key", this.getDb_key(), Field.Store.YES,
Field.Index.UN_TOKENIZED));
doc.add(new Field("acc_ids", this.getAcc_ids(), Field.Store.YES,
Field.Index.TOKENIZED));
doc.add(setBoost(new Field("name", this.getName(),
Field.Store.YES, Field.Index.TOKENIZED), 1.0f));
doc.add(setBoost(new Field("symbol", this.getSymbol(),
Field.Store.YES, Field.Index.TOKENIZED), 1.0f));
doc.add(setBoost(new Field("synonyms", this.getSynonyms(),
Field.Store.YES, Field.Index.TOKENIZED), .8f));
doc.add(setBoost(new Field("allele_nomen",
this.getAllele_nomen(), Field.Store.YES, Field.Index.TOKENIZED), .6f));
doc.add(setBoost(new Field("old_nomen", this.getOld_nomen(),
Field.Store.YES, Field.Index.TOKENIZED), .4f));
doc.add(setBoost(new Field("orth_nomen", this.getOrth_nomen(),
Field.Store.YES, Field.Index.TOKENIZED), .2f));
return doc;
}
private static Field setBoost (Field workField, float boost)
{
workField.setBoost(boost);
return workField;
}
It works out pretty well for me anyhow.
Илья Казначеев wrote:
> В сообщении от Thursday 03 April 2008 16:24:15 Илья Казначеев написал(а):
>
>
>> - Is there a way to set weights for different fields? Let's say, content
>> have a weight of 1, title have a weight of 5 and picture subscribe have a
>> weight of 0.5. If no, can I do that by hand?
>>
> Already found field.setBoost().
> Sorry for asking lame questions :(
>
> By the way, if setBoost() returned this, it would be much easier to assemble
> document, one line instead of three. Chaining rules.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
--
Matthew Hall
Software Engineer
Mouse Genome Informatics
mhall@informatics.jax.org
(207) 288-6012
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Implementing CMS search function using Lucene
Posted by Илья Казначеев <il...@office.uw.ru>.
В сообщении от Thursday 03 April 2008 16:24:15 Илья Казначеев написал(а):
> - Is there a way to set weights for different fields? Let's say, content
> have a weight of 1, title have a weight of 5 and picture subscribe have a
> weight of 0.5. If no, can I do that by hand?
Already found field.setBoost().
Sorry for asking lame questions :(
By the way, if setBoost() returned this, it would be much easier to assemble
document, one line instead of three. Chaining rules.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org