You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by chan kang <dd...@gmail.com> on 2006/03/25 05:53:12 UTC
Does Optimize preserve index order?
Hi,
I've been trying to search and show the results
in a sorted order by using search(query, sort).
However, the sorting seems to take too long when
the index size becomes large (currently 7GB).
So,,, I thought it would be nice if the index were
pre-sorted...
What I want to do is to show the results in
chronological order. (btw, the index contains the time field)
One solution I have thought up was:
1. index the whole set
2. read in all the time field values
3. re-index the whole set according to time
(heard that the index order is same as insertion order)
4. optimize.
However, although I think the step 3 would result
in a sorted index, isn't there a possibility that
step 4 might ruin all the sortedness?
- Wouldn't optimizing break the order in which they
are indexed?
Also, is there any other way that I can pre-sort the
index over a specific field (such as time), so that
I wouldn't have to sort while searching? I'm not
even sure if my idea would work well.
Any suggestions are welcome..
Thank you in advance.....
- Chan
Re: Does Optimize preserve index order?
Posted by chan kang <dd...@gmail.com>.
Thank you~
The sorting doesn't seem to take that long (not as long as I expected),
but unfortunately didn't get to measure it this time... Maybe next time I'l
try measuring..
Now I've got another problem..
My final goal is to keep the index sorted reverse-chronologically,
so that, when searching, the relevant documents are shown in the
reverse-chronological order(the most recent document at the top) even
without sorting.
Although presorting the index in chronological order is easy (just
addDocument() for
each new incoming document, and optimize), the reverse seems to be
difficult.
The way I'm handling it now is to
> 1. index without ordering.
2. sort the index reverse-chronologically
> 3. re-index and optimize.
> 4. when a new document comes in, do steps 1-3 again..
>
Steps 1-3 is not that different from sorting in chronological order, but
when it comes to step4, the process becomes very much redundant.
I mean, for example if I wanted to show every search results in a sorted
way, so that
the most recent document comes to the top, I would have to go through steps
1-3
every time when a new document is added(by crawling the web or whatever..).
So, i thought, if the following was possible, it would be much easier...
1. create a new index for incoming documents
2. sort it reverse-chronologically -> index_new
3. use addIndexes() and do "index_new.addIndexes(old_index)"
4. optimize
That way, the new index is sorted, and the old index(which is much much
larger than
incoming ones) is also sorted, and two sorted indexes can be merged to make
a final
sorted version, and this means not re-indexing the whole set of documents in
the original index.
However, I'm not sure whether the addIndexes() also preserves order.
Is it?
Also, is there a better way to do this?
Thanks in advance.
-Chan
2006/3/28, Yonik Seeley <ys...@gmail.com>:
>
> On 3/24/06, chan kang <dd...@gmail.com> wrote:
> > What I want to do is to show the results in
> > chronological order. (btw, the index contains the time field)
> > One solution I have thought up was:
> > 1. index the whole set
> > 2. read in all the time field values
> > 3. re-index the whole set according to time
> > (heard that the index order is same as insertion order)
> > 4. optimize.
> >
> >
> > However, although I think the step 3 would result
> > in a sorted index, isn't there a possibility that
> > step 4 might ruin all the sortedness?
> > - Wouldn't optimizing break the order in which they
> > are indexed?
>
> Index order is retained, so your plan should work fine.
>
> How long is sorting actually taking? FYI, the first time you sort on
> a field will take much longer because a fieldcache entry must be
> populated.
>
> -Yonik
> http://incubator.apache.org/solr Solr, The Open Source Lucene Search
> Server
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Nutch? Solr? Red-piranha?
Posted by Eivind Hasle Amundsen <ei...@ifi.uio.no>.
> I don't have any experience with Red-piranha.
I'm wondering if the Red-Piranha project is still going strong. The
so-called "community edition" seems to be lacking development according
to the CVS repository:
http://cvs.sourceforge.net/viewcvs.py/red-piranha/CVSROOT/
(The last commits seem to be ~8 months old.)
Does anyone know about this?
--
Eivind Hasle Amundsen | eivinha@ifi.uio.no
Student, Department of Informatics, University of Oslo
mob: +47 90789762 | web: http://www.eha.no/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Nutch? Solr? Red-piranha?
Posted by Yonik Seeley <ys...@gmail.com>.
On 3/28/06, Michael Levy <mi...@michaelrlevy.com> wrote:
> I'm looking for advice on selecting a search application. I'm
> responsible for developing a new search platform for use in a historical
> research organization and museum. I've pretty much decided on Lucene as
> the library for custom servlet apps that would use the Lucene API directly.
>
> At the same time, we have a number of application ideas that could
> probably use a flexible/customizable off-the-shelf crawling
> application. For starters, it would be pretty basic stuff like indexing
> PDF files using some library, returning links that have been translated
> to point to a Tomcat virtual directory containing the files. But our
> apps could quickly get more complex as we think of new search ideas.
>
> I'm having a hard time comparing and contrasting Nutch, Solr, and
> Red-piranha. I would appreciate anyone offering your ideas or
> experiences about which of these (or any other comprehensive search
> solutions) are good for which types of applications. TIA!
I'd use Solr for highly structured data (documents with multiple
fields) and very configurable text analysis per-field. Think of it
more like a database, but designed for full-text search. We are
working on adding easy faceted browsing and indexing of SQL databases.
I'd use Nutch for web-search (a free google replacement): crawling
(discovering), indexing web pages, and automatically handling
different types of human readable documents like HTML, PDF, etc.
I don't have any experience with Red-piranha.
Example use cases as I see it:
To index a music web site with lots of articles: Nutch
To index a music collection (structured data like title, album,
author, year, genre, etc): Solr
-Yonik
http://incubator.apache.org/solr Solr, The Open Source Lucene Search Server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Nutch? Solr? Red-piranha?
Posted by Michael Levy <mi...@michaelrlevy.com>.
I'm looking for advice on selecting a search application. I'm
responsible for developing a new search platform for use in a historical
research organization and museum. I've pretty much decided on Lucene as
the library for custom servlet apps that would use the Lucene API directly.
At the same time, we have a number of application ideas that could
probably use a flexible/customizable off-the-shelf crawling
application. For starters, it would be pretty basic stuff like indexing
PDF files using some library, returning links that have been translated
to point to a Tomcat virtual directory containing the files. But our
apps could quickly get more complex as we think of new search ideas.
I'm having a hard time comparing and contrasting Nutch, Solr, and
Red-piranha. I would appreciate anyone offering your ideas or
experiences about which of these (or any other comprehensive search
solutions) are good for which types of applications. TIA!
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Does Optimize preserve index order?
Posted by Yonik Seeley <ys...@gmail.com>.
On 3/24/06, chan kang <dd...@gmail.com> wrote:
> What I want to do is to show the results in
> chronological order. (btw, the index contains the time field)
> One solution I have thought up was:
> 1. index the whole set
> 2. read in all the time field values
> 3. re-index the whole set according to time
> (heard that the index order is same as insertion order)
> 4. optimize.
>
>
> However, although I think the step 3 would result
> in a sorted index, isn't there a possibility that
> step 4 might ruin all the sortedness?
> - Wouldn't optimizing break the order in which they
> are indexed?
Index order is retained, so your plan should work fine.
How long is sorting actually taking? FYI, the first time you sort on
a field will take much longer because a fieldcache entry must be
populated.
-Yonik
http://incubator.apache.org/solr Solr, The Open Source Lucene Search Server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org