You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael Ji <fj...@yahoo.com> on 2005/10/14 18:40:48 UTC
Document Duplication for Multiple Segment Merge
hi,
When Nutch's IndexMerger.java is called, the indexes
from multiple segment directories are merged to one
target directory.
I wonder how lucene deals with the case when identical
documents existing in two segments. Is the older
document ( lower time stamp ) deleted?
thanks,
Michael Ji,
__________________________________
Yahoo! Music Unlimited
Access over 1 million songs. Try it free.
http://music.yahoo.com/unlimited/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Document Duplication for Multiple Segment Merge
Posted by Michael Ji <fj...@yahoo.com>.
Sorry, I guess I point out a wrong java class name.
I want to be confirmed that if SegmentMerger.java in
Lucene do dedup or not. I tracing down couple of java
class from SegmentMerger.java, such as,
SegmentReader.java, IndexWriter.java,
SegmentReader.java. I didn't see a dedup mechanism
yet.
thanks,
Micheal Ji,
--- Yonik Seeley <ys...@gmail.com> wrote:
> Sorry, I've only briefly looked at Nutch, so you
> should ask on that mailing
> list.
> Lucene doesn't do deduping.
>
>
> -Yonik
> Now hiring -- http://tinyurl.com/7m67g
>
> On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
> >
> > hi Yonik:
> >
> > Does that mean when two documents has same MD5
> content
> > in two different segments, IndexMerger.java will
> keep
> > both of them?
> >
> > When I look at the code of IndexSegment.java, it
> > handle MD5 dedupling by keeping the one with
> higher
> > document ID.
> >
>
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Document Duplication for Multiple Segment Merge
Posted by Yonik Seeley <ys...@gmail.com>.
Sorry, I've only briefly looked at Nutch, so you should ask on that mailing
list.
Lucene doesn't do deduping.
-Yonik
Now hiring -- http://tinyurl.com/7m67g
On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
>
> hi Yonik:
>
> Does that mean when two documents has same MD5 content
> in two different segments, IndexMerger.java will keep
> both of them?
>
> When I look at the code of IndexSegment.java, it
> handle MD5 dedupling by keeping the one with higher
> document ID.
>
Re: Document Duplication for Multiple Segment Merge
Posted by Michael Ji <fj...@yahoo.com>.
hi Yonik:
Does that mean when two documents has same MD5 content
in two different segments, IndexMerger.java will keep
both of them?
When I look at the code of IndexSegment.java, it
handle MD5 dedupling by keeping the one with higher
document ID.
So, when refetching happens, the old segment should be
discarded totally. And, a strategy must be made in
such a way that each segment should relate to a
fetchlist with same interval time. Is it the way Nutch
handling refetching case?
Michael Ji,
--- Yonik Seeley <ys...@gmail.com> wrote:
> There is no concept in Lucene of document identity
> linked to any fields of a
> document.
> You need to handle removal of duplicates yourself.
>
> -Yonik
> Now hiring -- http://tinyurl.com/7m67g
>
>
> On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
> >
> > hi,
> >
> > When Nutch's IndexMerger.java is called, the
> indexes
> > from multiple segment directories are merged to
> one
> > target directory.
> >
> > I wonder how lucene deals with the case when
> identical
> > documents existing in two segments. Is the older
> > document ( lower time stamp ) deleted?
> >
> > thanks,
> >
> > Michael Ji,
> >
> >
> >
> > __________________________________
> > Yahoo! Music Unlimited
> > Access over 1 million songs. Try it free.
> > http://music.yahoo.com/unlimited/
> >
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail:
> java-dev-help@lucene.apache.org
> >
> >
>
__________________________________
Yahoo! Music Unlimited
Access over 1 million songs. Try it free.
http://music.yahoo.com/unlimited/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Document Duplication for Multiple Segment Merge
Posted by Yonik Seeley <ys...@gmail.com>.
There is no concept in Lucene of document identity linked to any fields of a
document.
You need to handle removal of duplicates yourself.
-Yonik
Now hiring -- http://tinyurl.com/7m67g
On 10/14/05, Michael Ji <fj...@yahoo.com> wrote:
>
> hi,
>
> When Nutch's IndexMerger.java is called, the indexes
> from multiple segment directories are merged to one
> target directory.
>
> I wonder how lucene deals with the case when identical
> documents existing in two segments. Is the older
> document ( lower time stamp ) deleted?
>
> thanks,
>
> Michael Ji,
>
>
>
> __________________________________
> Yahoo! Music Unlimited
> Access over 1 million songs. Try it free.
> http://music.yahoo.com/unlimited/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>