You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by 朱诗雄 <zs...@gmail.com> on 2010/11/15 06:07:31 UTC
how to cut indexes for distributed searching?
hi,all
I want to use nutch for distributed searching. But I don't know how to
cut indexes for distributed searching?
Is there a guide for that?
--
Thanks and best regards.
zsx
Re: how to cut indexes for distributed searching?
Posted by 朱诗雄 <zs...@gmail.com>.
Thanks.
I find that parse_text and indexes are using same partitioner--hash partitioner.
So I just put each part in parse_text and indexes dirs to each server.
For example, server0 contains only part-00000, and server1 contains
only part-00001.
I find it works. However, one problem is that some summaries miss but
results are right when I search.
I review the source code of
org.apache.nutch.searcher.DistributedSegmentBean.getSummary(HitDetails[],
Query).
But I can't understand why it happens that some summaries miss.
Can you help me?
2010/11/17 Muwonge Ronald <ss...@gmail.com>:
> Hmmm can you do it on web application level?
> Ronny
> On 11/15/10, 朱诗雄 <zs...@gmail.com> wrote:
>> hi,all
>>
>> I want to use nutch for distributed searching. But I don't know how to
>> cut indexes for distributed searching?
>> Is there a guide for that?
>>
>> --
>> Thanks and best regards.
>>
>> zsx
>>
>
--
Thanks and best regards.
zsx
Re: how to cut indexes for distributed searching?
Posted by Muwonge Ronald <ss...@gmail.com>.
Hmmm can you do it on web application level?
Ronny
On 11/15/10, 朱诗雄 <zs...@gmail.com> wrote:
> hi,all
>
> I want to use nutch for distributed searching. But I don't know how to
> cut indexes for distributed searching?
> Is there a guide for that?
>
> --
> Thanks and best regards.
>
> zsx
>
Re: how to cut indexes for distributed searching?
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-11-15 15:10, Dennis Kubes wrote:
> Usually you wouldn't cut indexes. When doing distributed searching
> usually you are crawling, processing, and indexing a batch of documents
> (say 10 million) at a time and pushing them out to a distributed search
> server on a local file system along with their segments. Then you would
> move on to the next batch and the next until you run out of available
> hardware resources. Then you reset the crawldb so every document is
> crawlable again and you start the process all over.
>
> There isn't an index cutter per se. You can use the segment merger to
> put multiple segments together and then index that segment. I have found
> that the shard approach above is a better option in most cases.
I fully agree with what Dennis said above, just wanted to point out that
there is an IndexSplitter tool (actually two similar tools - a limited
IndexSplitter and flexible MultiPassIndexSplitter) in Lucene that you
can use to split indexes.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: how to cut indexes for distributed searching?
Posted by Dennis Kubes <ku...@apache.org>.
There are two schools of thought on distributed tf/idf. The lazy way and
exact way.
1) Lazy way says that if you have consistent number of docs in each
shard (index) then your tf/idf should be approximate even though the
scoring only pulls from each index individually during calculation.
2) Exact way says we should know the global counts. To do this you would
need to store tf/idf outside of the indexes, create a custom similarity
in Lucene, override Nutch's default Similarity, and pass in the
distributed calculation when doing searches. This isn't impossible. It
just hasn't been implemented yet in Nutch.
To answer your question. The idf is relative to a single index according
to Lucene's DefaultSimilarity which Nutch overrides.
Dennis
On 11/15/2010 07:25 PM, 朱诗雄 wrote:
> So the indexes are combined from some batches of documents in
> different servers. Is there any mechanism to make the df accurate?
> Or the df is just relative to indexes from one batch?
>
> 2010/11/15 Dennis Kubes <ku...@apache.org>:
>> Usually you wouldn't cut indexes. When doing distributed searching usually
>> you are crawling, processing, and indexing a batch of documents (say 10
>> million) at a time and pushing them out to a distributed search server on a
>> local file system along with their segments. Then you would move on to the
>> next batch and the next until you run out of available hardware resources.
>> Then you reset the crawldb so every document is crawlable again and you
>> start the process all over.
>>
>> There isn't an index cutter per se. You can use the segment merger to put
>> multiple segments together and then index that segment. I have found that
>> the shard approach above is a better option in most cases.
>>
>> Dennis
>>
>> On 11/14/2010 11:07 PM, 朱诗雄 wrote:
>>> hi,all
>>>
>>> I want to use nutch for distributed searching. But I don't know how to
>>> cut indexes for distributed searching?
>>> Is there a guide for that?
>>>
>
>
Re: how to cut indexes for distributed searching?
Posted by 朱诗雄 <zs...@gmail.com>.
So the indexes are combined from some batches of documents in
different servers. Is there any mechanism to make the df accurate?
Or the df is just relative to indexes from one batch?
2010/11/15 Dennis Kubes <ku...@apache.org>:
> Usually you wouldn't cut indexes. When doing distributed searching usually
> you are crawling, processing, and indexing a batch of documents (say 10
> million) at a time and pushing them out to a distributed search server on a
> local file system along with their segments. Then you would move on to the
> next batch and the next until you run out of available hardware resources.
> Then you reset the crawldb so every document is crawlable again and you
> start the process all over.
>
> There isn't an index cutter per se. You can use the segment merger to put
> multiple segments together and then index that segment. I have found that
> the shard approach above is a better option in most cases.
>
> Dennis
>
> On 11/14/2010 11:07 PM, 朱诗雄 wrote:
>>
>> hi,all
>>
>> I want to use nutch for distributed searching. But I don't know how to
>> cut indexes for distributed searching?
>> Is there a guide for that?
>>
>
--
Thanks and best regards.
zsx
Re: how to cut indexes for distributed searching?
Posted by Dennis Kubes <ku...@apache.org>.
Usually you wouldn't cut indexes. When doing distributed searching
usually you are crawling, processing, and indexing a batch of documents
(say 10 million) at a time and pushing them out to a distributed search
server on a local file system along with their segments. Then you would
move on to the next batch and the next until you run out of available
hardware resources. Then you reset the crawldb so every document is
crawlable again and you start the process all over.
There isn't an index cutter per se. You can use the segment merger to
put multiple segments together and then index that segment. I have found
that the shard approach above is a better option in most cases.
Dennis
On 11/14/2010 11:07 PM, 朱诗雄 wrote:
> hi,all
>
> I want to use nutch for distributed searching. But I don't know how to
> cut indexes for distributed searching?
> Is there a guide for that?
>