You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by hzhong <he...@gmail.com> on 2007/12/15 17:25:37 UTC
DFS search
Hello,
Why can't we search on the Hadoop DFS?
Thanks
--
View this message in context: http://www.nabble.com/DFS-search-tp14352284p14352284.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: DFS search
Posted by Erick Erickson <er...@gmail.com>.
Do you realize that your question is completely unanswerable? You
provide no details about what "can't search" means. You provide no
details about how you set things up. You provide no details about...
anything.
Please try this again and try reading your message as though you
were on the receiving end. Is there enough information to make
any kind of educated guess about what's really going on?
Best
Erick
On Dec 15, 2007 11:25 AM, hzhong <he...@gmail.com> wrote:
>
> Hello,
>
> Why can't we search on the Hadoop DFS?
>
> Thanks
> --
> View this message in context:
> http://www.nabble.com/DFS-search-tp14352284p14352284.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>
Re: DFS search
Posted by Ken Krugler <kk...@transpac.com>.
>So, whether we do the crawling by using multiple nodes or running it
>on a single node with the regular bin/generate, bin/fetcher, series of
>commands, finally we get a single crawldb on the local system where
>Nutch is supposed to run. Am I right?
The "crawldb" is just where Nutch keeps track of the state of URLs it
knows about.
The end result of a crawl is series of "segments" that contain the
downloaded web page content, and (once you've indexed them) the
Lucene index for each segment.
At this point you have the choice of merging the segments and indexes
into a single segment that you'd serve up with a regular Nutch search
bean, or you can generate multiple segments that live on multiple
"remote searcher" boxes.
Part of the confusion with Nutch is that it is both a crawler and a
search server, and the architecture/mechanism for running a
distributed crawl (Hadoop DFS/MapReduce) is completely different from
the architecture/mechanism for running a distributed search server.
-- Ken
>On Dec 16, 2007 11:47 AM, Dennis Kubes <ku...@apache.org> wrote:
>> If you are talking about the clustering plugin, that is about grouping
>> (hopefully) related documents in the search results.
>>
>> Running the crawler and other nutch processes on multiple nodes is nutch
>> and hadoop running the map reduce paradigm. Moving final indexes and
>> database to local file systems for searching is simply best practices.
>>
>> Dennis Kubes
>>
>>
>> Bent Hugh wrote:
>> > I am a little confused. In the Nutch wiki there are chapters on
>> > clustering. I have never tried them though. So what is clustering
>> > about? Is it running the crawler on multiple nodes and creating
>> > crawldb on multiple nodes? And then finally merging all these on a
>> > local system and running the Nutch web-gui from that?
>> >
>> >
>> > On Dec 16, 2007 10:17 AM, Dennis Kubes <ku...@apache.org> wrote:
>> >> Technically you can. The speed for most search applications would be
>> >> unacceptable. Searching of indexes is best done on local files systems
>> >> for speed.
>> >>
>> >> Dennis Kubes
>> >>
>> >>
>> >> hzhong wrote:
>> >>> Hello,
>> >>>
>> >>> Why can't we search on the Hadoop DFS?
>> >>>
>> >>> Thanks
>>
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"If you can't find it, you can't fix it"
Re: DFS search
Posted by Dennis Kubes <ku...@apache.org>.
Bent Hugh wrote:
> So, whether we do the crawling by using multiple nodes or running it
> on a single node with the regular bin/generate, bin/fetcher, series of
> commands, finally we get a single crawldb on the local system where
> Nutch is supposed to run. Am I right?
Yes and no. The artifacts of nutch are the crawldb, segments (which
contains a generated fetch list, fetched content, content text, and
content meta data), linkdb, and indexes. If you are running everything
on a single node then everything would be on the local file system.
We run over 100 nodes so the way we do it is we have a master crawldb
and master linkdb. We generate, fetch, update, index, and deploy pieces
or as google calls them shards of 1-2M pages. The processing happens
using map reduce and dfs.
During each shard processing cycle we update both the master crawldb and
linkdb. We use these master databases for generating new fetch lists
and for scoring during indexing. Each shard eventually has its own
segments, indexes, and a portion of the master linkdb copied to a shard
linkdb to serve only the urls in that shard. These three are generated
and stored on the dfs and then copied to a local file system where they
can be searched. The deploy of the shard is to a dedicated search
server on its local file system. The number of shards grows linearly
with the size of our index.
Dennis Kubes
>
> On Dec 16, 2007 11:47 AM, Dennis Kubes <ku...@apache.org> wrote:
>> If you are talking about the clustering plugin, that is about grouping
>> (hopefully) related documents in the search results.
>>
>> Running the crawler and other nutch processes on multiple nodes is nutch
>> and hadoop running the map reduce paradigm. Moving final indexes and
>> database to local file systems for searching is simply best practices.
>>
>> Dennis Kubes
>>
>>
>> Bent Hugh wrote:
>>> I am a little confused. In the Nutch wiki there are chapters on
>>> clustering. I have never tried them though. So what is clustering
>>> about? Is it running the crawler on multiple nodes and creating
>>> crawldb on multiple nodes? And then finally merging all these on a
>>> local system and running the Nutch web-gui from that?
>>>
>>>
>>> On Dec 16, 2007 10:17 AM, Dennis Kubes <ku...@apache.org> wrote:
>>>> Technically you can. The speed for most search applications would be
>>>> unacceptable. Searching of indexes is best done on local files systems
>>>> for speed.
>>>>
>>>> Dennis Kubes
>>>>
>>>>
>>>> hzhong wrote:
>>>>> Hello,
>>>>>
>>>>> Why can't we search on the Hadoop DFS?
>>>>>
>>>>> Thanks
Re: DFS search
Posted by Bent Hugh <be...@gmail.com>.
So, whether we do the crawling by using multiple nodes or running it
on a single node with the regular bin/generate, bin/fetcher, series of
commands, finally we get a single crawldb on the local system where
Nutch is supposed to run. Am I right?
On Dec 16, 2007 11:47 AM, Dennis Kubes <ku...@apache.org> wrote:
> If you are talking about the clustering plugin, that is about grouping
> (hopefully) related documents in the search results.
>
> Running the crawler and other nutch processes on multiple nodes is nutch
> and hadoop running the map reduce paradigm. Moving final indexes and
> database to local file systems for searching is simply best practices.
>
> Dennis Kubes
>
>
> Bent Hugh wrote:
> > I am a little confused. In the Nutch wiki there are chapters on
> > clustering. I have never tried them though. So what is clustering
> > about? Is it running the crawler on multiple nodes and creating
> > crawldb on multiple nodes? And then finally merging all these on a
> > local system and running the Nutch web-gui from that?
> >
> >
> > On Dec 16, 2007 10:17 AM, Dennis Kubes <ku...@apache.org> wrote:
> >> Technically you can. The speed for most search applications would be
> >> unacceptable. Searching of indexes is best done on local files systems
> >> for speed.
> >>
> >> Dennis Kubes
> >>
> >>
> >> hzhong wrote:
> >>> Hello,
> >>>
> >>> Why can't we search on the Hadoop DFS?
> >>>
> >>> Thanks
>
Re: DFS search
Posted by Dennis Kubes <ku...@apache.org>.
If you are talking about the clustering plugin, that is about grouping
(hopefully) related documents in the search results.
Running the crawler and other nutch processes on multiple nodes is nutch
and hadoop running the map reduce paradigm. Moving final indexes and
database to local file systems for searching is simply best practices.
Dennis Kubes
Bent Hugh wrote:
> I am a little confused. In the Nutch wiki there are chapters on
> clustering. I have never tried them though. So what is clustering
> about? Is it running the crawler on multiple nodes and creating
> crawldb on multiple nodes? And then finally merging all these on a
> local system and running the Nutch web-gui from that?
>
>
> On Dec 16, 2007 10:17 AM, Dennis Kubes <ku...@apache.org> wrote:
>> Technically you can. The speed for most search applications would be
>> unacceptable. Searching of indexes is best done on local files systems
>> for speed.
>>
>> Dennis Kubes
>>
>>
>> hzhong wrote:
>>> Hello,
>>>
>>> Why can't we search on the Hadoop DFS?
>>>
>>> Thanks
Re: DFS search
Posted by Bent Hugh <be...@gmail.com>.
I am a little confused. In the Nutch wiki there are chapters on
clustering. I have never tried them though. So what is clustering
about? Is it running the crawler on multiple nodes and creating
crawldb on multiple nodes? And then finally merging all these on a
local system and running the Nutch web-gui from that?
On Dec 16, 2007 10:17 AM, Dennis Kubes <ku...@apache.org> wrote:
> Technically you can. The speed for most search applications would be
> unacceptable. Searching of indexes is best done on local files systems
> for speed.
>
> Dennis Kubes
>
>
> hzhong wrote:
> > Hello,
> >
> > Why can't we search on the Hadoop DFS?
> >
> > Thanks
>
Re: DFS search
Posted by Dennis Kubes <ku...@apache.org>.
Technically you can. The speed for most search applications would be
unacceptable. Searching of indexes is best done on local files systems
for speed.
Dennis Kubes
hzhong wrote:
> Hello,
>
> Why can't we search on the Hadoop DFS?
>
> Thanks