You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by blackwater dev <bl...@gmail.com> on 2005/07/25 02:09:12 UTC
where to pull indexed files?
Forgive me if this is a dumb question but how can I tell nutch where
to pull the files for the index? I am just getting exceptions now
when I do a search. How does the code under tomcat know where I pull
the files from the crawl?
Thanks!
Re: hosting multiple independent searches - was: where to pull indexed files?
Posted by Rob Pettengill <ro...@earthlink.net>.
Doing multiple independent indexes with a single bean and
differentiating based on field metadata seems very awkward. In many
cases you would want a different appearance for the search form for
each specialized search.
If you want to host multiple independent searches of different index
sets, with the minimum resources, I would expect the following
approach to work:
Create multiple search.jsp pages each of which instantiates is own
NutchBean which is initailized with a different location for the
segments directory.
Will this work or are there threading issues that force the use of
multiple jsp containers?
;rob
On 2005, Jul 24, at 8:40 PM, blackwater dev wrote:
> So there is no way to set up different databases? I mean if I crawl a
> series of web pages and put them one place then crawl a series of job
> webplaces and put them in another folder, how can I let the web user
> click a button and search either the general web pages or the specific
> job web sites? To do this I would somehow be able to tell tomcat
> where to get the folder from within the jsp.
>
> Thanks!
>
> On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
>
>>
>> I think tomcat doesn't do indexing at all;
>>
>> just follow the tutorial under doc in Nutch home;
>>
>> after getting your segments, starting tomcat at the
>> same level will give the tomcat direction where to
>> find SE indexed data;
>>
>> Hope that helps,
>>
>> Michael,
>>
>> --- blackwater dev <bl...@gmail.com> wrote:
>>
>>
>>> Forgive me if this is a dumb question but how can I
>>> tell nutch where
>>> to pull the files for the index? I am just getting
>>> exceptions now
>>> when I do a search. How does the code under tomcat
>>> know where I pull
>>> the files from the crawl?
>>>
>>>
>>> Thanks!
>>>
>>>
>>
>>
>> __________________________________________________
>> Do You Yahoo!?
>> Tired of spam? Yahoo! Mail has the best spam protection around
>> http://mail.yahoo.com
>>
>
Re: where to pull indexed files?
Posted by Jack Tang <hi...@gmail.com>.
Hi
Please search the maillist.
I remembered one thread talked about dynamically delete index some
days ago. Maybe it helps. Index-hot-swap is not supported in Nutch by
default now.
Regards
/Jack
On 7/25/05, blackwater dev <bl...@gmail.com> wrote:
> So there is no way to set up different databases? I mean if I crawl a
> series of web pages and put them one place then crawl a series of job
> webplaces and put them in another folder, how can I let the web user
> click a button and search either the general web pages or the specific
> job web sites? To do this I would somehow be able to tell tomcat
> where to get the folder from within the jsp.
>
> Thanks!
>
> On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
> >
> > I think tomcat doesn't do indexing at all;
> >
> > just follow the tutorial under doc in Nutch home;
> >
> > after getting your segments, starting tomcat at the
> > same level will give the tomcat direction where to
> > find SE indexed data;
> >
> > Hope that helps,
> >
> > Michael,
> >
> > --- blackwater dev <bl...@gmail.com> wrote:
> >
> > > Forgive me if this is a dumb question but how can I
> > > tell nutch where
> > > to pull the files for the index? I am just getting
> > > exceptions now
> > > when I do a search. How does the code under tomcat
> > > know where I pull
> > > the files from the crawl?
> > >
> > >
> > > Thanks!
> > >
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam? Yahoo! Mail has the best spam protection around
> > http://mail.yahoo.com
> >
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
Re: crawling Doc and Pdf
Posted by Jack Tang <hi...@gmail.com>.
Michael
Error logs helps. pls post them on the email. Thanks
/Jack
On 7/27/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
> hi,
>
> I checked my log file, found crawler generates error
> when met a page with word file and pdf file inside.
>
> Any configuration file I have to change to let crawler
> could fetch linkage with word and pdf files?
>
> thanks,
>
> Michael,
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
segments, whats used when searching?
Posted by EM <em...@cpuedge.com>.
I'm trying to grasp something here, I need a quick confirmation about the
following, a yes/no would suffice:
When searching and generating summaries, tomcat uses only:
1. <segment>/index
2. <segment>/parse_text
When retrieving the "cached" copy of the document, tomcat uses:
1. <segment>/parse_data
Are <segment>/fetcher and <segment>/context used during the searching stage?
Regards,
E.
crawling Doc and Pdf
Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
hi,
I checked my log file, found crawler generates error
when met a page with word file and pdf file inside.
Any configuration file I have to change to let crawler
could fetch linkage with word and pdf files?
thanks,
Michael,
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: where to pull indexed files?
Posted by Howie Wang <ho...@hotmail.com>.
To answer your first question, change the conf/nutch-default.xml
file. The parameter in question is searcher.dir.
I'm not sure how to search two different indexes using the same
instance of Tomcat. You could have multiple instances of Tomcat
running and map one to http://general.mycompany.com and the
other to http://jobs.mycompany.com. That might work.
Or you could just index all the job pages with some Field.
Something like category:jobs. Then create a QueryFilter plugin
that subclasses RawFieldQueryFilter to search for your "category"
field. When you do a job search, you just put "category:jobs"
in the search string.
Howie
>So there is no way to set up different databases? I mean if I crawl a
>series of web pages and put them one place then crawl a series of job
>webplaces and put them in another folder, how can I let the web user
>click a button and search either the general web pages or the specific
>job web sites? To do this I would somehow be able to tell tomcat
>where to get the folder from within the jsp.
>
>Thanks!
>
>On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
> >
> > I think tomcat doesn't do indexing at all;
> >
> > just follow the tutorial under doc in Nutch home;
> >
> > after getting your segments, starting tomcat at the
> > same level will give the tomcat direction where to
> > find SE indexed data;
> >
> > Hope that helps,
> >
> > Michael,
> >
> > --- blackwater dev <bl...@gmail.com> wrote:
> >
> > > Forgive me if this is a dumb question but how can I
> > > tell nutch where
> > > to pull the files for the index? I am just getting
> > > exceptions now
> > > when I do a search. How does the code under tomcat
> > > know where I pull
> > > the files from the crawl?
> > >
> > >
> > > Thanks!
> > >
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam? Yahoo! Mail has the best spam protection around
> > http://mail.yahoo.com
> >
Re: where to pull indexed files?
Posted by blackwater dev <bl...@gmail.com>.
So there is no way to set up different databases? I mean if I crawl a
series of web pages and put them one place then crawl a series of job
webplaces and put them in another folder, how can I let the web user
click a button and search either the general web pages or the specific
job web sites? To do this I would somehow be able to tell tomcat
where to get the folder from within the jsp.
Thanks!
On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
>
> I think tomcat doesn't do indexing at all;
>
> just follow the tutorial under doc in Nutch home;
>
> after getting your segments, starting tomcat at the
> same level will give the tomcat direction where to
> find SE indexed data;
>
> Hope that helps,
>
> Michael,
>
> --- blackwater dev <bl...@gmail.com> wrote:
>
> > Forgive me if this is a dumb question but how can I
> > tell nutch where
> > to pull the files for the index? I am just getting
> > exceptions now
> > when I do a search. How does the code under tomcat
> > know where I pull
> > the files from the crawl?
> >
> >
> > Thanks!
> >
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
Re: where to pull indexed files?
Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
I think tomcat doesn't do indexing at all;
just follow the tutorial under doc in Nutch home;
after getting your segments, starting tomcat at the
same level will give the tomcat direction where to
find SE indexed data;
Hope that helps,
Michael,
--- blackwater dev <bl...@gmail.com> wrote:
> Forgive me if this is a dumb question but how can I
> tell nutch where
> to pull the files for the index? I am just getting
> exceptions now
> when I do a search. How does the code under tomcat
> know where I pull
> the files from the crawl?
>
>
> Thanks!
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com