You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by blackwater dev <bl...@gmail.com> on 2005/07/25 02:09:12 UTC

where to pull indexed files?

Forgive me if this is a dumb question but how can I tell nutch where
to pull the files for the index?  I am just getting exceptions now
when I do a search.  How does the code under tomcat know where I pull
the files from the crawl?


Thanks!

Re: hosting multiple independent searches - was: where to pull indexed files?

Posted by Rob Pettengill <ro...@earthlink.net>.
Doing multiple independent indexes with a single bean and  
differentiating based on field metadata seems very awkward.  In many  
cases you would want a different appearance for the search form for  
each specialized search.

If you want to host multiple independent searches of different index  
sets, with the minimum resources, I would expect the following  
approach to work:

Create multiple search.jsp pages each of which instantiates is own  
NutchBean which is initailized with a different location for the  
segments directory.

Will this work or are there threading issues that force the use of  
multiple jsp containers?

;rob


On 2005, Jul 24, at 8:40 PM, blackwater dev wrote:

> So there is no way to set up different databases?  I mean if I crawl a
> series of web pages and put them one place then crawl a series of job
> webplaces and put them in another folder, how can I let the web user
> click a button and search either the general web pages or the specific
> job web sites?  To do this I would somehow be able to tell tomcat
> where to get the folder from within the jsp.
>
> Thanks!
>
> On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
>
>>
>> I think tomcat doesn't do indexing at all;
>>
>> just follow the tutorial under doc in Nutch home;
>>
>> after getting your segments, starting tomcat at the
>> same level will give the tomcat direction where to
>> find  SE indexed data;
>>
>> Hope that helps,
>>
>> Michael,
>>
>> --- blackwater dev <bl...@gmail.com> wrote:
>>
>>
>>> Forgive me if this is a dumb question but how can I
>>> tell nutch where
>>> to pull the files for the index?  I am just getting
>>> exceptions now
>>> when I do a search.  How does the code under tomcat
>>> know where I pull
>>> the files from the crawl?
>>>
>>>
>>> Thanks!
>>>
>>>
>>
>>
>> __________________________________________________
>> Do You Yahoo!?
>> Tired of spam?  Yahoo! Mail has the best spam protection around
>> http://mail.yahoo.com
>>
>


Re: where to pull indexed files?

Posted by Jack Tang <hi...@gmail.com>.
Hi 

Please search the maillist.
I remembered one thread talked about dynamically delete index some
days ago. Maybe it helps. Index-hot-swap is not supported in Nutch by
default now.

Regards
/Jack

On 7/25/05, blackwater dev <bl...@gmail.com> wrote:
> So there is no way to set up different databases?  I mean if I crawl a
> series of web pages and put them one place then crawl a series of job
> webplaces and put them in another folder, how can I let the web user
> click a button and search either the general web pages or the specific
> job web sites?  To do this I would somehow be able to tell tomcat
> where to get the folder from within the jsp.
> 
> Thanks!
> 
> On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
> >
> > I think tomcat doesn't do indexing at all;
> >
> > just follow the tutorial under doc in Nutch home;
> >
> > after getting your segments, starting tomcat at the
> > same level will give the tomcat direction where to
> > find  SE indexed data;
> >
> > Hope that helps,
> >
> > Michael,
> >
> > --- blackwater dev <bl...@gmail.com> wrote:
> >
> > > Forgive me if this is a dumb question but how can I
> > > tell nutch where
> > > to pull the files for the index?  I am just getting
> > > exceptions now
> > > when I do a search.  How does the code under tomcat
> > > know where I pull
> > > the files from the crawl?
> > >
> > >
> > > Thanks!
> > >
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam protection around
> > http://mail.yahoo.com
> >
> 


-- 
Keep Discovering ... ...
http://www.jroller.com/page/jmars

Re: crawling Doc and Pdf

Posted by Jack Tang <hi...@gmail.com>.
Michael

Error logs helps. pls post them on the email. Thanks

/Jack

On 7/27/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
> hi,
> 
> I checked my log file, found crawler generates error
> when met a page with word file and pdf file inside.
> 
> Any configuration file I have to change to let crawler
> could fetch linkage with word and pdf files?
> 
> thanks,
> 
> Michael,
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
> 


-- 
Keep Discovering ... ...
http://www.jroller.com/page/jmars

segments, whats used when searching?

Posted by EM <em...@cpuedge.com>.
I'm trying to grasp something here, I need a quick confirmation about the
following, a yes/no would suffice:

When searching and generating summaries, tomcat uses only:
1. <segment>/index
2. <segment>/parse_text

When retrieving the "cached" copy of the document, tomcat uses:
1. <segment>/parse_data

Are <segment>/fetcher and <segment>/context used during the searching stage?


Regards,
E.


crawling Doc and Pdf

Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
hi,

I checked my log file, found crawler generates error
when met a page with word file and pdf file inside.

Any configuration file I have to change to let crawler
could fetch linkage with word and pdf files?

thanks,

Michael,

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: where to pull indexed files?

Posted by Howie Wang <ho...@hotmail.com>.
To answer your first question, change the conf/nutch-default.xml
file. The parameter in question is searcher.dir.

I'm not sure how to search two different indexes using the same
instance of Tomcat. You could have multiple instances of Tomcat
running and map one to http://general.mycompany.com and the
other to http://jobs.mycompany.com. That might work.

Or you could just index all the job pages with some Field.
Something like category:jobs.  Then create a QueryFilter plugin
that subclasses RawFieldQueryFilter to search for your "category"
field. When you do a job search, you just put "category:jobs"
in the search string.

Howie


>So there is no way to set up different databases?  I mean if I crawl a
>series of web pages and put them one place then crawl a series of job
>webplaces and put them in another folder, how can I let the web user
>click a button and search either the general web pages or the specific
>job web sites?  To do this I would somehow be able to tell tomcat
>where to get the folder from within the jsp.
>
>Thanks!
>
>On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
> >
> > I think tomcat doesn't do indexing at all;
> >
> > just follow the tutorial under doc in Nutch home;
> >
> > after getting your segments, starting tomcat at the
> > same level will give the tomcat direction where to
> > find  SE indexed data;
> >
> > Hope that helps,
> >
> > Michael,
> >
> > --- blackwater dev <bl...@gmail.com> wrote:
> >
> > > Forgive me if this is a dumb question but how can I
> > > tell nutch where
> > > to pull the files for the index?  I am just getting
> > > exceptions now
> > > when I do a search.  How does the code under tomcat
> > > know where I pull
> > > the files from the crawl?
> > >
> > >
> > > Thanks!
> > >
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam protection around
> > http://mail.yahoo.com
> >



Re: where to pull indexed files?

Posted by blackwater dev <bl...@gmail.com>.
So there is no way to set up different databases?  I mean if I crawl a
series of web pages and put them one place then crawl a series of job
webplaces and put them in another folder, how can I let the web user
click a button and search either the general web pages or the specific
job web sites?  To do this I would somehow be able to tell tomcat
where to get the folder from within the jsp.

Thanks!

On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
> 
> I think tomcat doesn't do indexing at all;
> 
> just follow the tutorial under doc in Nutch home;
> 
> after getting your segments, starting tomcat at the
> same level will give the tomcat direction where to
> find  SE indexed data;
> 
> Hope that helps,
> 
> Michael,
> 
> --- blackwater dev <bl...@gmail.com> wrote:
> 
> > Forgive me if this is a dumb question but how can I
> > tell nutch where
> > to pull the files for the index?  I am just getting
> > exceptions now
> > when I do a search.  How does the code under tomcat
> > know where I pull
> > the files from the crawl?
> >
> >
> > Thanks!
> >
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>

Re: where to pull indexed files?

Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
I think tomcat doesn't do indexing at all;

just follow the tutorial under doc in Nutch home;

after getting your segments, starting tomcat at the
same level will give the tomcat direction where to
find  SE indexed data;

Hope that helps,

Michael,

--- blackwater dev <bl...@gmail.com> wrote:

> Forgive me if this is a dumb question but how can I
> tell nutch where
> to pull the files for the index?  I am just getting
> exceptions now
> when I do a search.  How does the code under tomcat
> know where I pull
> the files from the crawl?
> 
> 
> Thanks!
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com