You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Raymond Balmès <ra...@gmail.com> on 2009/05/28 23:14:42 UTC
good documentation for nutch generate ?
Hi,
Is there a good place to find what the options of nutch generate are ?
-force : presumably forces a fetch against any date ?
-numFetchers : why and when do you want to create several fetchers,
presumably running in parallel (I'm only using one server)
-Ray-
Styling -- was Re: good documentation for nutch generate ?
Posted by "David M. Cole" <dm...@colegroup.com>.
At 12:50 PM +0100 5/29/09, Chris Beard wrote:
>1: styling the search page and results pages
At 2:08 PM +0200 5/29/09, Raymond Balmès wrote:
>1. styling is one, by the way I'd like to know if people have
>recommendations on styling tools for either nutch or solr
Well, I decided to use PHP send and receive the results from the
OpenSearch option of Nutch. I used a function I found on the
Internet, called get_tag_contents, by John W. List of
www.technotoad.com, to parse the results and then use standard PHP
techniques to sort the results (I prefer them in most-recent-first
order) and for styling.
You can see my Nutch implementation at http://www.newsinc.net/isearch2/
There are only about 1800 files indexed. Good search terms are proper
names such as "Dean Singleton", "John Sturm" or "Robert Decherd". You
won't be able to visit the actual resulting files, as they are
password-protected.
There are three major bugs in this implementation:
*There are some double-entries of files in the index; this is a
problem with the case insensitivity of the Macintosh file system
(where Nutch and Apache reside) versus the case sensitivity of Nutch.
I'm sure there's a way to make the crawler become case-insensitive,
but I just haven't figured out how to do it yet.
*I haven't yet provided a way for the end user to get to any results
that exceed 50, which I'm currently truncating. This is just a
feature I need to add and haven't had the time/inclination to do.
*Some results don't present the in-context, highlighted search terms.
I'm not sure why. In the field where the contextualized search term
should be, Nutch is returning the HTML title field, which I just pass
along.
I hope my humble little effort helps someone.
\dmc
--
*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
David M. Cole dmc@colegroup.com
Editor & Publisher, NewsInc. <http://newsinc.net> V: (650) 557-2993
Consultant: The Cole Group <http://colegroup.com/> F: (650) 475-8479
*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
Re: good documentation for nutch generate ?
Posted by Raymond Balmès <ra...@gmail.com>.
I think there are at least two reasons, so I'm considering to use Solr as
well but have not made up my mind yet.
1. styling is one, by the way I'd like to know if people have
recommendations on styling tools for either nutch or solr
2. search capabilities are much better with Solr. Nutch queries are rather
limited I found.
-Raymond-
2009/5/29 Chris Beard <c....@isc.keele.ac.uk>
> Hi all,
>
> I'm new to the maillinglist and new to nutch.
>
> I'm installing nutch on a university webserver and I've got it up and
> running fine. I'm indexing pages (it takes a while as there's 20,000 plus !)
> and the search results are ok.
>
> I was wondering whether anyone has any links or details in regards to:
>
> 1: styling the search page and results pages
> 2: using search narrowing eg, adding the ability to search within just one
> sub section of the site
> 3: does anyone use Solr as well as nutch? I've installed Solr but I'm not
> sure why I would need it if Nutch can do the job.
>
> Many thanks,
> Chris
>
>
>
>
> Raymond Balmès wrote:
>
>> Well not so helpfull to me. I'm using 1.0 and there is a new option
>> "-force"
>> not described there I didn't find any so far.
>>
>> Also I'd like to better understand the why & how to use the -numFetchers
>> option.
>>
>> -Ray-
>>
>> 2009/5/29 Jaime Martín <ja...@gmail.com>
>>
>>
>>
>>> Hi! excelent link! do you know if all commands behave the same in version
>>> 1.0? maybe some of them have changed or dissapeared?
>>>
>>> 2009/5/28 Malaviya, Sanjay X <Sa...@questdiagnostics.com>
>>>
>>>
>>>
>>>> This link has the details on command line option for nutch. Hope it
>>>>
>>>>
>>> helps.
>>>
>>>
>>>> http://wiki.apache.org/nutch/08CommandLineOptions
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Raymond Balmès [mailto:raymond.balmes@gmail.com]
>>>> Sent: Thursday, May 28, 2009 5:15 PM
>>>> To: nutch-user@lucene.apache.org
>>>> Subject: good documentation for nutch generate ?
>>>>
>>>> Hi,
>>>>
>>>> Is there a good place to find what the options of nutch generate are ?
>>>>
>>>> -force : presumably forces a fetch against any date ?
>>>> -numFetchers : why and when do you want to create several fetchers,
>>>> presumably running in parallel (I'm only using one server)
>>>>
>>>> -Ray-
>>>> ------------------------------------------
>>>> The contents of this message, together with any attachments, are
>>>> intended only for the use of the person(s) to which they are
>>>> addressed and may contain confidential and/or privileged
>>>> information. Further, any medical information herein is
>>>> confidential and protected by law. It is unlawful for unauthorized
>>>> persons to use, review, copy, disclose, or disseminate confidential
>>>> medical information. If you are not the intended recipient,
>>>> immediately advise the sender and delete this message and any
>>>> attachments. Any distribution, or copying of this message, or any
>>>> attachment, is prohibited.
>>>>
>>>>
>>>>
>>>
>>
>>
>
> --
> Chris Beard
> IT Adviser, Web Development
> Finance & IT
> Keele University
> Keele, Staffordshire, ST5 5BG
>
> t.01782 734223
>
>
Re: good documentation for nutch generate ?
Posted by Chris Beard <c....@isc.keele.ac.uk>.
Hi all,
I'm new to the maillinglist and new to nutch.
I'm installing nutch on a university webserver and I've got it up and
running fine. I'm indexing pages (it takes a while as there's 20,000
plus !) and the search results are ok.
I was wondering whether anyone has any links or details in regards to:
1: styling the search page and results pages
2: using search narrowing eg, adding the ability to search within just
one sub section of the site
3: does anyone use Solr as well as nutch? I've installed Solr but I'm
not sure why I would need it if Nutch can do the job.
Many thanks,
Chris
Raymond Balmès wrote:
> Well not so helpfull to me. I'm using 1.0 and there is a new option "-force"
> not described there I didn't find any so far.
>
> Also I'd like to better understand the why & how to use the -numFetchers
> option.
>
> -Ray-
>
> 2009/5/29 Jaime Martín <ja...@gmail.com>
>
>
>> Hi! excelent link! do you know if all commands behave the same in version
>> 1.0? maybe some of them have changed or dissapeared?
>>
>> 2009/5/28 Malaviya, Sanjay X <Sa...@questdiagnostics.com>
>>
>>
>>> This link has the details on command line option for nutch. Hope it
>>>
>> helps.
>>
>>> http://wiki.apache.org/nutch/08CommandLineOptions
>>>
>>>
>>> -----Original Message-----
>>> From: Raymond Balmès [mailto:raymond.balmes@gmail.com]
>>> Sent: Thursday, May 28, 2009 5:15 PM
>>> To: nutch-user@lucene.apache.org
>>> Subject: good documentation for nutch generate ?
>>>
>>> Hi,
>>>
>>> Is there a good place to find what the options of nutch generate are ?
>>>
>>> -force : presumably forces a fetch against any date ?
>>> -numFetchers : why and when do you want to create several fetchers,
>>> presumably running in parallel (I'm only using one server)
>>>
>>> -Ray-
>>> ------------------------------------------
>>> The contents of this message, together with any attachments, are
>>> intended only for the use of the person(s) to which they are
>>> addressed and may contain confidential and/or privileged
>>> information. Further, any medical information herein is
>>> confidential and protected by law. It is unlawful for unauthorized
>>> persons to use, review, copy, disclose, or disseminate confidential
>>> medical information. If you are not the intended recipient,
>>> immediately advise the sender and delete this message and any
>>> attachments. Any distribution, or copying of this message, or any
>>> attachment, is prohibited.
>>>
>>>
>
>
--
Chris Beard
IT Adviser, Web Development
Finance & IT
Keele University
Keele, Staffordshire, ST5 5BG
t.01782 734223
Re: good documentation for nutch generate ?
Posted by Raymond Balmès <ra...@gmail.com>.
Well not so helpfull to me. I'm using 1.0 and there is a new option "-force"
not described there I didn't find any so far.
Also I'd like to better understand the why & how to use the -numFetchers
option.
-Ray-
2009/5/29 Jaime Martín <ja...@gmail.com>
> Hi! excelent link! do you know if all commands behave the same in version
> 1.0? maybe some of them have changed or dissapeared?
>
> 2009/5/28 Malaviya, Sanjay X <Sa...@questdiagnostics.com>
>
> >
> > This link has the details on command line option for nutch. Hope it
> helps.
> >
> > http://wiki.apache.org/nutch/08CommandLineOptions
> >
> >
> > -----Original Message-----
> > From: Raymond Balmès [mailto:raymond.balmes@gmail.com]
> > Sent: Thursday, May 28, 2009 5:15 PM
> > To: nutch-user@lucene.apache.org
> > Subject: good documentation for nutch generate ?
> >
> > Hi,
> >
> > Is there a good place to find what the options of nutch generate are ?
> >
> > -force : presumably forces a fetch against any date ?
> > -numFetchers : why and when do you want to create several fetchers,
> > presumably running in parallel (I'm only using one server)
> >
> > -Ray-
> > ------------------------------------------
> > The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are
> > addressed and may contain confidential and/or privileged
> > information. Further, any medical information herein is
> > confidential and protected by law. It is unlawful for unauthorized
> > persons to use, review, copy, disclose, or disseminate confidential
> > medical information. If you are not the intended recipient,
> > immediately advise the sender and delete this message and any
> > attachments. Any distribution, or copying of this message, or any
> > attachment, is prohibited.
> >
>
Re: good documentation for nutch generate ?
Posted by Jaime Martín <ja...@gmail.com>.
Hi! excelent link! do you know if all commands behave the same in version
1.0? maybe some of them have changed or dissapeared?
2009/5/28 Malaviya, Sanjay X <Sa...@questdiagnostics.com>
>
> This link has the details on command line option for nutch. Hope it helps.
>
> http://wiki.apache.org/nutch/08CommandLineOptions
>
>
> -----Original Message-----
> From: Raymond Balmès [mailto:raymond.balmes@gmail.com]
> Sent: Thursday, May 28, 2009 5:15 PM
> To: nutch-user@lucene.apache.org
> Subject: good documentation for nutch generate ?
>
> Hi,
>
> Is there a good place to find what the options of nutch generate are ?
>
> -force : presumably forces a fetch against any date ?
> -numFetchers : why and when do you want to create several fetchers,
> presumably running in parallel (I'm only using one server)
>
> -Ray-
> ------------------------------------------
> The contents of this message, together with any attachments, are
> intended only for the use of the person(s) to which they are
> addressed and may contain confidential and/or privileged
> information. Further, any medical information herein is
> confidential and protected by law. It is unlawful for unauthorized
> persons to use, review, copy, disclose, or disseminate confidential
> medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any
> attachments. Any distribution, or copying of this message, or any
> attachment, is prohibited.
>
RE: good documentation for nutch generate ?
Posted by "Malaviya, Sanjay X" <Sa...@questdiagnostics.com>.
This link has the details on command line option for nutch. Hope it helps.
http://wiki.apache.org/nutch/08CommandLineOptions
-----Original Message-----
From: Raymond Balmès [mailto:raymond.balmes@gmail.com]
Sent: Thursday, May 28, 2009 5:15 PM
To: nutch-user@lucene.apache.org
Subject: good documentation for nutch generate ?
Hi,
Is there a good place to find what the options of nutch generate are ?
-force : presumably forces a fetch against any date ?
-numFetchers : why and when do you want to create several fetchers, presumably running in parallel (I'm only using one server)
-Ray-
------------------------------------------
The contents of this message, together with any attachments, are
intended only for the use of the person(s) to which they are
addressed and may contain confidential and/or privileged
information. Further, any medical information herein is
confidential and protected by law. It is unlawful for unauthorized
persons to use, review, copy, disclose, or disseminate confidential
medical information. If you are not the intended recipient,
immediately advise the sender and delete this message and any
attachments. Any distribution, or copying of this message, or any
attachment, is prohibited.