You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Peter Gaines <pg...@deveire.com> on 2013/06/12 13:56:40 UTC
Running Nutch standalone (without Solr)
Hi There,
Is it possible to run Nutch as a standalone crawler without integration with Solr?
I need to do this in order to do a performance comparison of it’s raw crawling functionality.
It seems like it may be possible using the bin/nutch crawl command but this is now deprecated.
Is there any way to do this using bin/crawl without commenting out chunks of it?
Regards,
Peter
Re: Installing Nutch1.6 on Windows7
Posted by Andrea Lanzoni <a....@alice.it>.
Thanks Tejas for your reply. I'll follow your advice and install Ubuntu
and then Tomcat, Nutch and Solr.
Il 13/06/2013 20:35, Tejas Patil ha scritto:
> On Thu, Jun 13, 2013 at 4:50 AM, Andrea Lanzoni <a....@alice.it> wrote:
>
>> Hi Lewis and thanks for your reply. I will try to be as much detailed as
>> possible to allow you to understand the shortcomings I incurred during
>> installation of Nutch and Solr on my PC, which is a 64 bit running Windows
>> 7.
>>
>> I canvassed wiki apache and found this link:
>>
>> http://wiki.apache.org/nutch/**FabioGiavazzi/**
>> HowtoGettingNutchRunningonWind**ows?highlight=%28download%29|%**
>> 28nutch%29|%28for%29|%**28windows%29|%287%29<http://wiki.apache.org/nutch/FabioGiavazzi/HowtoGettingNutchRunningonWindows?highlight=%28download%29%7C%28nutch%29%7C%28for%29%7C%28windows%29%7C%287%29><
>> http://wiki.apache.org/nutch/**FabioGiavazzi/**
>> HowtoGettingNutchRunningonWind**ows?highlight=%28download%29%**
>> 7C%28nutch%29%7C%28for%29%7C%**28windows%29%7C%287%29<http://wiki.apache.org/nutch/FabioGiavazzi/HowtoGettingNutchRunningonWindows?highlight=%28download%29%7C%28nutch%29%7C%28for%29%7C%28windows%29%7C%287%29>
>> Even though it explains Nutch 1.2 installation on Windows 7 I thought it
>> might fit for Nutch 1.6 as well.
>>
>> Everything went smooth until step 4 of the presentation. Step 4 puzzled me
>> and decided to skip it.
>>
>> I went on and made the changes in XML as stated in ensuing steps.
>>
>> Problem arouse at Step 7 where it reads: Go to *nutch*-1.2\conf\ and edit
>> the file crawl-urlfilter.txt
>>
> Those steps are old. Skip step #7 and proceed.
>>
>> I didn't find it in my Nutch directories and got into a cul de sac.
>>
> PS: I just learned what "cul de sac" means !! thanks for adding to my vocab
> :)
>
>> Presently in my PC I have installed as follows:
>>
>> C:\Users\Andrea\Documents\**apache-tomcat-7.0.40\apache-**tomcat-7.0.40
>>
>> C:\cygwin\home\apache-nutch-1.**6-bin\apache-nutch-1.6
>>
>> C:\cygwin\home\solr-4.2.0\**solr-4.2.0
>>
>>
>> I am also asking your advice on this: is it only my laziness/stubborness
>> to continue to install them on Windows, in other terms would you suggest to
>> drop Windows and install Ubuntu and restart the procedure in the new op.
>> system?
>> I happened only once to come across Ubuntu, does it permit to host
>> simultaneously and use on the same PC Windows and Ubuntu itself?
>>
>>
> You can have Ubuntu + Windows installed on the same m/c. See this:
> http://www.ubuntu.com/download/desktop/install-ubuntu-with-windows
>
> Its super easy to setup Ubuntu with that and its worth of spending an hour
> or two.
>
>> Thanks for your welcome opinion.
>> Andrea
>>
>>
>>
>>
>> Il 13/06/2013 01:49, Lewis John Mcgibbney ha scritto:
>>
>>> Hi Andrea,
>>> Please describe the problem here. There is an absence of any detail about
>>> what is wrong here.
>>> Thanks
>>>
>>> On Wednesday, June 12, 2013, Andrea Lanzoni <a....@alice.it> wrote:
>>>
>>>> Hi everyone, I am a newcomer to Nutch and Solr and, after studying
>>>>
>>> literature available on web, I tried to install them on _Windows 7_.
>>>
>>>> I have not been able to match the few instructions on the wikiapache site
>>>>
>>> nor I could find a guide updated to Nutch 1.6 but only for older versions
>>>
>>>> I tried by following old versions guides on the web but never succeeded
>>>>
>>> in the installation, often because of differences from what the guide read
>>> and what I saw on screen.
>>>
>>>> I followed the steps by installing:
>>>> - Tomcat
>>>> - Java jdk 7
>>>> - Cygwin, Nutch 1.6 and Solr 4
>>>>
>>>> Everything went apparently smooth and I copied Nutch and Solr in:
>>>> C:\cygwin\home\apache-nutch-1.**6-bin
>>>> and
>>>> C:\cygwin\home\solr-4.2.0\**solr-4.2.0
>>>>
>>>> Whilst the two folders: jdk1.7.0_21 and jre7, are within the Java folder
>>>>
>>> in Programs directory
>>>
>>>> I apologize for my dumbness but I couldn't find how to manage it. If
>>>>
>>> somebody has a clear and detailed step by step pattern to follow for
>>> installing Nutch 1.6 and Solr 4 I would be very grateful.
>>>
>>>> Thanks in advance.
>>>> Andrea Lanzoni
>>>>
>>>>
Re: Installing Nutch1.6 on Windows7
Posted by Tejas Patil <te...@gmail.com>.
On Thu, Jun 13, 2013 at 4:50 AM, Andrea Lanzoni <a....@alice.it> wrote:
> Hi Lewis and thanks for your reply. I will try to be as much detailed as
> possible to allow you to understand the shortcomings I incurred during
> installation of Nutch and Solr on my PC, which is a 64 bit running Windows
> 7.
>
> I canvassed wiki apache and found this link:
>
> http://wiki.apache.org/nutch/**FabioGiavazzi/**
> HowtoGettingNutchRunningonWind**ows?highlight=%28download%29|%**
> 28nutch%29|%28for%29|%**28windows%29|%287%29<http://wiki.apache.org/nutch/FabioGiavazzi/HowtoGettingNutchRunningonWindows?highlight=%28download%29%7C%28nutch%29%7C%28for%29%7C%28windows%29%7C%287%29><
> http://wiki.apache.org/nutch/**FabioGiavazzi/**
> HowtoGettingNutchRunningonWind**ows?highlight=%28download%29%**
> 7C%28nutch%29%7C%28for%29%7C%**28windows%29%7C%287%29<http://wiki.apache.org/nutch/FabioGiavazzi/HowtoGettingNutchRunningonWindows?highlight=%28download%29%7C%28nutch%29%7C%28for%29%7C%28windows%29%7C%287%29>
> >
>
> Even though it explains Nutch 1.2 installation on Windows 7 I thought it
> might fit for Nutch 1.6 as well.
>
> Everything went smooth until step 4 of the presentation. Step 4 puzzled me
> and decided to skip it.
>
> I went on and made the changes in XML as stated in ensuing steps.
>
> Problem arouse at Step 7 where it reads: Go to *nutch*-1.2\conf\ and edit
> the file crawl-urlfilter.txt
>
Those steps are old. Skip step #7 and proceed.
>
>
> I didn't find it in my Nutch directories and got into a cul de sac.
>
PS: I just learned what "cul de sac" means !! thanks for adding to my vocab
:)
>
> Presently in my PC I have installed as follows:
>
> C:\Users\Andrea\Documents\**apache-tomcat-7.0.40\apache-**tomcat-7.0.40
>
> C:\cygwin\home\apache-nutch-1.**6-bin\apache-nutch-1.6
>
> C:\cygwin\home\solr-4.2.0\**solr-4.2.0
>
>
> I am also asking your advice on this: is it only my laziness/stubborness
> to continue to install them on Windows, in other terms would you suggest to
> drop Windows and install Ubuntu and restart the procedure in the new op.
> system?
> I happened only once to come across Ubuntu, does it permit to host
> simultaneously and use on the same PC Windows and Ubuntu itself?
>
>
You can have Ubuntu + Windows installed on the same m/c. See this:
http://www.ubuntu.com/download/desktop/install-ubuntu-with-windows
Its super easy to setup Ubuntu with that and its worth of spending an hour
or two.
>
> Thanks for your welcome opinion.
> Andrea
>
>
>
>
> Il 13/06/2013 01:49, Lewis John Mcgibbney ha scritto:
>
>> Hi Andrea,
>> Please describe the problem here. There is an absence of any detail about
>> what is wrong here.
>> Thanks
>>
>> On Wednesday, June 12, 2013, Andrea Lanzoni <a....@alice.it> wrote:
>>
>>> Hi everyone, I am a newcomer to Nutch and Solr and, after studying
>>>
>> literature available on web, I tried to install them on _Windows 7_.
>>
>>> I have not been able to match the few instructions on the wikiapache site
>>>
>> nor I could find a guide updated to Nutch 1.6 but only for older versions
>>
>>> I tried by following old versions guides on the web but never succeeded
>>>
>> in the installation, often because of differences from what the guide read
>> and what I saw on screen.
>>
>>> I followed the steps by installing:
>>> - Tomcat
>>> - Java jdk 7
>>> - Cygwin, Nutch 1.6 and Solr 4
>>>
>>> Everything went apparently smooth and I copied Nutch and Solr in:
>>> C:\cygwin\home\apache-nutch-1.**6-bin
>>> and
>>> C:\cygwin\home\solr-4.2.0\**solr-4.2.0
>>>
>>> Whilst the two folders: jdk1.7.0_21 and jre7, are within the Java folder
>>>
>> in Programs directory
>>
>>> I apologize for my dumbness but I couldn't find how to manage it. If
>>>
>> somebody has a clear and detailed step by step pattern to follow for
>> installing Nutch 1.6 and Solr 4 I would be very grateful.
>>
>>> Thanks in advance.
>>> Andrea Lanzoni
>>>
>>>
>
Re: Installing Nutch1.6 on Windows7
Posted by Andrea Lanzoni <a....@alice.it>.
Hi Lewis and thanks for your reply. I will try to be as much detailed as
possible to allow you to understand the shortcomings I incurred during
installation of Nutch and Solr on my PC, which is a 64 bit running
Windows 7.
I canvassed wiki apache and found this link:
http://wiki.apache.org/nutch/FabioGiavazzi/HowtoGettingNutchRunningonWindows?highlight=%28download%29|%28nutch%29|%28for%29|%28windows%29|%287%29
<http://wiki.apache.org/nutch/FabioGiavazzi/HowtoGettingNutchRunningonWindows?highlight=%28download%29%7C%28nutch%29%7C%28for%29%7C%28windows%29%7C%287%29>
Even though it explains Nutch 1.2 installation on Windows 7 I thought it
might fit for Nutch 1.6 as well.
Everything went smooth until step 4 of the presentation. Step 4 puzzled
me and decided to skip it.
I went on and made the changes in XML as stated in ensuing steps.
Problem arouse at Step 7 where it reads: Go to *nutch*-1.2\conf\ and
edit the file crawl-urlfilter.txt
I didn't find it in my Nutch directories and got into a cul de sac.
Presently in my PC I have installed as follows:
C:\Users\Andrea\Documents\apache-tomcat-7.0.40\apache-tomcat-7.0.40
C:\cygwin\home\apache-nutch-1.6-bin\apache-nutch-1.6
C:\cygwin\home\solr-4.2.0\solr-4.2.0
I am also asking your advice on this: is it only my
laziness/stubborness to continue to install them on Windows, in other
terms would you suggest to drop Windows and install Ubuntu and restart
the procedure in the new op. system?
I happened only once to come across Ubuntu, does it permit to host
simultaneously and use on the same PC Windows and Ubuntu itself?
Thanks for your welcome opinion.
Andrea
Il 13/06/2013 01:49, Lewis John Mcgibbney ha scritto:
> Hi Andrea,
> Please describe the problem here. There is an absence of any detail about
> what is wrong here.
> Thanks
>
> On Wednesday, June 12, 2013, Andrea Lanzoni <a....@alice.it> wrote:
>> Hi everyone, I am a newcomer to Nutch and Solr and, after studying
> literature available on web, I tried to install them on _Windows 7_.
>> I have not been able to match the few instructions on the wikiapache site
> nor I could find a guide updated to Nutch 1.6 but only for older versions
>> I tried by following old versions guides on the web but never succeeded
> in the installation, often because of differences from what the guide read
> and what I saw on screen.
>> I followed the steps by installing:
>> - Tomcat
>> - Java jdk 7
>> - Cygwin, Nutch 1.6 and Solr 4
>>
>> Everything went apparently smooth and I copied Nutch and Solr in:
>> C:\cygwin\home\apache-nutch-1.6-bin
>> and
>> C:\cygwin\home\solr-4.2.0\solr-4.2.0
>>
>> Whilst the two folders: jdk1.7.0_21 and jre7, are within the Java folder
> in Programs directory
>> I apologize for my dumbness but I couldn't find how to manage it. If
> somebody has a clear and detailed step by step pattern to follow for
> installing Nutch 1.6 and Solr 4 I would be very grateful.
>> Thanks in advance.
>> Andrea Lanzoni
>>
Re: Installing Nutch1.6 on Windows7
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Andrea,
Please describe the problem here. There is an absence of any detail about
what is wrong here.
Thanks
On Wednesday, June 12, 2013, Andrea Lanzoni <a....@alice.it> wrote:
> Hi everyone, I am a newcomer to Nutch and Solr and, after studying
literature available on web, I tried to install them on _Windows 7_.
>
> I have not been able to match the few instructions on the wikiapache site
nor I could find a guide updated to Nutch 1.6 but only for older versions
> I tried by following old versions guides on the web but never succeeded
in the installation, often because of differences from what the guide read
and what I saw on screen.
>
> I followed the steps by installing:
> - Tomcat
> - Java jdk 7
> - Cygwin, Nutch 1.6 and Solr 4
>
> Everything went apparently smooth and I copied Nutch and Solr in:
> C:\cygwin\home\apache-nutch-1.6-bin
> and
> C:\cygwin\home\solr-4.2.0\solr-4.2.0
>
> Whilst the two folders: jdk1.7.0_21 and jre7, are within the Java folder
in Programs directory
>
> I apologize for my dumbness but I couldn't find how to manage it. If
somebody has a clear and detailed step by step pattern to follow for
installing Nutch 1.6 and Solr 4 I would be very grateful.
> Thanks in advance.
> Andrea Lanzoni
>
--
*Lewis*
Installing Nutch1.6 on Windows7
Posted by Andrea Lanzoni <a....@alice.it>.
Hi everyone, I am a newcomer to Nutch and Solr and, after studying
literature available on web, I tried to install them on _Windows 7_.
I have not been able to match the few instructions on the wikiapache
site nor I could find a guide updated to Nutch 1.6 but only for older
versions
I tried by following old versions guides on the web but never succeeded
in the installation, often because of differences from what the guide
read and what I saw on screen.
I followed the steps by installing:
- Tomcat
- Java jdk 7
- Cygwin, Nutch 1.6 and Solr 4
Everything went apparently smooth and I copied Nutch and Solr in:
C:\cygwin\home\apache-nutch-1.6-bin
and
C:\cygwin\home\solr-4.2.0\solr-4.2.0
Whilst the two folders: jdk1.7.0_21 and jre7, are within the Java folder
in Programs directory
I apologize for my dumbness but I couldn't find how to manage it. If
somebody has a clear and detailed step by step pattern to follow for
installing Nutch 1.6 and Solr 4 I would be very grateful.
Thanks in advance.
Andrea Lanzoni
RE: Running Nutch standalone (without Solr)
Posted by Markus Jelsma <ma...@openindex.io>.
Hi,
Sure, you don't need to index the data and can use the individual commands or the new bin/crawl script.
Cheers
-----Original message-----
> From:Peter Gaines <pg...@deveire.com>
> Sent: Wed 12-Jun-2013 13:57
> To: user@nutch.apache.org
> Subject: Running Nutch standalone (without Solr)
>
> Hi There,
>
> Is it possible to run Nutch as a standalone crawler without integration with Solr?
> I need to do this in order to do a performance comparison of it’s raw crawling functionality.
>
> It seems like it may be possible using the bin/nutch crawl command but this is now deprecated.
> Is there any way to do this using bin/crawl without commenting out chunks of it?
>
> Regards,
> Peter
>
>
Re: Running Nutch standalone (without Solr)
Posted by "H. Coskun Gunduz" <co...@agmlab.com>.
Hi Peter,
Yes, it's possible.
You'll need a data store (my personal recommendation is HBase).
Regarding on the Nutch version you use, you can follow these tutorials:
Nutch 1.x: http://wiki.apache.org/nutch/NutchTutorial
Nutch 2: http://wiki.apache.org/nutch/Nutch2Tutorial
Happy crawling.
coskun...
On 06/12/2013 02:56 PM, Peter Gaines wrote:
> Hi There,
>
> Is it possible to run Nutch as a standalone crawler without integration with Solr?
> I need to do this in order to do a performance comparison of it’s raw crawling functionality.
>
> It seems like it may be possible using the bin/nutch crawl command but this is now deprecated.
> Is there any way to do this using bin/crawl without commenting out chunks of it?
>
> Regards,
> Peter
>
>