You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by abhayd <aj...@hotmail.com> on 2011/05/14 15:29:05 UTC

Re: [ANNOUNCE] Web Crawler

hi Dominique,

I am looking for a crawler to feed solr index. After looking at various
posts i have settled down on two
Nutch and crawl anywhere.

I dont see any activities on Nutch wiki so wondering if its not being
developed anymore. But most forums say Nutch is standard for solr.

Crawl Anywhere looks solid. Any way for users like me to decide which one we
should go for Nutch or Crawl Anywehre?


Concern with crawl anywhere is it supports solr 1.3 index not the latest
version

Any help on the is really appreciated 

--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607833p2937762.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: [ANNOUNCE] Web Crawler

Posted by Dominique Bejean <do...@eolya.fr>.
Hi,

Sorry for the delay, but I haven't been checking the mailing list for a 
long time.

Crawl-anywhere includes 3 piece of software : a crawler, a pipeline and 
a solr indexer.

There is a default Solr schema used by Crawl-anywhere, tested with Solr 
1.4.1 and Solr 3.1.0.

But, you can configure the pipeline stage responsible for mapping 
crawled data to Solr field. IN the absolute, you can use any schema with 
any Solr version.

Regards

Dominique



Le 14/05/11 15:29, abhayd a écrit :
> hi Dominique,
>
> I am looking for a crawler to feed solr index. After looking at various
> posts i have settled down on two
> Nutch and crawl anywhere.
>
> I dont see any activities on Nutch wiki so wondering if its not being
> developed anymore. But most forums say Nutch is standard for solr.
>
> Crawl Anywhere looks solid. Any way for users like me to decide which one we
> should go for Nutch or Crawl Anywehre?
>
>
> Concern with crawl anywhere is it supports solr 1.3 index not the latest
> version
>
> Any help on the is really appreciated
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607833p2937762.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

RE: [ANNOUNCE] Web Crawler

Posted by ka...@nokia.com.
You might want to look at ManifoldCF also.

Karl

-----Original Message-----
From: ext abhayd [mailto:ajdabholkar@hotmail.com] 
Sent: Saturday, May 14, 2011 9:29 AM
To: java-user@lucene.apache.org
Subject: Re: [ANNOUNCE] Web Crawler

hi Dominique,

I am looking for a crawler to feed solr index. After looking at various
posts i have settled down on two
Nutch and crawl anywhere.

I dont see any activities on Nutch wiki so wondering if its not being
developed anymore. But most forums say Nutch is standard for solr.

Crawl Anywhere looks solid. Any way for users like me to decide which one we
should go for Nutch or Crawl Anywehre?


Concern with crawl anywhere is it supports solr 1.3 index not the latest
version

Any help on the is really appreciated 

--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607833p2937762.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: [ANNOUNCE] Web Crawler

Posted by abhayd <aj...@hotmail.com>.
hi Julien ,

I m not sure what do u mean by 
"SOLR is now used by default for indexing in Nutch." Does that mean SOLR has
integrated Nutch for crawling web resources? 

I checked SOLR wiki but i didnt see something like that, Could u please
provide some details ? 


--
View this message in context: http://lucene.472066.n3.nabble.com/ANNOUNCE-Web-Crawler-tp2607833p2947623.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: [ANNOUNCE] Web Crawler

Posted by Julien Nioche <li...@gmail.com>.
> I dont see any activities on Nutch wiki so wondering if its not being
> developed anymore. But most forums say Nutch is standard for solr.
>

Looking at the mail archives is a good clue of whether a project is still
alive or not. In the case of Nutch, the project is active as you can see on
the list archives below :

http://www.mail-archive.com/user%40nutch.apache.org/
http://www.mail-archive.com/dev%40nutch.apache.org/

We're about to release a new version (1.3) and have a 2.0 in beta. SOLR is
now used by default for indexing in Nutch.

HTH

Julien Nioche

-- 
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com