Nutch vs Lucidworks Fusion

[Jorge Luis Betancourt Gonzalez <jl...@uci.cu>]

Hi the new Fusion product from Lucidworks provides “advanced filesystem and web crawlers” anyone have had any time to check this out and how to compare to the current and future plans with Nutch? Just interested I personally haven’t been able to download the product and test it but I’m a bit curious and I would appreciate your comments on this topic. 

Regards,Concurso "Mi selfie por los 5". Detalles en http://justiciaparaloscinco.wordpress.com

Re: Nutch vs Lucidworks Fusion

[Bayu Widyasanyata <bw...@gmail.com>]

I haven't used Fusion yet, but already play with Lucidworks 2.8.
The native embedded crawler for Lucidworks is Apperture [0].
IMHO nutch is better than Apperture in terms of stability, speed and
features.

[0] http://sourceforge.net/projects/aperture/

On Wed, Oct 1, 2014 at 9:19 AM, Jorge Luis Betancourt Gonzalez <
jlbetancourt@uci.cu> wrote:

> Hi the new Fusion product from Lucidworks provides “advanced filesystem
> and web crawlers” anyone have had any time to check this out and how to
> compare to the current and future plans with Nutch? Just interested I
> personally haven’t been able to download the product and test it but I’m a
> bit curious and I would appreciate your comments on this topic.
>
> Regards,Concurso "Mi selfie por los 5". Detalles en
> http://justiciaparaloscinco.wordpress.com
>



-- 
wassalam,
[bayu]

Re: Nutch vs Lucidworks Fusion

["Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>]

Thanks for the info Grant. Hope to see more info about the
crawler at some point and maybe even some day an ASF Fusion
crawler (which you guys already contribute a ton to open source
and maybe it will happen some day anyways).

Lots of good stuff going on in Nutch, Tika, Solr, OODT, your guys
stuff, CommonsCrawl and tons of good crawling stuff going on lately.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Grant Ignersoll <gs...@apache.org>
Reply-To: "user@nutch.apache.org" <us...@nutch.apache.org>
Date: Sunday, October 5, 2014 at 5:04 AM
To: "user@nutch.apache.org" <us...@nutch.apache.org>
Subject: Re: Nutch vs Lucidworks Fusion

>I'd also add a few things from my view from Lucidworks.  Nutch is a very
>good crawler.  When we truly need large scale distributed crawls at
>Lucid, we use Nutch and all the rest of the features of Fusion work
>(recommendation engine, signal capture, pipelines, etc.) just fine w/ it.
>
>As for our crawler (we call it Anda):
>
>1. It is geared towards the small to medium-largish crawls.  We don't
>rely on Hadoop (which is a significant overhead for most people who just
>want to crawl a few hundred thousand to a few million pages, in my
>experience. YMMV.) and it isn't a distributed crawler, although it can
>execute multiple parallel crawls across a large number of nodes.
>
>2. We've made it highly configurable, yet simple to start (an id and a
>URL to start)
>
>3. The foundation of our web crawler is actually the basis for several of
>our crawlers: Filesystems, Dropbox, Box, Jive (coming next release) and
>many more.  One of our goals for rewriting the crawler was to have it
>support a wide variety of crawl activities and make it easy to implement
>those crawlers.  Given the rate at which we are able to add new data
>sources, I would say we have achieved that (especially compared to our
>old crawlers)
>
>As for why we wrote our own?  To be honest, it started as a side project
>by one of our team members b/c he wanted to write a crawler and was never
>satisfied with what was currently available.  After a year or so of doing
>that, we started to use it in select commercial engagements where
>Aperture (our old crawler) was failing and where Nutch was too "heavy"
>(in reality, again, it was that Hadoop was too heavy), for lack of a
>better word.   From there, it has taken on numerous tasks for us and
>solved many very complex crawls at highly demanding customers due to its
>configurability and speed.
>
>
>On Oct 3, 2014, at 10:16 PM, David Arthur <mu...@gmail.com> wrote:
>
>> Hello, all. My name is David Arthur and I work with AB on Fusion over
>>here at
>> Lucidworks. Just wanted to point out that we are _not_ using Aperture
>> anymore. We were not super happy with it and have been looking to
>>replace it
>> for some time. We now have a home-grown crawler which is included with
>> Fusion and is significantly more performant than Aperture. I'm sure AB
>>will
>> chime in when the sun rises in his timezone ;)
>> 
>> Cheers
>> -David
>> 
>> 
>> 
>> --
>> View this message in context:
>>http://lucene.472066.n3.nabble.com/Nutch-vs-Lucidworks-Fusion-tp4162022p4
>>162626.html
>> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>--------------------------------------------
>Grant Ingersoll | @gsingers
>http://www.lucidworks.com
>
>
>
>
>

Re: Nutch vs Lucidworks Fusion

[Grant Ingersoll <gs...@apache.org>]

I'd also add a few things from my view from Lucidworks.  Nutch is a very good crawler.  When we truly need large scale distributed crawls at Lucid, we use Nutch and all the rest of the features of Fusion work (recommendation engine, signal capture, pipelines, etc.) just fine w/ it.

As for our crawler (we call it Anda):  

1. It is geared towards the small to medium-largish crawls.  We don't rely on Hadoop (which is a significant overhead for most people who just want to crawl a few hundred thousand to a few million pages, in my experience. YMMV.) and it isn't a distributed crawler, although it can execute multiple parallel crawls across a large number of nodes.  

2. We've made it highly configurable, yet simple to start (an id and a URL to start)

3. The foundation of our web crawler is actually the basis for several of our crawlers: Filesystems, Dropbox, Box, Jive (coming next release) and many more.  One of our goals for rewriting the crawler was to have it support a wide variety of crawl activities and make it easy to implement those crawlers.  Given the rate at which we are able to add new data sources, I would say we have achieved that (especially compared to our old crawlers)

As for why we wrote our own?  To be honest, it started as a side project by one of our team members b/c he wanted to write a crawler and was never satisfied with what was currently available.  After a year or so of doing that, we started to use it in select commercial engagements where Aperture (our old crawler) was failing and where Nutch was too "heavy" (in reality, again, it was that Hadoop was too heavy), for lack of a better word.   From there, it has taken on numerous tasks for us and solved many very complex crawls at highly demanding customers due to its configurability and speed.


On Oct 3, 2014, at 10:16 PM, David Arthur <mu...@gmail.com> wrote:

> Hello, all. My name is David Arthur and I work with AB on Fusion over here at
> Lucidworks. Just wanted to point out that we are _not_ using Aperture
> anymore. We were not super happy with it and have been looking to replace it
> for some time. We now have a home-grown crawler which is included with
> Fusion and is significantly more performant than Aperture. I'm sure AB will
> chime in when the sun rises in his timezone ;)
> 
> Cheers
> -David
> 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Nutch-vs-Lucidworks-Fusion-tp4162022p4162626.html
> Sent from the Nutch - User mailing list archive at Nabble.com.

--------------------------------------------
Grant Ingersoll | @gsingers
http://www.lucidworks.com

Re: Nutch vs Lucidworks Fusion

[David Arthur <mu...@gmail.com>]

Hello, all. My name is David Arthur and I work with AB on Fusion over here at
Lucidworks. Just wanted to point out that we are _not_ using Aperture
anymore. We were not super happy with it and have been looking to replace it
for some time. We now have a home-grown crawler which is included with
Fusion and is significantly more performant than Aperture. I'm sure AB will
chime in when the sun rises in his timezone ;)

Cheers
-David



--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-vs-Lucidworks-Fusion-tp4162022p4162626.html
Sent from the Nutch - User mailing list archive at Nabble.com.