You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ye T Thet <ye...@gmail.com> on 2012/07/08 18:10:08 UTC
Recommendation Nutch as Web Search Engine, Pre Solr or Solr
Hi Folks,
I am seeking recommendation whether I should use Pre Nutch 1.3 (without
Solr) or New Nutch (2.x) with Solr integration for my research project.
Little background information,
I developed prototype for web search engine during my post grad days using
Nutch as crawler, indexer and searcher. It was developed using < Nutch 1.3,
meaning not using Solr as searcher.
I am continuing my research after a year of on hold. I noticed a huge
changes in Nutch such as using SOLR as indexer and searcher, 2.x has
changed crawling implementation and etc.
The requirements for my project is similar typical web search engine with
lesser volume (less than 1 million pages for now). Additional requirements
are
1. Language Identification, (used language ID plug-in in Nutch using ngram
profile VS New Nutch used Tika for lang ID)
2. Custom lucene analyzer for the analysis (done in Nutch for Pre 1.3 VS
done in SOLR)
I would appreciate suggestions/comments on whether I should continue with
pre 1.3 or new Nutch with SOLR.
Thanks,
Y T Thet
Re: Recommendation Nutch as Web Search Engine, Pre Solr or Solr
Posted by Ye T Thet <ye...@gmail.com>.
Alexander,
Thanks for the recommendations. Those are very valuable to me.
I would probably lean towards the Nutch 2.x then.
On the side note, I do not find much of the tutorials/wiki entries on Nutch
2.x yet. I would go ahead and start re-implement custom plugins to SOLR and
Tika while waiting for Nutch 2.x documentations.
Thanks,
Y T Thet
On Mon, Jul 9, 2012 at 2:08 AM, Alexander Aristov <
alexander.aristov@gmail.com> wrote:
> Hi
>
> I would suggest you to take recent nutch versions anyway. Not only has
> indexer/web part changed but a lot of bugs and very handy cookies have been
> implemented. One of such noticable improvement was replacement of many doc
> parsers with 3rd party tika parser.
>
> Another good improvement since old days was fetcher improvement.It works
> much better and doesn't hang in some situations.
>
> as for which version to choose there are 2 versions:
>
> 1.5.x and 2.0
>
> 2.0 version contains all stuff from 1.5.x but it uses "database" instead of
> hdfs to keep data.
>
> Both versions send crawled data to solr which provides indexing and
> searching capabilities.
>
> Unfortunately there is no easy way to migrate from 1.3 to newest version
> and the easiest way will be to re-implement your custom plugins for these
> versions.
>
> Best Regards
> Alexander Aristov
>
>
> On 8 July 2012 20:10, Ye T Thet <ye...@gmail.com> wrote:
>
> > Hi Folks,
> >
> > I am seeking recommendation whether I should use Pre Nutch 1.3 (without
> > Solr) or New Nutch (2.x) with Solr integration for my research project.
> >
> > Little background information,
> > I developed prototype for web search engine during my post grad days
> using
> > Nutch as crawler, indexer and searcher. It was developed using < Nutch
> 1.3,
> > meaning not using Solr as searcher.
> >
> > I am continuing my research after a year of on hold. I noticed a huge
> > changes in Nutch such as using SOLR as indexer and searcher, 2.x has
> > changed crawling implementation and etc.
> >
> > The requirements for my project is similar typical web search engine with
> > lesser volume (less than 1 million pages for now). Additional
> requirements
> > are
> >
> > 1. Language Identification, (used language ID plug-in in Nutch using
> ngram
> > profile VS New Nutch used Tika for lang ID)
> > 2. Custom lucene analyzer for the analysis (done in Nutch for Pre 1.3 VS
> > done in SOLR)
> >
> > I would appreciate suggestions/comments on whether I should continue with
> > pre 1.3 or new Nutch with SOLR.
> >
> > Thanks,
> >
> > Y T Thet
> >
>
Re: Recommendation Nutch as Web Search Engine, Pre Solr or Solr
Posted by Alexander Aristov <al...@gmail.com>.
Hi
I would suggest you to take recent nutch versions anyway. Not only has
indexer/web part changed but a lot of bugs and very handy cookies have been
implemented. One of such noticable improvement was replacement of many doc
parsers with 3rd party tika parser.
Another good improvement since old days was fetcher improvement.It works
much better and doesn't hang in some situations.
as for which version to choose there are 2 versions:
1.5.x and 2.0
2.0 version contains all stuff from 1.5.x but it uses "database" instead of
hdfs to keep data.
Both versions send crawled data to solr which provides indexing and
searching capabilities.
Unfortunately there is no easy way to migrate from 1.3 to newest version
and the easiest way will be to re-implement your custom plugins for these
versions.
Best Regards
Alexander Aristov
On 8 July 2012 20:10, Ye T Thet <ye...@gmail.com> wrote:
> Hi Folks,
>
> I am seeking recommendation whether I should use Pre Nutch 1.3 (without
> Solr) or New Nutch (2.x) with Solr integration for my research project.
>
> Little background information,
> I developed prototype for web search engine during my post grad days using
> Nutch as crawler, indexer and searcher. It was developed using < Nutch 1.3,
> meaning not using Solr as searcher.
>
> I am continuing my research after a year of on hold. I noticed a huge
> changes in Nutch such as using SOLR as indexer and searcher, 2.x has
> changed crawling implementation and etc.
>
> The requirements for my project is similar typical web search engine with
> lesser volume (less than 1 million pages for now). Additional requirements
> are
>
> 1. Language Identification, (used language ID plug-in in Nutch using ngram
> profile VS New Nutch used Tika for lang ID)
> 2. Custom lucene analyzer for the analysis (done in Nutch for Pre 1.3 VS
> done in SOLR)
>
> I would appreciate suggestions/comments on whether I should continue with
> pre 1.3 or new Nutch with SOLR.
>
> Thanks,
>
> Y T Thet
>