You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by ca...@qualidade.info on 2011/07/08 23:59:29 UTC
Are we losing Nutch?
Relating to softwares, I think there are two types of people: on one side
there are the users , that want some funcionalities from them and support
those softwares by using them and, sometimes, with some comments or
sugestions. On the other side we have the programmers, many of them very
happy in developping things that can be useful, or not quite so, for
some main tasks, defined as the basis of project.
In the case o Nutch, it seems to me that in the very begining of the
project there was a clear vision of the mission: to have an open source,
powerful search engine.
But with open source projects, sometimes there is on the developers side,
a lack, or a deviation of the real focus and the people, even if they
follow some pattern, sometimes, don't go strictly after the real
objectives, established at the start of the project.
Today, with all those Tika, Solr, etc, it all became quite complicate for
me and other non-programmer people and it seems that the original aim
was lost, and Nutch is being used as a basis or as an excuse for a lot
of programming ventures, no matter how worthwhile they can be. But I
think that the real pedestrian users, those who use local instalations of
Nutch, vertical search engines, with about some hundred of thousands of
indexed pages, the people who really support the efforts to make Nutch
run, were forgotten on the road followed nowadays.
I think that Nutch should have been, allways, a primary project and other
projects should have been considered as add-ons to some particular,
special needs. Today it seems quite the oposite. Just look at the
mailing list and see how many questions are asked about Solr.
I am, really, very sad with the latest version, where Nutch was delegated
to just crawling functions.
I can't find an alternative to set up a real production search engine,
without any suitable web search interface that can be of any use to the
general audience. I really miss the searching function of Nutch with all
those nice features and easy of customization and it seems, to me, that
the original reason for building Nutch was defeated. For my purposes - a
real web search engine- the situation as it is in this version 1.3, led
me to think that Nutch is becoming a very handicapped project.
I was proud to use the logo Powered by Nutch. Now, I am afraid that I
will end up with a Google Custom Engine.
Re: Are we losing Nutch?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Carmmello and Lewis,
I'd also like to express my concern for your email. In my mind,
there are some immediate actions that might help you in your adoption and use
of Nutch:
* downgrade back to 1.2 and get the web app that you are missing
* wait for hopefully 1.4 where maybe one of us (I guess it's me
since I've mouthed off I want it back too) can put together a webapp
again
* identify what specifically is more complicated about Nutch now?
- running Solr as a requirement?
I haven't heard anything on his list that's more complicated besides
Solr based on your email so I just wanted to get some more insight
and information.
I can assure you there are a set of individuals here committed towards
making Apache Nutch the best open source horizontal search engine
that there is. That said, we are all volunteers here and trying to do this
the best we can.
Cheers,
Chris
On Jul 10, 2011, at 9:58 AM, lewis john mcgibbney wrote:
> Hi Carmmello,
>
> I would like to stress that I am only speaking from my own views on the way
> the project has been moving over the last year and a half or so but I would
> like to add the following points to address you quite obvious concerns
>
> There has been a lot of correspondence on closely linked topics over the
> past wee while, I think developers understand that there is a small step up
> to address the requirements of Nutch 1.3, and there is also always inherent
> problems when you individuals and communities are faced with change. I would
> like to say with confidence that the current version of Nutch is a well
> refined tool which is adapting very accurately to provide the best crawling
> functionality of a very dynamically developing web consisting of various
> complex graph structures. I would like to make it clear that it is extremely
> important to have a stable and well designed crawling implementation such as
> Nutch 1.3 as if you look on the Dev lists you will see the barrage of tasks,
> existing in an array of complexity, functionality and accuracy which keep
> Nutch running in parallel with < daily changes to the dynamic web. If Nutch
> is not focussing on crawling, then no matter whether we have a web
> application interface or a Lucene index the quality of data fetched will
> simply not be up to scratch. I hope you can appreciated the burden which
> this imposes on the directional decisions made within the Project Management
> Committee in the last year or so...
>
> Developers across many of the ASF projects understand that being user
> friendly is an excellent attribute to have in any open source Apache
> implementation. However projects develop and in the case of the Apache
> Software foundation, some of these project spawn sub-projects which graduate
> to become thier own top level independant projects. As I'm sure you are
> aware, this was the case with Nutch, therefore it means that as a community
> we should be able to make decisions independently in the best interests of
> the project. We have talk about not reinventing the wheel, well this is also
> going on across the ASF board of projects. One thing to consider is that
> many developers, contributors and PMCs do not belong to one project, they
> give up time, effort and resources to sometimes several projects, therefore
> it is very important that as a project Nutch reserves the priority of
> removing duplication across the board.
>
> Addressing you point regarding the real objectives established at the
> beginning of the project, there has been significant progress made within
> Nutch and excellent sub projects which have since graduated to top level
> projects (I'm sure there is no reason to name them) with their own bustling
> communities. Allowing Nutch to stagnate and claim to be a one-size-fits-all
> search engine would have jeopardised the viability of all of these
> successful projects and would have therefore prevented the very innovation
> that earns open source implementation under the ASF the reputation and
> widespread use that projects are renowned for.
>
> For example if we take the latest Nutch 1.3 release, we have two options to
> deploy, in local mode (running on one machine) or in deploy mode (harnessing
> the strength of parallel processing jobs for different kinds of Nutch
> users). The development has been driven purely by variances seen across the
> community usage of Nutch. We draw upon progress made in other delegated
> areas for the benefit of the project, not to isolate non-programmers from
> using newer versions of the code base.
>
> I would also like to add that there are many questions asked about Solr due
> to a number of criteria, namely:
>
>> Various developers/committers/PMC members of Nutch are also members of
> various Solr groups.
>> Developers and users are kind enough to take the time and effort to answer
> Solr related questions as it is commonly recognised now that Solr is the
> widespread indexing mechanism (which also has an easily configurable GUI).
>> It is not very often that users on the Nutch user@ list are ignored or
> thier queries unanswered, however if this is the case there is good reason
> behind it. In general, and in my opinion, when I started using Nutch I found
> the help on user@ not only extremely beneficial but also a confidence boost
> to get me working on Sold and other project lists.
>
> I suppose that there are always 2 sides to every story and it is very
> uncomforting to hear that you are really not happy with the latest release,
> there was a lot of hard work put into its development. Amongst bug fixes and
> other potential barriers mentioned above and previously on this list I would
> like to think that as the project matures it users can also recognise the
> dynamism which needs to exist in a project of Nutch's nature i order to
> present users with a stable and robust software choice. Instead of becoming
> handicapped we have a clear vision for Nutch 2.0, Nutch branches e.g. 1.4
> and many new fixes on the way. I suppose it depends from which side of the
> table you are on when you mention that it is becoming a handicapped project
> but my honest opinion is one from the other end of the spectrum.
>
> To conclude I would quite like to say that the option is always there to
> work with Nutch 1.2 and gradually make yourself aware of the great features
> and benefits your general audience could benefit from when using a nicely
> tunes Nutch Solr search engine. As a simple example, everyone uses ebay
> right? Well with some careful and well thought out design decisions the work
> to get something similar with probably better functionality failored for
> your specific users up and running is really not a hundred miles away if you
> were comfortable using < Nutch 1.3.
>
> As I said it is not good to hear that you are unhappy with the current
> release, but I hope I have put across some points in such a way that they
> provide justification as well as substance to not only the design decisions
> but also to the roadmap for the Nutch project moving forward.
>
> Thank you
>
> On Fri, Jul 8, 2011 at 10:59 PM, <ca...@qualidade.info> wrote:
>
>> Relating to softwares, I think there are two types of people: on one side
>> there are the users , that want some funcionalities from them and support
>> those softwares by using them and, sometimes, with some comments or
>> sugestions. On the other side we have the programmers, many of them very
>> happy in developping things that can be useful, or not quite so, for
>> some main tasks, defined as the basis of project.
>>
>> In the case o Nutch, it seems to me that in the very begining of the
>> project there was a clear vision of the mission: to have an open source,
>> powerful search engine.
>>
>> But with open source projects, sometimes there is on the developers side,
>> a lack, or a deviation of the real focus and the people, even if they
>> follow some pattern, sometimes, don't go strictly after the real
>> objectives, established at the start of the project.
>>
>> Today, with all those Tika, Solr, etc, it all became quite complicate for
>> me and other non-programmer people and it seems that the original aim
>> was lost, and Nutch is being used as a basis or as an excuse for a lot
>> of programming ventures, no matter how worthwhile they can be. But I
>> think that the real pedestrian users, those who use local instalations of
>> Nutch, vertical search engines, with about some hundred of thousands of
>> indexed pages, the people who really support the efforts to make Nutch
>> run, were forgotten on the road followed nowadays.
>> I think that Nutch should have been, allways, a primary project and other
>> projects should have been considered as add-ons to some particular,
>> special needs. Today it seems quite the oposite. Just look at the
>> mailing list and see how many questions are asked about Solr.
>>
>> I am, really, very sad with the latest version, where Nutch was delegated
>> to just crawling functions.
>> I can't find an alternative to set up a real production search engine,
>> without any suitable web search interface that can be of any use to the
>> general audience. I really miss the searching function of Nutch with all
>> those nice features and easy of customization and it seems, to me, that
>> the original reason for building Nutch was defeated. For my purposes - a
>> real web search engine- the situation as it is in this version 1.3, led
>> me to think that Nutch is becoming a very handicapped project.
>>
>> I was proud to use the logo “Powered by Nutch”. Now, I am afraid that I
>> will end up with a Google Custom Engine.
>>
>>
>>
>>
>>
>>
>
>
> --
> *Lewis*
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Are we losing Nutch?
Posted by lewis john mcgibbney <le...@gmail.com>.
Hi Carmmello,
I would like to stress that I am only speaking from my own views on the way
the project has been moving over the last year and a half or so but I would
like to add the following points to address you quite obvious concerns
There has been a lot of correspondence on closely linked topics over the
past wee while, I think developers understand that there is a small step up
to address the requirements of Nutch 1.3, and there is also always inherent
problems when you individuals and communities are faced with change. I would
like to say with confidence that the current version of Nutch is a well
refined tool which is adapting very accurately to provide the best crawling
functionality of a very dynamically developing web consisting of various
complex graph structures. I would like to make it clear that it is extremely
important to have a stable and well designed crawling implementation such as
Nutch 1.3 as if you look on the Dev lists you will see the barrage of tasks,
existing in an array of complexity, functionality and accuracy which keep
Nutch running in parallel with < daily changes to the dynamic web. If Nutch
is not focussing on crawling, then no matter whether we have a web
application interface or a Lucene index the quality of data fetched will
simply not be up to scratch. I hope you can appreciated the burden which
this imposes on the directional decisions made within the Project Management
Committee in the last year or so...
Developers across many of the ASF projects understand that being user
friendly is an excellent attribute to have in any open source Apache
implementation. However projects develop and in the case of the Apache
Software foundation, some of these project spawn sub-projects which graduate
to become thier own top level independant projects. As I'm sure you are
aware, this was the case with Nutch, therefore it means that as a community
we should be able to make decisions independently in the best interests of
the project. We have talk about not reinventing the wheel, well this is also
going on across the ASF board of projects. One thing to consider is that
many developers, contributors and PMCs do not belong to one project, they
give up time, effort and resources to sometimes several projects, therefore
it is very important that as a project Nutch reserves the priority of
removing duplication across the board.
Addressing you point regarding the real objectives established at the
beginning of the project, there has been significant progress made within
Nutch and excellent sub projects which have since graduated to top level
projects (I'm sure there is no reason to name them) with their own bustling
communities. Allowing Nutch to stagnate and claim to be a one-size-fits-all
search engine would have jeopardised the viability of all of these
successful projects and would have therefore prevented the very innovation
that earns open source implementation under the ASF the reputation and
widespread use that projects are renowned for.
For example if we take the latest Nutch 1.3 release, we have two options to
deploy, in local mode (running on one machine) or in deploy mode (harnessing
the strength of parallel processing jobs for different kinds of Nutch
users). The development has been driven purely by variances seen across the
community usage of Nutch. We draw upon progress made in other delegated
areas for the benefit of the project, not to isolate non-programmers from
using newer versions of the code base.
I would also like to add that there are many questions asked about Solr due
to a number of criteria, namely:
>Various developers/committers/PMC members of Nutch are also members of
various Solr groups.
>Developers and users are kind enough to take the time and effort to answer
Solr related questions as it is commonly recognised now that Solr is the
widespread indexing mechanism (which also has an easily configurable GUI).
>It is not very often that users on the Nutch user@ list are ignored or
thier queries unanswered, however if this is the case there is good reason
behind it. In general, and in my opinion, when I started using Nutch I found
the help on user@ not only extremely beneficial but also a confidence boost
to get me working on Sold and other project lists.
I suppose that there are always 2 sides to every story and it is very
uncomforting to hear that you are really not happy with the latest release,
there was a lot of hard work put into its development. Amongst bug fixes and
other potential barriers mentioned above and previously on this list I would
like to think that as the project matures it users can also recognise the
dynamism which needs to exist in a project of Nutch's nature i order to
present users with a stable and robust software choice. Instead of becoming
handicapped we have a clear vision for Nutch 2.0, Nutch branches e.g. 1.4
and many new fixes on the way. I suppose it depends from which side of the
table you are on when you mention that it is becoming a handicapped project
but my honest opinion is one from the other end of the spectrum.
To conclude I would quite like to say that the option is always there to
work with Nutch 1.2 and gradually make yourself aware of the great features
and benefits your general audience could benefit from when using a nicely
tunes Nutch Solr search engine. As a simple example, everyone uses ebay
right? Well with some careful and well thought out design decisions the work
to get something similar with probably better functionality failored for
your specific users up and running is really not a hundred miles away if you
were comfortable using < Nutch 1.3.
As I said it is not good to hear that you are unhappy with the current
release, but I hope I have put across some points in such a way that they
provide justification as well as substance to not only the design decisions
but also to the roadmap for the Nutch project moving forward.
Thank you
On Fri, Jul 8, 2011 at 10:59 PM, <ca...@qualidade.info> wrote:
> Relating to softwares, I think there are two types of people: on one side
> there are the users , that want some funcionalities from them and support
> those softwares by using them and, sometimes, with some comments or
> sugestions. On the other side we have the programmers, many of them very
> happy in developping things that can be useful, or not quite so, for
> some main tasks, defined as the basis of project.
>
> In the case o Nutch, it seems to me that in the very begining of the
> project there was a clear vision of the mission: to have an open source,
> powerful search engine.
>
> But with open source projects, sometimes there is on the developers side,
> a lack, or a deviation of the real focus and the people, even if they
> follow some pattern, sometimes, don't go strictly after the real
> objectives, established at the start of the project.
>
> Today, with all those Tika, Solr, etc, it all became quite complicate for
> me and other non-programmer people and it seems that the original aim
> was lost, and Nutch is being used as a basis or as an excuse for a lot
> of programming ventures, no matter how worthwhile they can be. But I
> think that the real pedestrian users, those who use local instalations of
> Nutch, vertical search engines, with about some hundred of thousands of
> indexed pages, the people who really support the efforts to make Nutch
> run, were forgotten on the road followed nowadays.
> I think that Nutch should have been, allways, a primary project and other
> projects should have been considered as add-ons to some particular,
> special needs. Today it seems quite the oposite. Just look at the
> mailing list and see how many questions are asked about Solr.
>
> I am, really, very sad with the latest version, where Nutch was delegated
> to just crawling functions.
> I can't find an alternative to set up a real production search engine,
> without any suitable web search interface that can be of any use to the
> general audience. I really miss the searching function of Nutch with all
> those nice features and easy of customization and it seems, to me, that
> the original reason for building Nutch was defeated. For my purposes - a
> real web search engine- the situation as it is in this version 1.3, led
> me to think that Nutch is becoming a very handicapped project.
>
> I was proud to use the logo “Powered by Nutch”. Now, I am afraid that I
> will end up with a Google Custom Engine.
>
>
>
>
>
>
--
*Lewis*