You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by PJ Herring <he...@gmail.com> on 2011/11/21 22:14:19 UTC
Dependency Injection
Hey,
So I am admittedly a noob with Nutch, but have spent some time digging through the source code. I am just curious if anyone has talked about, in future developments of Nutch, replacing the whole way we register plugins? I ask because I am using Nutch on a project with Maven. At the moment I have to copy a whole bunch of JAR files with there plugin.xml files into a certain directory on build, which is fine, but is kind of breaking the whole Maven paradigm. It would be nice to have some sort of Maven repository where plugins lived, and then wire up which plugins I want to use using some kind of DI framework, like Spring or Guice. Then instead of writing XML Plugin Descriptor Files, every plugin could write a class extending PluginDescriptor and register its self with the PluginRepo, or something of the sort.
Also, I have never contributed to an open source project, so if I am being an ass I don't mean to be. Just would love to help make a great tool better in any way.
Best,
PJ Herring
Re: Dependency Injection
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi PJ,
I can only reinforce what Chris said about
1) Not being a pain, and
2) Helping out with the proposed work. If something can be proposed which
makes good sense and which adds value to the project then it is an
overwhelming possibility that a development effort will back such a
proposal. Of the threads that Chris has provided, I would ask you to take a
look at NUTCH-609 in particular. This is very interesting but as you can
see has lacked any recent development drive.
Thanks
On Mon, Nov 21, 2011 at 11:16 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey PJ,
>
> You aren't being an ass at all. You're asking an important question, and
> something I've been interested in for a while.
> Here are some relevant threads to take a look at:
>
> http://wiki.apache.org/nutch/Nutch2Architecture
>
> http://www.mail-archive.com/nutch-developers@lists.sourceforge.net/msg12688.html
>
> http://www.slideshare.net/chrismattmann/lessons-learned-in-the-development-of-a-webscale-search-engine-nutch2-and-beyond
> https://issues.apache.org/jira/browse/NUTCH-609
> http://osdir.com/ml/user.nutch.apache/2011-07/msg00080.html
> http://5341.com/list/48/349985.html
>
> If you're interested in contributing to Apache Nutch, check this great
> guide out written by Dennis Kubes:
>
> wiki.apache.org/nutch/Becoming_A_Nutch_Developer
>
> Before there wasn't a ton of interest in replacing the plugin system since
> it "worked" and we didn't get a lot of
> complaints or anything. That interest turned into the perception that a DI
> framework wouldn't be welcome.
> On the contrary, I'd say if you figured out how to get a DI framework
> working with the existing plugin system,
> I can personally say I'd dedicate some of my time towards helping you
> shepherd it in and I think the
> rest of the committers would be on board.
>
> Thanks for your interest. If you have any more questions, please ask!
>
> Cheers,
> Chris
>
>
> On Nov 21, 2011, at 1:14 PM, PJ Herring wrote:
>
> > Hey,
> >
> > So I am admittedly a noob with Nutch, but have spent some time digging
> through the source code. I am just curious if anyone has talked about, in
> future developments of Nutch, replacing the whole way we register plugins?
> I ask because I am using Nutch on a project with Maven. At the moment I
> have to copy a whole bunch of JAR files with there plugin.xml files into a
> certain directory on build, which is fine, but is kind of breaking the
> whole Maven paradigm. It would be nice to have some sort of Maven
> repository where plugins lived, and then wire up which plugins I want to
> use using some kind of DI framework, like Spring or Guice. Then instead of
> writing XML Plugin Descriptor Files, every plugin could write a class
> extending PluginDescriptor and register its self with the PluginRepo, or
> something of the sort.
> >
> > Also, I have never contributed to an open source project, so if I am
> being an ass I don't mean to be. Just would love to help make a great tool
> better in any way.
> >
> > Best,
> > PJ Herring
> >
> >
> >
> >
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
*Lewis*
Re: Dependency Injection
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey PJ,
On Nov 22, 2011, at 10:47 AM, PJ Herring wrote:
> Hey Chris,
>
> Thanks for the response. I looked at the documents you sent me, and I really do think incorporating some kind of DI Framework could be a great addition to Nutch.
>
> I have a general plan of attack, but I'll try to write that up more formally and send it out to get some kind of feedback.
+1, would love to see it.
>
> One question I had when looking at this stuff is what is the status of Nutch 2? It looks like the architecture has shifted quite a bit from 1.3?
Nutch2 was originally slated to be the Nutch Gora branch (see here [1]). We ended up deciding [2] that the trunk was more akin to folks who were
maintaining the 1.x series of Nutch and thus moved the Nutch Gora branch into [1].
We still have a lot of goals though for Nutch2, which I think we're just working to more incrementally, rather than radically, as before. There are
still folks here working on Nutch Gora though so if you're interested in that, check it out.
Cheers,
Chris
[1] http://svn.apache.org/repos/asf/nutch/branches/nutchgora
[2] http://s.apache.org/zX
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Dependency Injection
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 23/11/2011 01:02, Andrzej Bialecki wrote:
> On 22/11/2011 19:47, PJ Herring wrote:
>> Hey Chris,
>>
>> Thanks for the response. I looked at the documents you sent me, and I
>> really do think incorporating some kind of DI Framework could be a great
>> addition to Nutch.
>>
>> I have a general plan of attack, but I'll try to write that up more
>> formally and send it out to get some kind of feedback.
>
> This sounds interesting. As Chris mentioned, the current plugin system
> is far from ideal, but so far it worked reasonably well. The key
> functionality that it implements is:
>
> * self-discovery of services provided by each plugin,
> * easy pluggability, by the virtue of dropping "super-jars" (jars with
> impl. classes and nested library jars) to a predefined location,
> * controlled classloader isolation between plugins so that incompatible
> versions of libraries can be used
> * but also ability to export specified classes and libraries so that one
> plugin can use other plugin's exported resources on its classpath.
> * optional auto-loading of dependent plugins
>
> In the past one contributor made a bold attempt to port Nutch to OSGI,
> and it turned out to be much more complicated than we expected, and with
> a bigger impact on the way Nutch applications were supposed to run ...
> so at that time we didn't think this complication was justified.
>
> If we can figure out something between full-blown OSGI and the current
> system then that would be great.
>
You may also want to take a look at JSPF (http://code.google.com/p/jspf)
which perhaps could be made to satisfy the above requirements without
too much refactoring.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Dependency Injection
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 22/11/2011 19:47, PJ Herring wrote:
> Hey Chris,
>
> Thanks for the response. I looked at the documents you sent me, and I
> really do think incorporating some kind of DI Framework could be a great
> addition to Nutch.
>
> I have a general plan of attack, but I'll try to write that up more
> formally and send it out to get some kind of feedback.
This sounds interesting. As Chris mentioned, the current plugin system
is far from ideal, but so far it worked reasonably well. The key
functionality that it implements is:
* self-discovery of services provided by each plugin,
* easy pluggability, by the virtue of dropping "super-jars" (jars with
impl. classes and nested library jars) to a predefined location,
* controlled classloader isolation between plugins so that incompatible
versions of libraries can be used
* but also ability to export specified classes and libraries so that one
plugin can use other plugin's exported resources on its classpath.
* optional auto-loading of dependent plugins
In the past one contributor made a bold attempt to port Nutch to OSGI,
and it turned out to be much more complicated than we expected, and with
a bigger impact on the way Nutch applications were supposed to run ...
so at that time we didn't think this complication was justified.
If we can figure out something between full-blown OSGI and the current
system then that would be great.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Dependency Injection
Posted by PJ Herring <he...@gmail.com>.
Hey Chris,
Thanks for the response. I looked at the documents you sent me, and I really do think incorporating some kind of DI Framework could be a great addition to Nutch.
I have a general plan of attack, but I'll try to write that up more formally and send it out to get some kind of feedback.
One question I had when looking at this stuff is what is the status of Nutch 2? It looks like the architecture has shifted quite a bit from 1.3?
Thanks,
PJ
On Nov 21, 2011, at 6:16 PM, Mattmann, Chris A (388J) wrote:
> Hey PJ,
>
> You aren't being an ass at all. You're asking an important question, and something I've been interested in for a while.
> Here are some relevant threads to take a look at:
>
> http://wiki.apache.org/nutch/Nutch2Architecture
> http://www.mail-archive.com/nutch-developers@lists.sourceforge.net/msg12688.html
> http://www.slideshare.net/chrismattmann/lessons-learned-in-the-development-of-a-webscale-search-engine-nutch2-and-beyond
> https://issues.apache.org/jira/browse/NUTCH-609
> http://osdir.com/ml/user.nutch.apache/2011-07/msg00080.html
> http://5341.com/list/48/349985.html
>
> If you're interested in contributing to Apache Nutch, check this great guide out written by Dennis Kubes:
>
> wiki.apache.org/nutch/Becoming_A_Nutch_Developer
>
> Before there wasn't a ton of interest in replacing the plugin system since it "worked" and we didn't get a lot of
> complaints or anything. That interest turned into the perception that a DI framework wouldn't be welcome.
> On the contrary, I'd say if you figured out how to get a DI framework working with the existing plugin system,
> I can personally say I'd dedicate some of my time towards helping you shepherd it in and I think the
> rest of the committers would be on board.
>
> Thanks for your interest. If you have any more questions, please ask!
>
> Cheers,
> Chris
>
>
> On Nov 21, 2011, at 1:14 PM, PJ Herring wrote:
>
>> Hey,
>>
>> So I am admittedly a noob with Nutch, but have spent some time digging through the source code. I am just curious if anyone has talked about, in future developments of Nutch, replacing the whole way we register plugins? I ask because I am using Nutch on a project with Maven. At the moment I have to copy a whole bunch of JAR files with there plugin.xml files into a certain directory on build, which is fine, but is kind of breaking the whole Maven paradigm. It would be nice to have some sort of Maven repository where plugins lived, and then wire up which plugins I want to use using some kind of DI framework, like Spring or Guice. Then instead of writing XML Plugin Descriptor Files, every plugin could write a class extending PluginDescriptor and register its self with the PluginRepo, or something of the sort.
>>
>> Also, I have never contributed to an open source project, so if I am being an ass I don't mean to be. Just would love to help make a great tool better in any way.
>>
>> Best,
>> PJ Herring
>>
>>
>>
>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
...
PJ Herring
Re: Dependency Injection
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey PJ,
You aren't being an ass at all. You're asking an important question, and something I've been interested in for a while.
Here are some relevant threads to take a look at:
http://wiki.apache.org/nutch/Nutch2Architecture
http://www.mail-archive.com/nutch-developers@lists.sourceforge.net/msg12688.html
http://www.slideshare.net/chrismattmann/lessons-learned-in-the-development-of-a-webscale-search-engine-nutch2-and-beyond
https://issues.apache.org/jira/browse/NUTCH-609
http://osdir.com/ml/user.nutch.apache/2011-07/msg00080.html
http://5341.com/list/48/349985.html
If you're interested in contributing to Apache Nutch, check this great guide out written by Dennis Kubes:
wiki.apache.org/nutch/Becoming_A_Nutch_Developer
Before there wasn't a ton of interest in replacing the plugin system since it "worked" and we didn't get a lot of
complaints or anything. That interest turned into the perception that a DI framework wouldn't be welcome.
On the contrary, I'd say if you figured out how to get a DI framework working with the existing plugin system,
I can personally say I'd dedicate some of my time towards helping you shepherd it in and I think the
rest of the committers would be on board.
Thanks for your interest. If you have any more questions, please ask!
Cheers,
Chris
On Nov 21, 2011, at 1:14 PM, PJ Herring wrote:
> Hey,
>
> So I am admittedly a noob with Nutch, but have spent some time digging through the source code. I am just curious if anyone has talked about, in future developments of Nutch, replacing the whole way we register plugins? I ask because I am using Nutch on a project with Maven. At the moment I have to copy a whole bunch of JAR files with there plugin.xml files into a certain directory on build, which is fine, but is kind of breaking the whole Maven paradigm. It would be nice to have some sort of Maven repository where plugins lived, and then wire up which plugins I want to use using some kind of DI framework, like Spring or Guice. Then instead of writing XML Plugin Descriptor Files, every plugin could write a class extending PluginDescriptor and register its self with the PluginRepo, or something of the sort.
>
> Also, I have never contributed to an open source project, so if I am being an ass I don't mean to be. Just would love to help make a great tool better in any way.
>
> Best,
> PJ Herring
>
>
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++