You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Doug Cutting <cu...@nutch.org> on 2005/03/30 22:53:24 UTC

tools cleanup

I propose we cleanup Nutch's tools as follows.

First, some definitions:

1. An "action" is an operation on Nutch data.  For example, 
GenerateSegmentFromDB, FetchSegment, UpdateDB, IndexSegment, 
MergeIndexes, SearchServer, etc. are all actions.

2. A "tool" invokes an action from the command line.

The proposal:

1. Actions and tools should be separate classes, in separate files.

2. A tool class should define no methods other than a main() and perhaps 
those required to parse the command line.  All application logic should 
be in the action class.

3. All actions must implement the following interface:

   public interface NutchConfigurable {
     void setConf(NutchConf conf);
     NutchConf getConf();
   }

4. Most actions should implement this by extending:

   public class NutchConfigured implements NutchConfigurable {
     private NutchConf conf;
     public NutchConfigured(NutchConf conf) { setConf(conf); }
     public void setConf(NutchConf conf) { this.conf = conf; }
     public NutchConf getConf() { return conf; }
   }

5. All plugins must implement NutchConfigurable.

6. Plugin factory methods must accept a NutchConf.

For example:

   public static Protocol ProtocolFactory.getProtocol(String url);

will become:

   public static Protocol ProtocolFactory.getProtocol(NutchConf, String);

Comments?

Doug


Re: tools cleanup

Posted by Jérôme Charron <je...@gmail.com>.
> > The proposal:
Not really the same subject, but I was thinking about this since a while:
Isn't it time to split Nutch into modules? For instance:
* A Core module
* A Util module
* An API module
* A Plugin module

with the dependencies:
1. Core depends on API (it uses APIs to call plugins) and Util
2. API has no dependencies
3. Util has no dependencies
4. Plugin depends on API (it implements interfaces) and Util
5. and so on

This will clearly document "who can depends on who".
And it will ensure that no dangerous code dependencies are used,
because it will crashes during compilation if for instance a plugin
uses some core code.
 
Jerome

-- 
http://motrech.free.fr/
http://frutch.free.fr/

Re: tools cleanup

Posted by Doug Cutting <cu...@nutch.org>.
Doug Cutting wrote:

> The proposal:

One more:

7.  No code should call NutchConf.get() except a tool's main().

Doug

Re: tools cleanup

Posted by Sami Siren <s....@sonera.inet.fi>.
+1

--
  Sami Siren


Doug Cutting wrote:
> I propose we cleanup Nutch's tools as follows.
> 
> First, some definitions:
> 
> 1. An "action" is an operation on Nutch data.  For example, 
> GenerateSegmentFromDB, FetchSegment, UpdateDB, IndexSegment, 
> MergeIndexes, SearchServer, etc. are all actions.
> 
> 2. A "tool" invokes an action from the command line.
> 
> The proposal:
> 
> 1. Actions and tools should be separate classes, in separate files.
> 
> 2. A tool class should define no methods other than a main() and perhaps 
> those required to parse the command line.  All application logic should 
> be in the action class.
> 
> 3. All actions must implement the following interface:
> 
>   public interface NutchConfigurable {
>     void setConf(NutchConf conf);
>     NutchConf getConf();
>   }
> 
> 4. Most actions should implement this by extending:
> 
>   public class NutchConfigured implements NutchConfigurable {
>     private NutchConf conf;
>     public NutchConfigured(NutchConf conf) { setConf(conf); }
>     public void setConf(NutchConf conf) { this.conf = conf; }
>     public NutchConf getConf() { return conf; }
>   }
> 
> 5. All plugins must implement NutchConfigurable.
> 
> 6. Plugin factory methods must accept a NutchConf.
> 
> For example:
> 
>   public static Protocol ProtocolFactory.getProtocol(String url);
> 
> will become:
> 
>   public static Protocol ProtocolFactory.getProtocol(NutchConf, String);
> 
> Comments?
> 
> Doug
> 
> 


Re: tools cleanup

Posted by Doug Cutting <cu...@nutch.org>.
Sami Siren wrote:
> should we introduce a new package for these: NutchConfigurable, 
> NutchConfigured and the upcoming action classes -

I've added these in util in the mapred branch and will use them as I 
rewrite tools to use MapReduce.  I'll commit them soon.

Doug

Re: tools cleanup

Posted by Sami Siren <s....@sonera.inet.fi>.
should we introduce a new package for these: NutchConfigurable, 
NutchConfigured and the upcoming action classes -

org.apache.nutch.action ?

--
  Sami Siren

Stefan Groschupf wrote:
> Hi,
> Doug, can you or someone else please commit the classes you suggested, I 
> think most / all agree and we can start porting things, but if all 
> people create now own NutchConfigurable interfaces we will run in 
> trouble and people are unhappy when they need to correct patches they 
> submitted or patches are not accepted.
> 
> Thanks a lot.
> Stefan
> 

Re: tools cleanup

Posted by Stefan Groschupf <sg...@media-style.com>.
Hi,
Doug, can you or someone else please commit the classes you suggested, 
I think most / all agree and we can start porting things, but if all 
people create now own NutchConfigurable interfaces we will run in 
trouble and people are unhappy when they need to correct patches they 
submitted or patches are not accepted.

Thanks a lot.
Stefan

Am 30.03.2005 um 22:53 schrieb Doug Cutting:

> I propose we cleanup Nutch's tools as follows.
>
> First, some definitions:
>
> 1. An "action" is an operation on Nutch data.  For example, 
> GenerateSegmentFromDB, FetchSegment, UpdateDB, IndexSegment, 
> MergeIndexes, SearchServer, etc. are all actions.
>
> 2. A "tool" invokes an action from the command line.
>
> The proposal:
>
> 1. Actions and tools should be separate classes, in separate files.
>
> 2. A tool class should define no methods other than a main() and 
> perhaps those required to parse the command line.  All application 
> logic should be in the action class.
>
> 3. All actions must implement the following interface:
>
>   public interface NutchConfigurable {
>     void setConf(NutchConf conf);
>     NutchConf getConf();
>   }
>
> 4. Most actions should implement this by extending:
>
>   public class NutchConfigured implements NutchConfigurable {
>     private NutchConf conf;
>     public NutchConfigured(NutchConf conf) { setConf(conf); }
>     public void setConf(NutchConf conf) { this.conf = conf; }
>     public NutchConf getConf() { return conf; }
>   }
>
> 5. All plugins must implement NutchConfigurable.
>
> 6. Plugin factory methods must accept a NutchConf.
>
> For example:
>
>   public static Protocol ProtocolFactory.getProtocol(String url);
>
> will become:
>
>   public static Protocol ProtocolFactory.getProtocol(NutchConf, 
> String);
>
> Comments?
>
> Doug
>
>
>
---------------------------------------------------------------
company:		http://www.media-style.com
forum:		http://www.text-mining.org
blog:			http://www.find23.net


Re: tools cleanup

Posted by Feng Zhou <fe...@gmail.com>.
I second this. But it would still be useful to keep the current NDFS
config entries. This is because if these URI's become the main method
of using ndfs, they could end up in a lot of scripts users write. Then
it would be inconvenient to change the namenode. Maybe we could use
ndfs:///path (three slashes as in file:///) to specify NDFS dir on the
default namenode.

- Feng Zhou

On Wed, 30 Mar 2005 17:11:52 -0800, John X <jo...@neasys.com> wrote:
> On Thu, Mar 31, 2005 at 12:45:39AM +0200, Stefan Groschupf wrote:
> >
> > Actually it is difficult to have tools using ndfs and local file system.
> > What do people think about introducing a ndfs notation in paths like it
> > is used in protocol handlers? (ala http:// or file://)
> > I don't mean to write a protocoll hander for ndfs (this would be nice
> > to have) but I just mean something like:
> >
> > bin/nutch     generate   ndfs://namenode:8010/myNDFSFolder/mydb
> > /mylocalsegment/
> > ndfs path: ndfs://namenode:8010/myNDFSFolder/mydb
> > local path: /mylocalsegment/
> >
> 
> I agree.
> 
> John
> 


-- 
- Feng

Re: tools cleanup

Posted by Doug Cutting <cu...@nutch.org>.
Andrzej Bialecki wrote:
> This also nicely solves the non-obvious requirement that all ndfs paths 
> must begin with a slash...

I fixed that a while back.  Things that don't start with a slash are 
currently made relative to /user/$USER.

Doug

Re: tools cleanup

Posted by Andrzej Bialecki <ab...@getopt.org>.
John X wrote:
> On Thu, Mar 31, 2005 at 12:45:39AM +0200, Stefan Groschupf wrote:
> 
>>Actually it is difficult to have tools using ndfs and local file system.
>>What do people think about introducing a ndfs notation in paths like it 
>>is used in protocol handlers? (ala http:// or file://)
>>I don't mean to write a protocoll hander for ndfs (this would be nice 
>>to have) but I just mean something like:
>>
>>bin/nutch     generate   ndfs://namenode:8010/myNDFSFolder/mydb        
>>/mylocalsegment/
>>ndfs path: ndfs://namenode:8010/myNDFSFolder/mydb
>>local path: /mylocalsegment/
>>
> 
> 
> I agree.

Me too. It would be nice to use the standard URL class for parsing 
these, but you need to register the protocol first, I think there was a 
message recently that pointed to a howto.

This also nicely solves the non-obvious requirement that all ndfs paths 
must begin with a slash...

-- 
Best regards,
Andrzej Bialecki
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Re: tools cleanup

Posted by John X <jo...@neasys.com>.
On Thu, Mar 31, 2005 at 12:45:39AM +0200, Stefan Groschupf wrote:
> 
> Actually it is difficult to have tools using ndfs and local file system.
> What do people think about introducing a ndfs notation in paths like it 
> is used in protocol handlers? (ala http:// or file://)
> I don't mean to write a protocoll hander for ndfs (this would be nice 
> to have) but I just mean something like:
> 
> bin/nutch     generate   ndfs://namenode:8010/myNDFSFolder/mydb        
> /mylocalsegment/
> ndfs path: ndfs://namenode:8010/myNDFSFolder/mydb
> local path: /mylocalsegment/
> 

I agree.

John

Re: tools cleanup

Posted by Stefan Groschupf <sg...@media-style.com>.
Doug,

> The proposal:
>
> 1. Actions and tools should be separate classes, in separate files.

Wonderful! :-) That will make a set of things (e.g. run nutch in a 
container) very easy.
> 3. All actions must implement the following interface:
>
Inversion of control makes a lot of sense!

> 5. All plugins must implement NutchConfigurable.

What you mean with plugins? The plugin class? That is no problem and 
will not break the API.
In case you mean Extension implementation classes we have to port all 
plugin codes. :-/

> Comments?

This changes makes a lot sense and I like them very much.
I may have another improvement suggestion that can be done very easy.

Actually it is difficult to have tools using ndfs and local file system.
What do people think about introducing a ndfs notation in paths like it 
is used in protocol handlers? (ala http:// or file://)
I don't mean to write a protocoll hander for ndfs (this would be nice 
to have) but I just mean something like:

bin/nutch     generate   ndfs://namenode:8010/myNDFSFolder/mydb        
/mylocalsegment/
ndfs path: ndfs://namenode:8010/myNDFSFolder/mydb
local path: /mylocalsegment/

Advantages:
less parameter needed by tools
mixed file systems for tool


One day but this is out of scope now we can have classloading from 
ndfs, with such a syntax. :-)


Greetings,
Stefan



Re: tools cleanup

Posted by Sami Siren <s....@sonera.inet.fi>.
http://jakarta.apache.org/commons/cli/

could this be the way?

--
  Sami Siren


John X wrote:
> On Wed, Mar 30, 2005 at 12:53:24PM -0800, Doug Cutting wrote:
> 
>>2. A tool class should define no methods other than a main() and perhaps 
>>those required to parse the command line.  All application logic should 
>>be in the action class.
>>
> 
> 
> I think command line options should be processed uniformly too.
> There are various styles. We need to agree on one.
> 
> John
> 


Re: tools cleanup

Posted by John X <jo...@neasys.com>.
On Wed, Mar 30, 2005 at 12:53:24PM -0800, Doug Cutting wrote:
> 
> 2. A tool class should define no methods other than a main() and perhaps 
> those required to parse the command line.  All application logic should 
> be in the action class.
> 

I think command line options should be processed uniformly too.
There are various styles. We need to agree on one.

John

RE: tools cleanup

Posted by Chris Mattmann <ch...@jpl.nasa.gov>.
Hi Doug,

> 1. An "action" is an operation on Nutch data.  For example,
> GenerateSegmentFromDB, FetchSegment, UpdateDB, IndexSegment,
> MergeIndexes, SearchServer, etc. are all actions.
> 
> 2. A "tool" invokes an action from the command line.
> 
> The proposal:
> 
> 1. Actions and tools should be separate classes, in separate files.
> 
> 2. A tool class should define no methods other than a main() and perhaps
> those required to parse the command line.  All application logic should
> be in the action class.

I agree with the general distinction here between "tools" and "actions". I
think that it makes sense to separate the action specific code, such as
"update the DB", from typical command line code, e.g., parsing command line
parameters, and setting up the action objects and invoking them. One way
that we've addressed this in the project that I work on at JPL is adding
main() methods to the "action" classes that you define. So, for instance,
you would have:

class Action implements IAction{

     public void method1(){}
      
     public String getSomeAttribute(){}

     //more methods

     public static void main(String [] args){
       //put in your Tool code here
     }
}

So, it's just another way of thinking about it actually. 

> 
> 3. All actions must implement the following interface:
> 
>    public interface NutchConfigurable {
>      void setConf(NutchConf conf);
>      NutchConf getConf();
>    }

Is this so that actions can have different configuration files? I'm not *
that * familiar with the NutchConf class. Does it read a set of default
configuration files? If that is the case, why would you need to pass a
NutchConf to the action?


> 
> 4. Most actions should implement this by extending:
> 
>    public class NutchConfigured implements NutchConfigurable {
>      private NutchConf conf;
>      public NutchConfigured(NutchConf conf) { setConf(conf); }
>      public void setConf(NutchConf conf) { this.conf = conf; }
>      public NutchConf getConf() { return conf; }
>    }
> 
> 5. All plugins must implement NutchConfigurable.
> 
> 6. Plugin factory methods must accept a NutchConf.
> 
> For example:
> 
>    public static Protocol ProtocolFactory.getProtocol(String url);
> 
> will become:
> 
>    public static Protocol ProtocolFactory.getProtocol(NutchConf, String);
> 

I think that this totally makes sense if the NutchConf can have different
resource configuration files for different Actions...

> Comments?
> 
> Doug

Nice Proposal. I think it makes a lot of sense, and should help with some
changes that need to be made I think to the Nutch XML reading code within
the PluginRepository functionality within Nutch (i.e. refactoring the code
to only use one XML API).....


Cheers,
  Chris