You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Silvio Heuberger <si...@hsr.ch> on 2008/12/02 10:29:47 UTC
Ideal development environment for nutch plugins?
OK, here's supposedly an easy one:
What is the ideal setup to develop nutch plugins. I'm an eclipse-lover,
so I set myself up with a checkout of the 0.9-branch.
The eclipse project is set-up fine and I have been able to use the
debugger to trace where the code goes and where it doesn't.
I then ran the ant build from console. Now I'm getting sorta random
behaviour. Sometimes URLNormalizer cannot be found. I fix that by
readjusting the plugins directory.
Anyhow, what's with the nutch-default.xml and nutch-site.xml? I figured
nutch-site.xml overrides default, but I think that is not the case right
now.
After running the ant build, running inside Eclipse b0rks with several
errors. URLs are not picked up anymore an so on...
So how do I have to set things up, to enable small incremental
iterations of the form:
- Write skeleton for plugin (configs + code)
- write UnitTest
- add URLs that should use the plugin
- debug the code in eclipse
- use nutch-bean to verify the results (not sure about this. Is there a
better way??)
- use ant to deploy.
- Start over again.
Thanks in advance.
Re: AW: Ideal development environment for nutch plugins?
Posted by Silvio Heuberger <si...@hsr.ch>.
Hi Martina
With the current trunk this works much better than before. The only
thing I do not quite understand, is why some libraries for some plugins
have been seprated from both the lib folder and the plugins lib folder.
Took me some time to get the build path straight.
Now that it's set up like intended things work out nicely. I'm happily
coding away my parsers in Groovy and include them as linked libraries
into the main plugin File. I think I could even get away without the
java classes, but this is nice enough right now.
I have a few more questions, about lucene document fields, but I won't
bother this thread. Maybe later.
Thanks, I think I'm fine now.
Silvio
Koch Martina wrote:
> Hi Silvio,
>
> you're right, the plugins are compiled into tmp_build, but the relevant plugin.xml files are in the src/plugin sub-folders. By setting the value of the plugin.folders property in the nutch-site.xml file to "./src/plugin" it picks up the plugin.xml information from there.
> Did you change this property from the default value "plugins"?
>
> Hope this helps.
>
> Kind regards,
> Martina
>
AW: Ideal development environment for nutch plugins?
Posted by Koch Martina <Ko...@huberverlag.de>.
Hi Silvio,
you're right, the plugins are compiled into tmp_build, but the relevant plugin.xml files are in the src/plugin sub-folders. By setting the value of the plugin.folders property in the nutch-site.xml file to "./src/plugin" it picks up the plugin.xml information from there.
Did you change this property from the default value "plugins"?
Hope this helps.
Kind regards,
Martina
-----Ursprüngliche Nachricht-----
Von: Silvio Heuberger [mailto:silvio.heuberger@hsr.ch]
Gesendet: 03 December 2008 11:45
An: nutch-user@lucene.apache.org
Betreff: Re: AW: Ideal development environment for nutch plugins?
The main problem seems to be, that since the plugin java source folders
are on the eclipse build path, they end up compiled into tmp_build.
Now they are missing the plugin.xml file, and as a result are not picked
up by the PluginManifester class.
I'm trying to figure out a way, to rectify this situation.
Martina (or anyone else), have you solved this before?
Greetz
Silvio
Silvio Heuberger wrote:
> Hi Martina
>
> I knew that changes to the config files would not be picked up until a
> full rebuild of the project (or as a matter of fact, until I copy them
> over to the tmp_build directory.)
> What substantially contributes to my current headache is the fact, that
> suddenly I had ClassNotFoundExceptions and now the crawler ignores urls,
> that it did not ignore as of yesterday.
> I will definitely have to set my own SVN repo up, for the work on the
> plugin. Then I can tag versions that do run, just in case something goes
> bad again. Right now, the crawler is useless if started from Eclipse. If
> started from the command line, it works as inteded, however the plugin
> does not yet. I figure this is because the special fields I extract from
> the documents are not yet added as fields to the lucene documents.
> But as long as Nutch is not debuggable, there is no telling...
>
Re: AW: Ideal development environment for nutch plugins?
Posted by Silvio Heuberger <si...@hsr.ch>.
The main problem seems to be, that since the plugin java source folders
are on the eclipse build path, they end up compiled into tmp_build.
Now they are missing the plugin.xml file, and as a result are not picked
up by the PluginManifester class.
I'm trying to figure out a way, to rectify this situation.
Martina (or anyone else), have you solved this before?
Greetz
Silvio
Silvio Heuberger wrote:
> Hi Martina
>
> I knew that changes to the config files would not be picked up until a
> full rebuild of the project (or as a matter of fact, until I copy them
> over to the tmp_build directory.)
> What substantially contributes to my current headache is the fact, that
> suddenly I had ClassNotFoundExceptions and now the crawler ignores urls,
> that it did not ignore as of yesterday.
> I will definitely have to set my own SVN repo up, for the work on the
> plugin. Then I can tag versions that do run, just in case something goes
> bad again. Right now, the crawler is useless if started from Eclipse. If
> started from the command line, it works as inteded, however the plugin
> does not yet. I figure this is because the special fields I extract from
> the documents are not yet added as fields to the lucene documents.
> But as long as Nutch is not debuggable, there is no telling...
>
Re: AW: Ideal development environment for nutch plugins?
Posted by Silvio Heuberger <si...@hsr.ch>.
Hi Martina
I knew that changes to the config files would not be picked up until a
full rebuild of the project (or as a matter of fact, until I copy them
over to the tmp_build directory.)
What substantially contributes to my current headache is the fact, that
suddenly I had ClassNotFoundExceptions and now the crawler ignores urls,
that it did not ignore as of yesterday.
I will definitely have to set my own SVN repo up, for the work on the
plugin. Then I can tag versions that do run, just in case something goes
bad again. Right now, the crawler is useless if started from Eclipse. If
started from the command line, it works as inteded, however the plugin
does not yet. I figure this is because the special fields I extract from
the documents are not yet added as fields to the lucene documents.
But as long as Nutch is not debuggable, there is no telling...
Koch Martina wrote:
> Hi Silvio,
>
> I'm not sure, if I understood you correctly, but you might have a problem with the conf file locations.
> When I run Nutch within Eclipse not the standard conf directory in <NUTCH_HOME>/conf is used, instead the conf files in the build directory (<NUTCH_HOME>/tmp_build, if you followed the instructions in the wiki) are used. The conf files in the tmp_build directory are a copy of the conf directory and are copied during build. If you made changes to the files in conf, these are not considered, if you didn't do a clean and build afterwards.
>
> Hope this helps.
>
> Martina
>
>
> -----Ursprüngliche Nachricht-----
> Von: Silvio Heuberger [mailto:silvio.heuberger@hsr.ch]
> Gesendet: 02 December 2008 11:17
> An: nutch-user@lucene.apache.org
> Betreff: Re: Ideal development environment for nutch plugins?
>
> Let me clarify this:
> The behavious of running the Crawl class from within eclipse is
> radically different, than from command line.
>
> Silvio Heuberger wrote:
>
>> OK, here's supposedly an easy one:
>>
>> What is the ideal setup to develop nutch plugins. I'm an eclipse-lover,
>> so I set myself up with a checkout of the 0.9-branch.
>> The eclipse project is set-up fine and I have been able to use the
>> debugger to trace where the code goes and where it doesn't.
>> I then ran the ant build from console. Now I'm getting sorta random
>> behaviour. Sometimes URLNormalizer cannot be found. I fix that by
>> readjusting the plugins directory.
>> Anyhow, what's with the nutch-default.xml and nutch-site.xml? I figured
>> nutch-site.xml overrides default, but I think that is not the case right
>> now.
>>
>> After running the ant build, running inside Eclipse b0rks with several
>> errors. URLs are not picked up anymore an so on...
>> So how do I have to set things up, to enable small incremental
>> iterations of the form:
>>
>> - Write skeleton for plugin (configs + code)
>> - write UnitTest
>> - add URLs that should use the plugin
>> - debug the code in eclipse
>> - use nutch-bean to verify the results (not sure about this. Is there a
>> better way??)
>> - use ant to deploy.
>> - Start over again.
>>
>> Thanks in advance.
>>
>>
>
>
AW: Ideal development environment for nutch plugins?
Posted by Koch Martina <Ko...@huberverlag.de>.
Hi Silvio,
I'm not sure, if I understood you correctly, but you might have a problem with the conf file locations.
When I run Nutch within Eclipse not the standard conf directory in <NUTCH_HOME>/conf is used, instead the conf files in the build directory (<NUTCH_HOME>/tmp_build, if you followed the instructions in the wiki) are used. The conf files in the tmp_build directory are a copy of the conf directory and are copied during build. If you made changes to the files in conf, these are not considered, if you didn't do a clean and build afterwards.
Hope this helps.
Martina
-----Ursprüngliche Nachricht-----
Von: Silvio Heuberger [mailto:silvio.heuberger@hsr.ch]
Gesendet: 02 December 2008 11:17
An: nutch-user@lucene.apache.org
Betreff: Re: Ideal development environment for nutch plugins?
Let me clarify this:
The behavious of running the Crawl class from within eclipse is
radically different, than from command line.
Silvio Heuberger wrote:
> OK, here's supposedly an easy one:
>
> What is the ideal setup to develop nutch plugins. I'm an eclipse-lover,
> so I set myself up with a checkout of the 0.9-branch.
> The eclipse project is set-up fine and I have been able to use the
> debugger to trace where the code goes and where it doesn't.
> I then ran the ant build from console. Now I'm getting sorta random
> behaviour. Sometimes URLNormalizer cannot be found. I fix that by
> readjusting the plugins directory.
> Anyhow, what's with the nutch-default.xml and nutch-site.xml? I figured
> nutch-site.xml overrides default, but I think that is not the case right
> now.
>
> After running the ant build, running inside Eclipse b0rks with several
> errors. URLs are not picked up anymore an so on...
> So how do I have to set things up, to enable small incremental
> iterations of the form:
>
> - Write skeleton for plugin (configs + code)
> - write UnitTest
> - add URLs that should use the plugin
> - debug the code in eclipse
> - use nutch-bean to verify the results (not sure about this. Is there a
> better way??)
> - use ant to deploy.
> - Start over again.
>
> Thanks in advance.
>
Re: Ideal development environment for nutch plugins?
Posted by Silvio Heuberger <si...@hsr.ch>.
Let me clarify this:
The behavious of running the Crawl class from within eclipse is
radically different, than from command line.
Silvio Heuberger wrote:
> OK, here's supposedly an easy one:
>
> What is the ideal setup to develop nutch plugins. I'm an eclipse-lover,
> so I set myself up with a checkout of the 0.9-branch.
> The eclipse project is set-up fine and I have been able to use the
> debugger to trace where the code goes and where it doesn't.
> I then ran the ant build from console. Now I'm getting sorta random
> behaviour. Sometimes URLNormalizer cannot be found. I fix that by
> readjusting the plugins directory.
> Anyhow, what's with the nutch-default.xml and nutch-site.xml? I figured
> nutch-site.xml overrides default, but I think that is not the case right
> now.
>
> After running the ant build, running inside Eclipse b0rks with several
> errors. URLs are not picked up anymore an so on...
> So how do I have to set things up, to enable small incremental
> iterations of the form:
>
> - Write skeleton for plugin (configs + code)
> - write UnitTest
> - add URLs that should use the plugin
> - debug the code in eclipse
> - use nutch-bean to verify the results (not sure about this. Is there a
> better way??)
> - use ant to deploy.
> - Start over again.
>
> Thanks in advance.
>