You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Yash Thenuan Thenuan <ri...@iiita.ac.in> on 2018/05/04 11:13:05 UTC
Having plugin as a separate project
Hi all,
I want to compile my plugins separately so that I need not compile
the whole project again when I make a change in some plugin. How can I
achieve that?
Thanks
Re: Having plugin as a separate project
Posted by Yash Thenuan Thenuan <ri...@iiita.ac.in>.
Hey,
Thanks for the answer, But my question was can't we write plugin by
downloading the nutch jar and using it as a dependency, rather than adding
the code in nutch source code?
On Fri, May 4, 2018 at 8:08 PM, Jorge Betancourt <betancourt.jorge@gmail.com
> wrote:
> Usually we tend to develop everything inside the Nutch file structure,
> specially useful if you need to deploy to a Hadoop cluster later on
> (because you need to bundle everything in a job file).
>
> But, if you really want to develop the plugin in isolation you only need to
> create a new project in your preferred IDE/maven/ant/gradle and add the
> dependencies that you need from the lib/ directory (or the global
> dependencies with the same version).
>
> Then just compile everything to a jar and place it in the proper plugin
> structure in the Nutch installation. Although this should work is not
> really a smooth development experience.
> You need to be careful and not bundle all libs inside your jar, etc.
>
> The path suggested by Sebastian is much better, in the end while developing
> you want to have everything, perhaps just compile/test your plugin and
> later on you can copy the final jar of your plugin to the desired Nutch
> installation.
>
> Best Regards,
> Jorge
>
> On Fri, May 4, 2018 at 4:02 PM narendra singh arya <ns...@gmail.com>
> wrote:
>
> > Can we have nutch plugin as a separate project?
> >
> > On Fri, 4 May 2018, 19:26 Sebastian Nagel, <wa...@googlemail.com>
> > wrote:
> >
> > > That's trivial. Just run ant in the plugin's source folder:
> > >
> > > cd src/plugin/urlnormalizer-basic/
> > > ant
> > >
> > > or to run also the tests
> > >
> > > cd src/plugin/urlnormalizer-basic/
> > > ant test
> > >
> > > Note: you have to compile the core test classes first by running
> > >
> > > ant compile-core-test
> > >
> > > in the Nutch "root" folder.
> > >
> > > A little bit slower but guarantees that everything is compiled:
> > >
> > > ant -Dplugin=urlnormalizer-basic test-plugin
> > >
> > > Or sometimes it's enough to skip some of the long running tests:
> > >
> > > ant -Dtest.exclude='TestSegmentMerger*' clean runtime test
> > >
> > >
> > > Best,
> > > Sebastian
> > >
> > > On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote:
> > > > Hi all,
> > > > I want to compile my plugins separately so that I need not compile
> > > > the whole project again when I make a change in some plugin. How can
> I
> > > > achieve that?
> > > > Thanks
> > > >
> > >
> > >
> >
>
Re: Having plugin as a separate project
Posted by Jorge Betancourt <be...@gmail.com>.
Usually we tend to develop everything inside the Nutch file structure,
specially useful if you need to deploy to a Hadoop cluster later on
(because you need to bundle everything in a job file).
But, if you really want to develop the plugin in isolation you only need to
create a new project in your preferred IDE/maven/ant/gradle and add the
dependencies that you need from the lib/ directory (or the global
dependencies with the same version).
Then just compile everything to a jar and place it in the proper plugin
structure in the Nutch installation. Although this should work is not
really a smooth development experience.
You need to be careful and not bundle all libs inside your jar, etc.
The path suggested by Sebastian is much better, in the end while developing
you want to have everything, perhaps just compile/test your plugin and
later on you can copy the final jar of your plugin to the desired Nutch
installation.
Best Regards,
Jorge
On Fri, May 4, 2018 at 4:02 PM narendra singh arya <ns...@gmail.com>
wrote:
> Can we have nutch plugin as a separate project?
>
> On Fri, 4 May 2018, 19:26 Sebastian Nagel, <wa...@googlemail.com>
> wrote:
>
> > That's trivial. Just run ant in the plugin's source folder:
> >
> > cd src/plugin/urlnormalizer-basic/
> > ant
> >
> > or to run also the tests
> >
> > cd src/plugin/urlnormalizer-basic/
> > ant test
> >
> > Note: you have to compile the core test classes first by running
> >
> > ant compile-core-test
> >
> > in the Nutch "root" folder.
> >
> > A little bit slower but guarantees that everything is compiled:
> >
> > ant -Dplugin=urlnormalizer-basic test-plugin
> >
> > Or sometimes it's enough to skip some of the long running tests:
> >
> > ant -Dtest.exclude='TestSegmentMerger*' clean runtime test
> >
> >
> > Best,
> > Sebastian
> >
> > On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote:
> > > Hi all,
> > > I want to compile my plugins separately so that I need not compile
> > > the whole project again when I make a change in some plugin. How can I
> > > achieve that?
> > > Thanks
> > >
> >
> >
>
Re: Having plugin as a separate project
Posted by narendra singh arya <ns...@gmail.com>.
Can we have nutch plugin as a separate project?
On Fri, 4 May 2018, 19:26 Sebastian Nagel, <wa...@googlemail.com>
wrote:
> That's trivial. Just run ant in the plugin's source folder:
>
> cd src/plugin/urlnormalizer-basic/
> ant
>
> or to run also the tests
>
> cd src/plugin/urlnormalizer-basic/
> ant test
>
> Note: you have to compile the core test classes first by running
>
> ant compile-core-test
>
> in the Nutch "root" folder.
>
> A little bit slower but guarantees that everything is compiled:
>
> ant -Dplugin=urlnormalizer-basic test-plugin
>
> Or sometimes it's enough to skip some of the long running tests:
>
> ant -Dtest.exclude='TestSegmentMerger*' clean runtime test
>
>
> Best,
> Sebastian
>
> On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote:
> > Hi all,
> > I want to compile my plugins separately so that I need not compile
> > the whole project again when I make a change in some plugin. How can I
> > achieve that?
> > Thanks
> >
>
>
Re: Having plugin as a separate project
Posted by Sebastian Nagel <wa...@googlemail.com>.
That's trivial. Just run ant in the plugin's source folder:
cd src/plugin/urlnormalizer-basic/
ant
or to run also the tests
cd src/plugin/urlnormalizer-basic/
ant test
Note: you have to compile the core test classes first by running
ant compile-core-test
in the Nutch "root" folder.
A little bit slower but guarantees that everything is compiled:
ant -Dplugin=urlnormalizer-basic test-plugin
Or sometimes it's enough to skip some of the long running tests:
ant -Dtest.exclude='TestSegmentMerger*' clean runtime test
Best,
Sebastian
On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote:
> Hi all,
> I want to compile my plugins separately so that I need not compile
> the whole project again when I make a change in some plugin. How can I
> achieve that?
> Thanks
>
RE: Having plugin as a separate project
Posted by Markus Jelsma <ma...@openindex.io>.
Hi,
Here are examples using Maven:
https://github.com/ATLANTBH/nutch-plugins/tree/master/nutch-plugins
Regards,
Markus
-----Original message-----
> From:Yash Thenuan Thenuan <ri...@iiita.ac.in>
> Sent: Monday 7th May 2018 11:51
> To: user@nutch.apache.org
> Subject: Re: Having plugin as a separate project
>
> Hey,
> Thanks for the answer, But my question was can't we write plugin by
> downloading the nutch jar and using it as a dependency, rather than adding
> the code in nutch source code?
>
> On Fri, May 4, 2018 at 8:08 PM, Jorge Betancourt <betancourt.jorge@gmail.com
> > wrote:
>
> > Usually we tend to develop everything inside the Nutch file structure,
> > specially useful if you need to deploy to a Hadoop cluster later on
> > (because you need to bundle everything in a job file).
> >
> > But, if you really want to develop the plugin in isolation you only need to
> > create a new project in your preferred IDE/maven/ant/gradle and add the
> > dependencies that you need from the lib/ directory (or the global
> > dependencies with the same version).
> >
> > Then just compile everything to a jar and place it in the proper plugin
> > structure in the Nutch installation. Although this should work is not
> > really a smooth development experience.
> > You need to be careful and not bundle all libs inside your jar, etc.
> >
> > The path suggested by Sebastian is much better, in the end while developing
> > you want to have everything, perhaps just compile/test your plugin and
> > later on you can copy the final jar of your plugin to the desired Nutch
> > installation.
> >
> > Best Regards,
> > Jorge
> >
> > On Fri, May 4, 2018 at 4:02 PM narendra singh arya <ns...@gmail.com>
> > wrote:
> >
> > > Can we have nutch plugin as a separate project?
> > >
> > > On Fri, 4 May 2018, 19:26 Sebastian Nagel, <wa...@googlemail.com>
> > > wrote:
> > >
> > > > That's trivial. Just run ant in the plugin's source folder:
> > > >
> > > > cd src/plugin/urlnormalizer-basic/
> > > > ant
> > > >
> > > > or to run also the tests
> > > >
> > > > cd src/plugin/urlnormalizer-basic/
> > > > ant test
> > > >
> > > > Note: you have to compile the core test classes first by running
> > > >
> > > > ant compile-core-test
> > > >
> > > > in the Nutch "root" folder.
> > > >
> > > > A little bit slower but guarantees that everything is compiled:
> > > >
> > > > ant -Dplugin=urlnormalizer-basic test-plugin
> > > >
> > > > Or sometimes it's enough to skip some of the long running tests:
> > > >
> > > > ant -Dtest.exclude='TestSegmentMerger*' clean runtime test
> > > >
> > > >
> > > > Best,
> > > > Sebastian
> > > >
> > > > On 05/04/2018 01:13 PM, Yash Thenuan Thenuan wrote:
> > > > > Hi all,
> > > > > I want to compile my plugins separately so that I need not compile
> > > > > the whole project again when I make a change in some plugin. How can
> > I
> > > > > achieve that?
> > > > > Thanks
> > > > >
> > > >
> > > >
> > >
> >
>