You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Vijith <vi...@gmail.com> on 2012/05/10 13:14:41 UTC
Running nutch in eclipse
Hi,
I have successfully built and ran nutch in eclipse in local mode.
Is it possible to run nutch in deploy mode from eclipse.
--
*Thanks & Regards*
*
*
*Vijith V*
Re: Running nutch in eclipse
Posted by Vijith <vi...@gmail.com>.
Ok I got it. Its there in userlog folder in each of the slaves..
On Fri, May 11, 2012 at 12:28 AM, Vijith <vi...@gmail.com> wrote:
> Regarding the nutch log, i think i missed out something while running the
> job. In eclipse I have given the following as VM arguments
> - -Dhadoop.log.dir=logs -Dhadoop.log.file=nutch.log (local mode)
> How should i do the same while running the job file.
>
>
> On Thu, May 10, 2012 at 11:43 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi,
>>
>> Re your first couple of points please see this very informative
>> conversation
>>
>> http://www.mail-archive.com/user@nutch.apache.org/msg00538.html
>>
>> On Thu, May 10, 2012 at 5:22 PM, Vijith <vi...@gmail.com> wrote:
>>
>> > - Also i am not finding the nutch log file
>>
>> As you would expect, you are running Nutch jobs as part of Hadoop,
>> therefore your logs will be in your Hadoop setup (JT). Please see this
>> issue if you wish to submit a patch to change this
>> https://issues.apache.org/jira/browse/NUTCH-342
>>
>> > - Will it in anyway affect the distributed run if we are using some
>> > external libraries like Jena
>>
>> This is a near impossible question for us to answer given the real
>> brief nature of the description
>>
>> >
>> > - right now my filters are reading the ontology files and creating jena
>> > objects every time the filters are executed
>> > - how can i make this faster; is there any caching mechanism available
>>
>> For what?
>>
>
>
>
> --
> *Thanks & Regards*
> *
> *
> *Vijith V*
>
>
>
--
*Thanks & Regards*
*
*
*Vijith V*
Re: Running nutch in eclipse
Posted by Vijith <vi...@gmail.com>.
Regarding the nutch log, i think i missed out something while running the
job. In eclipse I have given the following as VM arguments
- -Dhadoop.log.dir=logs -Dhadoop.log.file=nutch.log (local mode)
How should i do the same while running the job file.
On Thu, May 10, 2012 at 11:43 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> Hi,
>
> Re your first couple of points please see this very informative
> conversation
>
> http://www.mail-archive.com/user@nutch.apache.org/msg00538.html
>
> On Thu, May 10, 2012 at 5:22 PM, Vijith <vi...@gmail.com> wrote:
>
> > - Also i am not finding the nutch log file
>
> As you would expect, you are running Nutch jobs as part of Hadoop,
> therefore your logs will be in your Hadoop setup (JT). Please see this
> issue if you wish to submit a patch to change this
> https://issues.apache.org/jira/browse/NUTCH-342
>
> > - Will it in anyway affect the distributed run if we are using some
> > external libraries like Jena
>
> This is a near impossible question for us to answer given the real
> brief nature of the description
>
> >
> > - right now my filters are reading the ontology files and creating jena
> > objects every time the filters are executed
> > - how can i make this faster; is there any caching mechanism available
>
> For what?
>
--
*Thanks & Regards*
*
*
*Vijith V*
Re: Running nutch in eclipse
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,
Re your first couple of points please see this very informative conversation
http://www.mail-archive.com/user@nutch.apache.org/msg00538.html
On Thu, May 10, 2012 at 5:22 PM, Vijith <vi...@gmail.com> wrote:
> - Also i am not finding the nutch log file
As you would expect, you are running Nutch jobs as part of Hadoop,
therefore your logs will be in your Hadoop setup (JT). Please see this
issue if you wish to submit a patch to change this
https://issues.apache.org/jira/browse/NUTCH-342
> - Will it in anyway affect the distributed run if we are using some
> external libraries like Jena
This is a near impossible question for us to answer given the real
brief nature of the description
>
> - right now my filters are reading the ontology files and creating jena
> objects every time the filters are executed
> - how can i make this faster; is there any caching mechanism available
For what?
Re: Running nutch in eclipse
Posted by Vijith <vi...@gmail.com>.
Thanks for the info. I will give it a try and will let you know if its
working.
For the time being I have tried running the nutch job file in the cluster.
It ran slower than that in local mode.
As to me it looks like the job ran without any errors. I have some
questions regarding deploy mode execution...
- Is there any recommended hadoop config for running nutch, like # of
map/reduce tasks etc, so that it runs faster/efficiently
- Also i am not finding the nutch log file
- Will it in anyway affect the distributed run if we are using some
external libraries like Jena
- right now my filters are reading the ontology files and creating jena
objects every time the filters are executed
- how can i make this faster; is there any caching mechanism available
correct me if i am wrong.
On Thu, May 10, 2012 at 7:32 PM, Julien Nioche <
lists.digitalpebble@gmail.com> wrote:
> There are already quite a few tutorials on how to use Eclipse and Hadoop,
> see for instance
>
> http://developer.yahoo.com/hadoop/tutorial/module3.html
> http://ebiquity.umbc.edu/Tutorials/Hadoop/00%20-%20Intro.html
>
>
> On 10 May 2012 14:41, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com
> >wrote:
>
> > Hi Vijith,
> >
> > I don't suppose this would be a problem (not that I have tried),
> > however I suspect it would require you to have your JT on the same
> > node as from where you run eclipse, and also for your convenience for
> > you to import hadoop on the JT node into the eclipse workspace as
> > well.
> >
> > Please let us know how you get on as this would be nice to add to the
> > eclipse tutorial.
> >
> > Lewis
> >
> > On Thu, May 10, 2012 at 12:14 PM, Vijith <vi...@gmail.com> wrote:
> > > Hi,
> > > I have successfully built and ran nutch in eclipse in local mode.
> > > Is it possible to run nutch in deploy mode from eclipse.
> > >
> > > --
> > > *Thanks & Regards*
> > > *
> > > *
> > > *Vijith V*
> >
> >
> >
> > --
> > Lewis
> >
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
--
*Thanks & Regards*
*
*
*Vijith V*
Re: Running nutch in eclipse
Posted by Julien Nioche <li...@gmail.com>.
There are already quite a few tutorials on how to use Eclipse and Hadoop,
see for instance
http://developer.yahoo.com/hadoop/tutorial/module3.html
http://ebiquity.umbc.edu/Tutorials/Hadoop/00%20-%20Intro.html
On 10 May 2012 14:41, Lewis John Mcgibbney <le...@gmail.com>wrote:
> Hi Vijith,
>
> I don't suppose this would be a problem (not that I have tried),
> however I suspect it would require you to have your JT on the same
> node as from where you run eclipse, and also for your convenience for
> you to import hadoop on the JT node into the eclipse workspace as
> well.
>
> Please let us know how you get on as this would be nice to add to the
> eclipse tutorial.
>
> Lewis
>
> On Thu, May 10, 2012 at 12:14 PM, Vijith <vi...@gmail.com> wrote:
> > Hi,
> > I have successfully built and ran nutch in eclipse in local mode.
> > Is it possible to run nutch in deploy mode from eclipse.
> >
> > --
> > *Thanks & Regards*
> > *
> > *
> > *Vijith V*
>
>
>
> --
> Lewis
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Running nutch in eclipse
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Vijith,
I don't suppose this would be a problem (not that I have tried),
however I suspect it would require you to have your JT on the same
node as from where you run eclipse, and also for your convenience for
you to import hadoop on the JT node into the eclipse workspace as
well.
Please let us know how you get on as this would be nice to add to the
eclipse tutorial.
Lewis
On Thu, May 10, 2012 at 12:14 PM, Vijith <vi...@gmail.com> wrote:
> Hi,
> I have successfully built and ran nutch in eclipse in local mode.
> Is it possible to run nutch in deploy mode from eclipse.
>
> --
> *Thanks & Regards*
> *
> *
> *Vijith V*
--
Lewis