You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Doug Cook <na...@candiru.com> on 2006/08/21 22:55:50 UTC
Re: Problem with logging of Fetcher output in 0.8-dev
Hi, Ed-
I'm seeing the same problem. If anyone has had a similar experience and
solved it, please let me know. In the mean time, I'll keep investigating and
post back if I figure out what's going wrong.
This may or may not matter, but I'm running everything on a single MP
machine w/o DFS.
Doug
e w wrote:
>
> Logging of the Fetcher output in 0.8-dev used to work (writing to the
> corresponding tasktracker output log) but doesn't appear to any more with
> the nightly build from a couple of weeks ago and also the one from last
> night.
>
> I've enabled DEBUG for the first 4 logging properties in
> conf/log4j.properties and also:
>
> <property>
> <name>fetcher.verbose</name>
> <value>true</value>
> <description>If true, fetcher will log more verbosely.</description>
> </property>
>
> in nutch-site.xml.
>
> Does anyone else have this problem? If not, can anyone suggest a likely
> cause? Thanks,
>
> -Ed
>
>
--
View this message in context: http://www.nabble.com/Problem-with-logging-of-Fetcher-output-in-0.8-dev-tf1999249.html#a5914756
Sent from the Nutch - User forum at Nabble.com.
Re: Problem with logging of Fetcher output in 0.8-dev
Posted by Stefan Groschupf <sg...@101tec.com>.
> I don't know if Chris Schneider's patch for HADOOP-406 will prove
> to be the
> "long-term" solution, but it certainly works for me.
If you like please vote for this issue! I also use it in several
projects and wonder why it is not yet part of hadoop.
Thanks.
Stefan
Re: Problem with logging of Fetcher output in 0.8-dev
Posted by Doug Cook <na...@candiru.com>.
Hi, Ed,
Funny you should choose just now to reply. I just solved the problem on my
own system and was about to post what I found.
This appears to be related to HADOOP-406:
https://issues.apache.org/jira/browse/HADOOP-406
That appears to be why the child JVM fails to inherit hadoop.log.dir, etc.
I applied the patch for that, and then added the following to
mapred.child.java.opts in conf/hadoop-site.xml:
-Dhadoop.log.dir=@property[hadoop.log.dir]@
-Dhadoop.log.file=@property[hadoop.log.file]@
I also made my HADOOP_LOG_DIR environment var an absolute path instead of a
relative one, so that any inheriting processes don't care where they're
running (that did seem to be a problem, too).
I don't know if Chris Schneider's patch for HADOOP-406 will prove to be the
"long-term" solution, but it certainly works for me.
Thanks for following up!
Doug
e w wrote:
>
> Hi Doug,
>
> There was a discussion under the subject "log4j.properties bug (?)" a
> couple
> of weeks back. Please check it out. My (temporary) solution was to
> hardwire
> the "log4j.appender.DRFA.File" variable in log4j.properties to
> hadoop.logand then all the fetcher output from all tasks gets written
> there as well.
> Not an elegant solution but it works for debugging purposes. The problem
> seems to be to do with the environment variable ${hadoop.log.dir} not
> being
> set when log4j.properties is parsed and so nutch tries to write to a file
> in
> "/".
>
> -Ed
>
> On 8/22/06, Doug Cook <na...@candiru.com> wrote:
>>
>>
>> Hi, Ed-
>>
>> I'm seeing the same problem. If anyone has had a similar experience and
>> solved it, please let me know. In the mean time, I'll keep investigating
>> and
>> post back if I figure out what's going wrong.
>>
>> This may or may not matter, but I'm running everything on a single MP
>> machine w/o DFS.
>>
>> Doug
>>
>>
>> e w wrote:
>> >
>> > Logging of the Fetcher output in 0.8-dev used to work (writing to the
>> > corresponding tasktracker output log) but doesn't appear to any more
>> with
>> > the nightly build from a couple of weeks ago and also the one from last
>> > night.
>> >
>> > I've enabled DEBUG for the first 4 logging properties in
>> > conf/log4j.properties and also:
>> >
>> > <property>
>> > <name>fetcher.verbose</name>
>> > <value>true</value>
>> > <description>If true, fetcher will log more verbosely.</description>
>> > </property>
>> >
>> > in nutch-site.xml.
>> >
>> > Does anyone else have this problem? If not, can anyone suggest a likely
>> > cause? Thanks,
>> >
>> > -Ed
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-with-logging-of-Fetcher-output-in-0.8-dev-tf1999249.html#a5914756
>> Sent from the Nutch - User forum at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Problem-with-logging-of-Fetcher-output-in-0.8-dev-tf1999249.html#a5956385
Sent from the Nutch - User forum at Nabble.com.
Re: Problem with logging of Fetcher output in 0.8-dev
Posted by e w <ep...@gmail.com>.
Hi Doug,
There was a discussion under the subject "log4j.properties bug (?)" a couple
of weeks back. Please check it out. My (temporary) solution was to hardwire
the "log4j.appender.DRFA.File" variable in log4j.properties to
hadoop.logand then all the fetcher output from all tasks gets written
there as well.
Not an elegant solution but it works for debugging purposes. The problem
seems to be to do with the environment variable ${hadoop.log.dir} not being
set when log4j.properties is parsed and so nutch tries to write to a file in
"/".
-Ed
On 8/22/06, Doug Cook <na...@candiru.com> wrote:
>
>
> Hi, Ed-
>
> I'm seeing the same problem. If anyone has had a similar experience and
> solved it, please let me know. In the mean time, I'll keep investigating
> and
> post back if I figure out what's going wrong.
>
> This may or may not matter, but I'm running everything on a single MP
> machine w/o DFS.
>
> Doug
>
>
> e w wrote:
> >
> > Logging of the Fetcher output in 0.8-dev used to work (writing to the
> > corresponding tasktracker output log) but doesn't appear to any more
> with
> > the nightly build from a couple of weeks ago and also the one from last
> > night.
> >
> > I've enabled DEBUG for the first 4 logging properties in
> > conf/log4j.properties and also:
> >
> > <property>
> > <name>fetcher.verbose</name>
> > <value>true</value>
> > <description>If true, fetcher will log more verbosely.</description>
> > </property>
> >
> > in nutch-site.xml.
> >
> > Does anyone else have this problem? If not, can anyone suggest a likely
> > cause? Thanks,
> >
> > -Ed
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Problem-with-logging-of-Fetcher-output-in-0.8-dev-tf1999249.html#a5914756
> Sent from the Nutch - User forum at Nabble.com.
>
>