You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Murat Ali Bayir <mu...@agmlab.com> on 2006/07/05 16:05:23 UTC
logging
I couldn't get any log output from nutch nightly, when I run ./nutch
from linux shell.
what can be the problem?
RE: logging
Posted by Teruhiko Kurosaka <Ku...@basistech.com>.
For how to configure log4j, take a look at:
http://logging.apache.org/log4j/docs/manual.html
> > How can I change redirection in log4j.propertires file?
> >
> As mentionned in my previous mail, simply by editing the file
> $NUTCH_HOME/conf/log4j.properties.
Re: logging
Posted by Jérôme Charron <je...@gmail.com>.
> How can I change redirection in log4j.propertires file?
>
As mentionned in my previous mail, simply by editing the file
$NUTCH_HOME/conf/log4j.properties.
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/
Re: logging
Posted by Murat Ali Bayir <mu...@agmlab.com>.
Murat Ali Bayir wrote:
> Jérôme Charron wrote:
>
>> all the logs are now redirected to $NUTCH_HOME/log dir
>> you can edit the $NUTCH_HOME/conf/log4j.properties to change the logging
>> behavior.
>>
>> Regards
>>
>> Jérôme
>>
>> On 7/5/06, Honda-Search Administrator <ad...@honda-search.com> wrote:
>>
>>>
>>> please post your syntax/commands/script used.
>>>
>>> ----- Original Message -----
>>> From: "Murat Ali Bayir" <mu...@agmlab.com>
>>> To: <nu...@lucene.apache.org>
>>> Sent: Wednesday, July 05, 2006 7:05 AM
>>> Subject: logging
>>>
>>>
>>> >I couldn't get any log output from nutch nightly, when I run ./nutch
>>> > from linux shell.
>>> > what can be the problem?
>>> >
>>> >
>>>
>>
>>
>>
> How can I change redirection in log4j.propertires file?
> my script is ./nutch mergedb ...
>
>
Here is content of My script
#!/bin/bash
#
# The Nutch command script
#
# Environment Variables
#
# NUTCH_JAVA_HOME The java implementation to use. Overrides JAVA_HOME.
#
# NUTCH_HEAPSIZE The maximum amount of heap to use, in MB.
# Default is 1000.
#
# NUTCH_OPTS Extra Java runtime options.
#
# resolve links - $0 may be a softlink
THIS="$0"
while [ -h "$THIS" ]; do
ls=`ls -ld "$THIS"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '.*/.*' > /dev/null; then
THIS="$link"
else
THIS=`dirname "$THIS"`/"$link"
fi
done
# if no args specified, show usage
if [ $# = 0 ]; then
echo "Usage: nutch COMMAND"
echo "where COMMAND is one of:"
echo " crawl one-step crawler for intranets"
echo " readdb read / dump crawl db"
echo " mergedb merge crawldb-s, with optional filtering"
echo " readlinkdb read / dump link db"
echo " inject inject new urls into the database"
echo " generate generate new segments to fetch"
echo " fetch fetch a segment's pages"
echo " parse parse a segment's pages"
echo " segread read / dump segment data"
echo " mergesegs merge several segments, with optional
filtering and slicing"
echo " updatedb update crawl db from segments after fetching"
echo " invertlinks create a linkdb from parsed segments"
echo " mergelinkdb merge linkdb-s, with optional filtering"
echo " index run the indexer on parsed segments and linkdb"
echo " merge merge several segment indexes"
echo " dedup remove duplicates from a set of segment indexes"
echo " plugin load a plugin and run one of its classes main()"
echo " server run a search server"
echo " or"
echo " CLASSNAME run the class named CLASSNAME"
echo "Most commands print help when invoked w/o parameters."
exit 1
fi
# get arguments
COMMAND=$1
shift
# some directories
THIS_DIR=`dirname "$THIS"`
NUTCH_HOME=`cd "$THIS_DIR/.." ; pwd`
# some Java parameters
if [ "$NUTCH_JAVA_HOME" != "" ]; then
#echo "run java in $NUTCH_JAVA_HOME"
JAVA_HOME=$NUTCH_JAVA_HOME
fi
if [ "$JAVA_HOME" = "" ]; then
echo "Error: JAVA_HOME is not set."
exit 1
fi
JAVA=$JAVA_HOME/bin/java
JAVA_HEAP_MAX=-Xmx1000m
# check envvars which might override default args
if [ "$NUTCH_HEAPSIZE" != "" ]; then
#echo "run with heapsize $NUTCH_HEAPSIZE"
JAVA_HEAP_MAX="-Xmx""$NUTCH_HEAPSIZE""m"
#echo $JAVA_HEAP_MAX
fi
# CLASSPATH initially contains $NUTCH_CONF_DIR, or defaults to
$NUTCH_HOME/conf
CLASSPATH=${NUTCH_CONF_DIR:=$NUTCH_HOME/conf}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
# so that filenames w/ spaces are handled correctly in loops below
IFS=
# for developers, add plugins, job & test code to CLASSPATH
if [ -d "$NUTCH_HOME/build/plugins" ]; then
CLASSPATH=${CLASSPATH}:$NUTCH_HOME/build
fi
for f in $NUTCH_HOME/build/nutch-*.job; do
CLASSPATH=${CLASSPATH}:$f;
done
if [ -d "$NUTCH_HOME/build/test/classes" ]; then
CLASSPATH=${CLASSPATH}:$NUTCH_HOME/build/test/classes
fi
# for releases, add Nutch job to CLASSPATH
for f in $NUTCH_HOME/nutch-*.job; do
CLASSPATH=${CLASSPATH}:$f;
done
# add plugins to classpath
if [ -d "$NUTCH_HOME/plugins" ]; then
CLASSPATH=${NUTCH_HOME}:${CLASSPATH}
fi
# add libs to CLASSPATH
for f in $NUTCH_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
for f in $NUTCH_HOME/lib/jetty-ext/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
# restore ordinary behaviour
unset IFS
# default log directory & file
if [ "$NUTCH_LOG_DIR" = "" ]; then
NUTCH_LOG_DIR="$NUTCH_HOME/logs"
fi
if [ "$NUTCH_LOGFILE" = "" ]; then
NUTCH_LOGFILE='hadoop.log'
fi
NUTCH_OPTS="$NUTCH_OPTS -Dhadoop.log.dir=$NUTCH_LOG_DIR"
NUTCH_OPTS="$NUTCH_OPTS -Dhadoop.log.file=$NUTCH_LOGFILE"
# figure out which class to run
if [ "$COMMAND" = "crawl" ] ; then
CLASS=org.apache.nutch.crawl.Crawl
elif [ "$COMMAND" = "inject" ] ; then
CLASS=org.apache.nutch.crawl.Injector
elif [ "$COMMAND" = "generate" ] ; then
CLASS=org.apache.nutch.crawl.Generator
elif [ "$COMMAND" = "fetch" ] ; then
CLASS=org.apache.nutch.fetcher.Fetcher
elif [ "$COMMAND" = "parse" ] ; then
CLASS=org.apache.nutch.parse.ParseSegment
elif [ "$COMMAND" = "readdb" ] ; then
CLASS=org.apache.nutch.crawl.CrawlDbReader
elif [ "$COMMAND" = "mergedb" ] ; then
CLASS=org.apache.nutch.crawl.CrawlDbMerger
elif [ "$COMMAND" = "readlinkdb" ] ; then
CLASS=org.apache.nutch.crawl.LinkDbReader
elif [ "$COMMAND" = "segread" ] ; then
CLASS=org.apache.nutch.segment.SegmentReader
elif [ "$COMMAND" = "mergesegs" ] ; then
CLASS=org.apache.nutch.segment.SegmentMerger
elif [ "$COMMAND" = "updatedb" ] ; then
CLASS=org.apache.nutch.crawl.CrawlDb
elif [ "$COMMAND" = "invertlinks" ] ; then
CLASS=org.apache.nutch.crawl.LinkDb
elif [ "$COMMAND" = "mergelinkdb" ] ; then
CLASS=org.apache.nutch.crawl.LinkDbMerger
elif [ "$COMMAND" = "index" ] ; then
CLASS=org.apache.nutch.indexer.Indexer
elif [ "$COMMAND" = "dedup" ] ; then
CLASS=org.apache.nutch.indexer.DeleteDuplicates
elif [ "$COMMAND" = "merge" ] ; then
CLASS=org.apache.nutch.indexer.IndexMerger
elif [ "$COMMAND" = "plugin" ] ; then
CLASS=org.apache.nutch.plugin.PluginRepository
elif [ "$COMMAND" = "server" ] ; then
CLASS='org.apache.nutch.searcher.DistributedSearch$Server'
else
CLASS=$COMMAND
fi
# cygwin path translation
if expr `uname` : 'CYGWIN*' > /dev/null; then
CLASSPATH=`cygpath -p -w "$CLASSPATH"`
fi
# run it
exec "$JAVA" $JAVA_HEAP_MAX $NUTCH_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> .
>
Re: logging
Posted by Murat Ali Bayir <mu...@agmlab.com>.
Jérôme Charron wrote:
> all the logs are now redirected to $NUTCH_HOME/log dir
> you can edit the $NUTCH_HOME/conf/log4j.properties to change the logging
> behavior.
>
> Regards
>
> Jérôme
>
> On 7/5/06, Honda-Search Administrator <ad...@honda-search.com> wrote:
>
>>
>> please post your syntax/commands/script used.
>>
>> ----- Original Message -----
>> From: "Murat Ali Bayir" <mu...@agmlab.com>
>> To: <nu...@lucene.apache.org>
>> Sent: Wednesday, July 05, 2006 7:05 AM
>> Subject: logging
>>
>>
>> >I couldn't get any log output from nutch nightly, when I run ./nutch
>> > from linux shell.
>> > what can be the problem?
>> >
>> >
>>
>
>
>
How can I change redirection in log4j.propertires file?
my script is ./nutch mergedb ...
Re: logging
Posted by Jérôme Charron <je...@gmail.com>.
all the logs are now redirected to $NUTCH_HOME/log dir
you can edit the $NUTCH_HOME/conf/log4j.properties to change the logging
behavior.
Regards
Jérôme
On 7/5/06, Honda-Search Administrator <ad...@honda-search.com> wrote:
>
> please post your syntax/commands/script used.
>
> ----- Original Message -----
> From: "Murat Ali Bayir" <mu...@agmlab.com>
> To: <nu...@lucene.apache.org>
> Sent: Wednesday, July 05, 2006 7:05 AM
> Subject: logging
>
>
> >I couldn't get any log output from nutch nightly, when I run ./nutch
> > from linux shell.
> > what can be the problem?
> >
> >
>
--
http://motrech.free.fr/
http://www.frutch.org/
Re: logging
Posted by Honda-Search Administrator <ad...@honda-search.com>.
please post your syntax/commands/script used.
----- Original Message -----
From: "Murat Ali Bayir" <mu...@agmlab.com>
To: <nu...@lucene.apache.org>
Sent: Wednesday, July 05, 2006 7:05 AM
Subject: logging
>I couldn't get any log output from nutch nightly, when I run ./nutch
> from linux shell.
> what can be the problem?
>
>