You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Apache Wiki <wi...@apache.org> on 2010/09/13 03:11:04 UTC
[Nutch Wiki] Update of "Crawl" by cmd
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "Crawl" page has been changed by cmd.
The comment on this change is: Does anybody can help me translate it to windows bat script. thanks. .
http://wiki.apache.org/nutch/Crawl?action=diff&rev1=10&rev2=11
--------------------------------------------------
The complete job of this script has been divided broadly into 8 steps.
1. Inject URLs
- 2. Generate, Fetch, Parse, Update Loop
+ 1. Generate, Fetch, Parse, Update Loop
- 3. Merge Segments
+ 1. Merge Segments
- 4. Invert Links
+ 1. Invert Links
- 5. Index
+ 1. Index
- 6. Dedup
+ 1. Dedup
- 7. Merge Indexes
+ 1. Merge Indexes
- 8. Load new indexes
+ 1. Load new indexes
== Modes of Execution ==
The script can be executed in two modes:-
+
* Normal Mode
* Safe Mode
@@ -42, +43 @@
then
NUTCH_HOME=.
}}}
-
Set 'NUTCH_HOME' to the path of the Nutch directory (if you are not setting it as an environment variable, since if environment variable is set, the above assignment is ignored).
=== CATALINA_HOME ===
@@ -53, +53 @@
then
CATALINA_HOME=/opt/apache-tomcat-6.0.10
}}}
-
Similar to the previous section, if this variable is set in the environment, then the above assignment is ignored.
== Can it re-crawl? ==
The author has used this script to re-crawl a couple of times. However, no real world testing has been done for re-crawling. Therefore, you may try to use the script for re-crawl. If it works fine or it doesn't work properly for re-crawl, please let us know.
== Script ==
- {{{
- #!/bin/sh
+ {{{#!/bin/sh
# runbot script to run the Nutch bot for crawling and re-crawling.
# Usage: bin/runbot [safe]
@@ -90, +88 @@
then
NUTCH_HOME=.
echo runbot: $0 could not find environment variable NUTCH_HOME
- echo runbot: NUTCH_HOME=$NUTCH_HOME has been set by the script
+ echo runbot: NUTCH_HOME=$NUTCH_HOME has been set by the script
else
- echo runbot: $0 found environment variable NUTCH_HOME=$NUTCH_HOME
+ echo runbot: $0 found environment variable NUTCH_HOME=$NUTCH_HOME
fi
if [ -z "$CATALINA_HOME" ]
then
CATALINA_HOME=/opt/apache-tomcat-6.0.10
echo runbot: $0 could not find environment variable NUTCH_HOME
- echo runbot: CATALINA_HOME=$CATALINA_HOME has been set by the script
+ echo runbot: CATALINA_HOME=$CATALINA_HOME has been set by the script
else
- echo runbot: $0 found environment variable CATALINA_HOME=$CATALINA_HOME
+ echo runbot: $0 found environment variable CATALINA_HOME=$CATALINA_HOME
fi
if [ -n "$topN" ]