You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Gajanan Watkar <ga...@gmail.com> on 2018/09/05 05:57:21 UTC

Nodemanager crashing repeatedly

I am running Nutch-2.3.1 over Hadoop-2.5.2 and Hbase-1.2.3 with
integration to Solr-6.5.1. I have crawled over 10 million pages. But
while doing all this I am continuously facing two problems:

1. My Nodemanager is crashing repeatedly during different phases of
crawl. It crashes my linux session and forces logout with nodemanager
killed. I log-in again, restart NodeManger and the same failed crawl
phase runs to success. [Nodemanager log has nothing to report]

2. I am running all my crawl phases one by one without crawl script, as
with crawl script most of the time my jobs were exiting with
"WaitForjobCompletion" error at different stages of crawl. So, I
decided to go ahead with one by one method which prevented
"WaitForjobCompletion" to occure.

Any help will be highly appreciated. New to mailing-list, New to Nutch.

-Gajanan