You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Sheham Izat <sh...@gmail.com> on 2024/04/19 01:42:24 UTC
Help posting question
Hi,
I'm trying to get Nutch to work and I have issues, how can I post questions
on the group?
Thank you,
Sheham
Re: Help posting question
Posted by Sebastian Nagel <sn...@apache.org>.
Hi Sheham,
the nutch-site.xml configures
<property>
<name>mapreduce.task.timeout</name>
<value>1800</value>
</property>
1.8 seconds (1800 milliseconds) is very short. The default is 600 seconds or 10
minutes, see [1]. Since Nutch needs to finish fetching before the task timeout
applies, threads fetching not quickly enough and still running at the end are
killed.
I would suggest to keep the property "mapreduce.task.timeout" on its default value.
Best,
Sebastian
[1]
https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml#mapreduce.task.timeout
On 4/24/24 16:38, Lewis John McGibbney wrote:
> Hi Sheham,
>
> On 2024/04/20 08:47:41 Sheham Izat wrote:
>
>> The Fetcher job was aborted, does that still mean that it went through the
>> entire list of seed urls?
>
> Yes it processed the entire generated segment but the fetcher…
>
> * hung on https://disneyland.disney.go.com/, https://api.onlyoffice.com/, https://www.adu.com/ and https://www.lowes.com/
> * was denied by robots.txt for https://sourceforge.net/, https://onsclothing.com/, https://kinto-usa.com/, https://twitter.com/, https://www.linkedin.com/, etc.
> * encountered problems processing some robots.txt files for https://twitter.com/, https://www.trustradius.com/
> There may be some other issues encountered buy the fetcher.
>
> This is not at all uncommon. The fetcher completed successfully after 7 seconds. You could progress with your crawl.
>
>>
>> I will go through the mailing list questions.
>
> If you need more assistance please let us know. You will find plenty of pointers on this mailing list archive though.
>
> lewismc
Re: Help posting question
Posted by Lewis John McGibbney <le...@apache.org>.
Hi Sheham,
On 2024/04/20 08:47:41 Sheham Izat wrote:
> The Fetcher job was aborted, does that still mean that it went through the
> entire list of seed urls?
Yes it processed the entire generated segment but the fetcher…
* hung on https://disneyland.disney.go.com/, https://api.onlyoffice.com/, https://www.adu.com/ and https://www.lowes.com/
* was denied by robots.txt for https://sourceforge.net/, https://onsclothing.com/, https://kinto-usa.com/, https://twitter.com/, https://www.linkedin.com/, etc.
* encountered problems processing some robots.txt files for https://twitter.com/, https://www.trustradius.com/
There may be some other issues encountered buy the fetcher.
This is not at all uncommon. The fetcher completed successfully after 7 seconds. You could progress with your crawl.
>
> I will go through the mailing list questions.
If you need more assistance please let us know. You will find plenty of pointers on this mailing list archive though.
lewismc
Re: Help posting question
Posted by Sheham Izat <sh...@gmail.com>.
Hi Lewis,
The Fetcher job was aborted, does that still mean that it went through the
entire list of seed urls?
I will go through the mailing list questions.
Thank you
On Fri, Apr 19, 2024 at 10:15 PM Lewis John McGibbney <le...@apache.org>
wrote:
> Hi Sheham,
>
> On 2024/04/19 15:18:01 Sheham Izat wrote:
> >
> > My questions are:
> >
> > 1) What do I need to do to get Nutch to continue working even if there
> are
> > hung threads?
>
> From what I can see in the log you provided, nothing is preventing Nutch
> from continuing to work. The Fetcher job finished successfully.
>
> > 2) Is there a way to avoid having these hanging threads in the first
> place?
>
> Several factors can lead to hung fetcher threads. Lots of questions have
> been asked on this mailing list relating to exactly this issue. I would
> encourage you to study some of the community responses and see if they
> assist you in a better understanding of the possible issues. You can filter
> questions in the mailing list search with the following criteria
> * date range: more than 1 days ago
> * body: hung
>
> https://lists.apache.org/list.html?user@nutch.apache.org
>
Re: Help posting question
Posted by Lewis John McGibbney <le...@apache.org>.
Hi Sheham,
On 2024/04/19 15:18:01 Sheham Izat wrote:
>
> My questions are:
>
> 1) What do I need to do to get Nutch to continue working even if there are
> hung threads?
From what I can see in the log you provided, nothing is preventing Nutch from continuing to work. The Fetcher job finished successfully.
> 2) Is there a way to avoid having these hanging threads in the first place?
Several factors can lead to hung fetcher threads. Lots of questions have been asked on this mailing list relating to exactly this issue. I would encourage you to study some of the community responses and see if they assist you in a better understanding of the possible issues. You can filter questions in the mailing list search with the following criteria
* date range: more than 1 days ago
* body: hung
https://lists.apache.org/list.html?user@nutch.apache.org
Re: Help posting question
Posted by Sheham Izat <sh...@gmail.com>.
Hi Shashanka, All,
Thank you for your reply!
I'm using Nutch 1.19. I did the injection and segment generation using the
following commands:
bin/nutch inject crawl/crawldb urls
bin/nutch generate crawl/crawldb crawl/segments
When I run the fetch command, Nutch stops with errors about hung threads.
I've attached the fetch command output and the nutch-site.xml.
s1=`ls -d crawl/segments/2* | tail -1`
bin/nutch fetch $s1
My questions are:
1) What do I need to do to get Nutch to continue working even if there are
hung threads?
2) Is there a way to avoid having these hanging threads in the first place?
Thank you
Sheham
On Fri, Apr 19, 2024 at 1:04 AM Shashanka Balakuntala <
shbalakuntala@gmail.com> wrote:
> Hi Shehamizat,
> Please feel free to drop questions on the email itself. One of us/community
> will be glad to help on the same.
>
> *Regards*
> Shashanka Balakuntala Srinivasa
>
>
>
> On Fri, 19 Apr 2024 at 7:15 AM, Sheham Izat <sh...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm trying to get Nutch to work and I have issues, how can I post
> questions
> > on the group?
> >
> > Thank you,
> > Sheham
> >
>
Re: Help posting question
Posted by Shashanka Balakuntala <sh...@gmail.com>.
Hi Shehamizat,
Please feel free to drop questions on the email itself. One of us/community
will be glad to help on the same.
*Regards*
Shashanka Balakuntala Srinivasa
On Fri, 19 Apr 2024 at 7:15 AM, Sheham Izat <sh...@gmail.com> wrote:
> Hi,
>
> I'm trying to get Nutch to work and I have issues, how can I post questions
> on the group?
>
> Thank you,
> Sheham
>