Re: Nutch 2 hanging after aborting hung threads

[Bai Shen <ba...@gmail.com>]

Just as an update, I did find out that I had forgotten to up the file limit
from 1024.  nproc was already at 19k(or somesuch).  I set files to 20k.

I also had to set max HBase connections to 0.  During a Solr reindex I was
apparently generating over 100 connections from Nutch.

I haven't tried relaxing my other settings to see if the errors and hung
threads come back yet.


On Tue, Apr 30, 2013 at 12:50 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> That would be very much appreciated.
> Lewis
>
>
> On Tue, Apr 30, 2013 at 5:00 AM, Bai Shen <ba...@gmail.com> wrote:
>
> > I'll let you know if I figure out any good defaults.
> >
> > Thanks.
> >
> >
> > On Sat, Apr 27, 2013 at 5:30 PM, Lewis John Mcgibbney <
> > lewis.mcgibbney@gmail.com> wrote:
> >
> > > Hi Bai,
> > >
> > > On Thu, Apr 25, 2013 at 4:33 AM, Bai Shen <ba...@gmail.com>
> > wrote:
> > >
> > > >
> > > > Well, I still ended up having to set a content limit.  Which is why
> I'm
> > > > wondering how the Nutch Gora integration works.  I didn't see a lot
> of
> > > > documentation on it.
> > > >
> > > > So far Nutch seems to be running okay with the changes I made.
> >  However,
> > > I
> > > > left it crawling overnight and came back to find that HBase is maxed
> > out
> > > > memory wise.  Any suggestions for dealing with that?
> > > >
> > > > OK so within your gora-hbase-mapping.xml file you have several
> options
> > > which are available when specifying table properties.
> > > I opened an issue for this over in Gora, simply because as you say this
> > > (critical) aspect of mapping is not explored or documented anywhere.
> > > Please drop in on GORA-218 [0].
> > > I need to say two things,
> > > 1) I don't know what these optional properties are set to by default,
> > > therefore I can advise you on what you are doing right/wrong or what
> you
> > > should do per your use case.
> > > 2) I am NOT HBase literate, I am merely interested in improving the
> code.
> > > If we could get to a stage where such default mapping attributes were
> > > included within the mapping file, set to reasonably sensible default
> > values
> > > (for Nutch use case) and sufficiently documented then I would be
> happier
> > > than I am now.
> > >
> > > Thanks
> > > Lewis
> > >
> > > [0] https://issues.apache.org/jira/browse/GORA-218
> > >
> >
>
>
>
> --
> *Lewis*
>