You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by srinivasarao v <sr...@gmail.com> on 2009/01/06 16:23:10 UTC
Indexing problem
Hi all,
I am trying to crawl a website with depth 5. But while indexing,
it is throwing an error "java.io.FileNotFoundException:
/tmp/hadoop/mapred/local/index/_-1541990873/_17.tis (Too many open files)".
Can anyone reply me with the solution?
Thanks
Srinivas
Re: Indexing problem
Posted by vishal vachhani <vi...@gmail.com>.
you can reduce the maxMerge and minMerge factors in nutch-default.xml file.
On Wed, Jan 7, 2009 at 9:48 AM, srinivasarao v <sr...@gmail.com>wrote:
> thanx Laurent...
>
> On Wed, Jan 7, 2009 at 6:00 AM, Laurent Laborde <ke...@gmail.com>
> wrote:
>
> > Sure...
> > you have too many open file :)
> > you can increase it using ulimit.
> > check the limit with ulimit -a
> > if i remember correctly the command for open file is "ulimit -n"
> > Default is usually 1024.
> >
> >
> > On Tue, Jan 6, 2009 at 4:23 PM, srinivasarao v
> > <sr...@gmail.com> wrote:
> > > Hi all,
> > > I am trying to crawl a website with depth 5. But while
> indexing,
> > > it is throwing an error "java.io.FileNotFoundException:
> > > /tmp/hadoop/mapred/local/index/_-1541990873/_17.tis (Too many open
> > files)".
> > > Can anyone reply me with the solution?
> >
> > Sure...
> > you have too many open file :)
> > you can increase it using ulimit.
> > check the limit with ulimit -a
> > if i remember correctly the command for max open file is "ulimit -n"
> > Default is usually 1024and not enough for hadoop, depending of the
> > configuration.
> >
> > --
> > F4FQM
> > Kerunix Flan
> > Laurent Laborde
> >
>
--
Thanks and Regards,
Vishal Vachhani
M.tech, CSE dept
Indian Institute of Technology, Bombay
http://www.cse.iitb.ac.in/~vishalv
Re: Indexing problem
Posted by srinivasarao v <sr...@gmail.com>.
thanx Laurent...
On Wed, Jan 7, 2009 at 6:00 AM, Laurent Laborde <ke...@gmail.com> wrote:
> Sure...
> you have too many open file :)
> you can increase it using ulimit.
> check the limit with ulimit -a
> if i remember correctly the command for open file is "ulimit -n"
> Default is usually 1024.
>
>
> On Tue, Jan 6, 2009 at 4:23 PM, srinivasarao v
> <sr...@gmail.com> wrote:
> > Hi all,
> > I am trying to crawl a website with depth 5. But while indexing,
> > it is throwing an error "java.io.FileNotFoundException:
> > /tmp/hadoop/mapred/local/index/_-1541990873/_17.tis (Too many open
> files)".
> > Can anyone reply me with the solution?
>
> Sure...
> you have too many open file :)
> you can increase it using ulimit.
> check the limit with ulimit -a
> if i remember correctly the command for max open file is "ulimit -n"
> Default is usually 1024and not enough for hadoop, depending of the
> configuration.
>
> --
> F4FQM
> Kerunix Flan
> Laurent Laborde
>
Re: Indexing problem
Posted by Laurent Laborde <ke...@gmail.com>.
Sure...
you have too many open file :)
you can increase it using ulimit.
check the limit with ulimit -a
if i remember correctly the command for open file is "ulimit -n"
Default is usually 1024.
On Tue, Jan 6, 2009 at 4:23 PM, srinivasarao v
<sr...@gmail.com> wrote:
> Hi all,
> I am trying to crawl a website with depth 5. But while indexing,
> it is throwing an error "java.io.FileNotFoundException:
> /tmp/hadoop/mapred/local/index/_-1541990873/_17.tis (Too many open files)".
> Can anyone reply me with the solution?
Sure...
you have too many open file :)
you can increase it using ulimit.
check the limit with ulimit -a
if i remember correctly the command for max open file is "ulimit -n"
Default is usually 1024and not enough for hadoop, depending of the
configuration.
--
F4FQM
Kerunix Flan
Laurent Laborde