You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by a a <mb...@msn.com> on 2010/11/09 15:28:36 UTC
RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
hi,
Any Ideas ??
mehdi
> From: mbellil@msn.com
> To: user@nutch.apache.org
> Subject: org.apache.hadoop.util.DiskChecker$DiskErrorException
> Date: Fri, 29 Oct 2010 14:44:42 +0000
>
>
> hi,
> i have errors when starting to merge my 9 segments: the error is
>
> mapred.LocalJobRunner - job_local_0001
> org.apache.hadoop.util.DiskChecker$DiskErrorException:
> Could not find any valid local directory for
> attempt_local_0001_r_000000_0/intermediate.8
>
> I have 30 GB of free space for my hadoop.tmp.dir. and all my segments are less than 1 GB
>
> i'm
> suspecting msoffice documents ! since i started indexing pdf and docs
> files i obtain this error and my merge never finish and my invertlinks
> fail because of the missing of
>
> directories in my merged segment
>
> is there anything i can do ?
>
>
> mehdi
>
>
>
RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
Posted by a a <mb...@msn.com>.
hi,
every day i perform a new index. and i could merge the 9 segments without any problem.
but since i added a path /documents (to be crawled) that contains around 150 msoffice documents, i obtain this error.
now i excluded this path and dont obtain the error anymore. but still i'm obliged to index those msoffice documents.
thx for your answer
mehdi
> From: markus.jelsma@openindex.io
> To: user@nutch.apache.org
> Subject: Re: org.apache.hadoop.util.DiskChecker$DiskErrorException
> Date: Fri, 26 Nov 2010 16:53:13 +0100
> CC: mbellil@msn.com
>
> Well yes, but i guess many of us don't have the answer either. Are you capable
> of merging just 2 segments? Which segments fail? What data is in the failed
> segment(s)?
>
> On Friday 26 November 2010 16:40:03 a a wrote:
> > no body is interrested to look at my case ?
> >
> >
> > mehdi
> >
> > > From: mbellil@msn.com
> > > To: user@nutch.apache.org
> > > Subject: RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > > Date: Tue, 9 Nov 2010 14:28:36 +0000
> > >
> > >
> > > hi,
> > >
> > > Any Ideas ??
> > >
> > >
> > >
> > > mehdi
> > >
> > > > From: mbellil@msn.com
> > > > To: user@nutch.apache.org
> > > > Subject: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > > > Date: Fri, 29 Oct 2010 14:44:42 +0000
> > > >
> > > >
> > > > hi,
> > > > i have errors when starting to merge my 9 segments: the error is
> > > >
> > > > mapred.LocalJobRunner - job_local_0001
> > > >
> > > > org.apache.hadoop.util.DiskChecker$DiskErrorException:
> > > > Could not find any valid local directory for
> > > >
> > > > attempt_local_0001_r_000000_0/intermediate.8
> > > >
> > > > I have 30 GB of free space for my hadoop.tmp.dir. and all my segments
> > > > are less than 1 GB
> > > >
> > > > i'm
> > > >
> > > > suspecting msoffice documents ! since i started indexing pdf and docs
> > > >
> > > > files i obtain this error and my merge never finish and my invertlinks
> > > > fail because of the missing of
> > > >
> > > > directories in my merged segment
> > > >
> > > > is there anything i can do ?
> > > >
> > > >
> > > > mehdi
>
> --
> Markus Jelsma - CTO - Openindex
> http://www.linkedin.com/in/markus17
> 050-8536620 / 06-50258350
Re: org.apache.hadoop.util.DiskChecker$DiskErrorException
Posted by Markus Jelsma <ma...@openindex.io>.
Well yes, but i guess many of us don't have the answer either. Are you capable
of merging just 2 segments? Which segments fail? What data is in the failed
segment(s)?
On Friday 26 November 2010 16:40:03 a a wrote:
> no body is interrested to look at my case ?
>
>
> mehdi
>
> > From: mbellil@msn.com
> > To: user@nutch.apache.org
> > Subject: RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > Date: Tue, 9 Nov 2010 14:28:36 +0000
> >
> >
> > hi,
> >
> > Any Ideas ??
> >
> >
> >
> > mehdi
> >
> > > From: mbellil@msn.com
> > > To: user@nutch.apache.org
> > > Subject: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > > Date: Fri, 29 Oct 2010 14:44:42 +0000
> > >
> > >
> > > hi,
> > > i have errors when starting to merge my 9 segments: the error is
> > >
> > > mapred.LocalJobRunner - job_local_0001
> > >
> > > org.apache.hadoop.util.DiskChecker$DiskErrorException:
> > > Could not find any valid local directory for
> > >
> > > attempt_local_0001_r_000000_0/intermediate.8
> > >
> > > I have 30 GB of free space for my hadoop.tmp.dir. and all my segments
> > > are less than 1 GB
> > >
> > > i'm
> > >
> > > suspecting msoffice documents ! since i started indexing pdf and docs
> > >
> > > files i obtain this error and my merge never finish and my invertlinks
> > > fail because of the missing of
> > >
> > > directories in my merged segment
> > >
> > > is there anything i can do ?
> > >
> > >
> > > mehdi
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350
RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
Posted by a a <mb...@msn.com>.
no body is interrested to look at my case ?
mehdi
> From: mbellil@msn.com
> To: user@nutch.apache.org
> Subject: RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
> Date: Tue, 9 Nov 2010 14:28:36 +0000
>
>
> hi,
>
> Any Ideas ??
>
>
>
> mehdi
>
>
>
>
> > From: mbellil@msn.com
> > To: user@nutch.apache.org
> > Subject: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > Date: Fri, 29 Oct 2010 14:44:42 +0000
> >
> >
> > hi,
> > i have errors when starting to merge my 9 segments: the error is
> >
> > mapred.LocalJobRunner - job_local_0001
> > org.apache.hadoop.util.DiskChecker$DiskErrorException:
> > Could not find any valid local directory for
> > attempt_local_0001_r_000000_0/intermediate.8
> >
> > I have 30 GB of free space for my hadoop.tmp.dir. and all my segments are less than 1 GB
> >
> > i'm
> > suspecting msoffice documents ! since i started indexing pdf and docs
> > files i obtain this error and my merge never finish and my invertlinks
> > fail because of the missing of
> >
> > directories in my merged segment
> >
> > is there anything i can do ?
> >
> >
> > mehdi
> >
> >
> >
>
RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
Posted by a a <mb...@msn.com>.
THx for your reply,
i have only 9 segments
but as i said i have enought space, i have 30 GB free. the 9 segments are 397M and i merge them in one segment
thx
mehdi
> From: jafsnano@hotmail.com
> To: user@nutch.apache.org
> Subject: RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
> Date: Tue, 9 Nov 2010 10:24:15 -0600
>
>
> How many segments do you have?
>
> This error is because the space. You can increase the file system space.
>
> > From: mbellil@msn.com
> > To: user@nutch.apache.org
> > Subject: RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > Date: Tue, 9 Nov 2010 14:28:36 +0000
> >
> >
> > hi,
> >
> > Any Ideas ??
> >
> >
> >
> > mehdi
> >
> >
> >
> >
> > > From: mbellil@msn.com
> > > To: user@nutch.apache.org
> > > Subject: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > > Date: Fri, 29 Oct 2010 14:44:42 +0000
> > >
> > >
> > > hi,
> > > i have errors when starting to merge my 9 segments: the error is
> > >
> > > mapred.LocalJobRunner - job_local_0001
> > > org.apache.hadoop.util.DiskChecker$DiskErrorException:
> > > Could not find any valid local directory for
> > > attempt_local_0001_r_000000_0/intermediate.8
> > >
> > > I have 30 GB of free space for my hadoop.tmp.dir. and all my segments are less than 1 GB
> > >
> > > i'm
> > > suspecting msoffice documents ! since i started indexing pdf and docs
> > > files i obtain this error and my merge never finish and my invertlinks
> > > fail because of the missing of
> > >
> > > directories in my merged segment
> > >
> > > is there anything i can do ?
> > >
> > >
> > > mehdi
> > >
> > >
> > >
> >
>
RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
Posted by Juan Felix <ja...@hotmail.com>.
How many segments do you have?
This error is because the space. You can increase the file system space.
> From: mbellil@msn.com
> To: user@nutch.apache.org
> Subject: RE: org.apache.hadoop.util.DiskChecker$DiskErrorException
> Date: Tue, 9 Nov 2010 14:28:36 +0000
>
>
> hi,
>
> Any Ideas ??
>
>
>
> mehdi
>
>
>
>
> > From: mbellil@msn.com
> > To: user@nutch.apache.org
> > Subject: org.apache.hadoop.util.DiskChecker$DiskErrorException
> > Date: Fri, 29 Oct 2010 14:44:42 +0000
> >
> >
> > hi,
> > i have errors when starting to merge my 9 segments: the error is
> >
> > mapred.LocalJobRunner - job_local_0001
> > org.apache.hadoop.util.DiskChecker$DiskErrorException:
> > Could not find any valid local directory for
> > attempt_local_0001_r_000000_0/intermediate.8
> >
> > I have 30 GB of free space for my hadoop.tmp.dir. and all my segments are less than 1 GB
> >
> > i'm
> > suspecting msoffice documents ! since i started indexing pdf and docs
> > files i obtain this error and my merge never finish and my invertlinks
> > fail because of the missing of
> >
> > directories in my merged segment
> >
> > is there anything i can do ?
> >
> >
> > mehdi
> >
> >
> >
>