You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by MilleBii <mi...@gmail.com> on 2010/01/29 20:45:39 UTC
Could not obtain block
I'm looking for some help. I'm Nutch user, everything was working fine, but
now I get the following error when indexing.
I have a single note pseudo distributed set up.
Some people on the Nutch list indicated to me that I could full, so I remove
many things and hdfs is far from full.
This file & directory was perfectly OK the day before.
I did a "hadoop fsck"... report says healthy.
What can I do ?
Is is safe to do a Linux FSCK just in case ?
Caused by: java.io.IOException: Could not obtain block:
blk_8851198258748412820_9031
file=/user/nutch/crawl/indexed-segments/20100111233601/part-00000/_103.frq
--
-MilleBii-
Re: Could not obtain block
Posted by MilleBii <mi...@gmail.com>.
Ken,
FIXED !!! SO MUCH THANKS
Command prompt ulimit wasn't enough, one needs to hard set it and reboot
explained here
http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/
2010/1/30 MilleBii <mi...@gmail.com>
> Increased the "ulimit" to 64000 ... same problem
> stop/start-all ... same problem but on a different block which of course
> present, so it looks like there is nothing wrong with actual data in the
> hdfs.
>
> I use the Nutch default hadoop 0.19.x anything related ?
>
> 2010/1/30 Ken Goodhope <ke...@gmail.com>
>
> "Could not obtain block" errors are often caused by running out of
>> available
>> file handles. You can confirm this by going to the shell and entering
>> "ulimit -n". If it says 1024, the default, then you will want to increase
>> it to about 64,000.
>>
>> On Fri, Jan 29, 2010 at 4:06 PM, MilleBii <mi...@gmail.com> wrote:
>>
>> > X-POST with Nutch mailing list.
>> >
>> > HEEELP !!!
>> >
>> > Kind of get stuck on this one.
>> > I backed-up my hdfs data, reformated the hdfs, put data back, try to
>> merge
>> > my segments together and it explodes again.
>> >
>> > Exception in thread "Lucene Merge Thread #0"
>> > org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
>> > Could not obtain block: blk_4670839132945043210_1585
>> >
>> file=/user/nutch/crawl/indexed-segments/20100113003609/part-00000/_ym.frq
>> > at
>> >
>> >
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
>> >
>> > If I go into the hfds/data directory I DO find the faulty block ????
>> > Could it be a synchro problem on the segment merger code ?
>> >
>> > 2010/1/29 MilleBii <mi...@gmail.com>
>> >
>> > > I'm looking for some help. I'm Nutch user, everything was working
>> fine,
>> > but
>> > > now I get the following error when indexing.
>> > > I have a single note pseudo distributed set up.
>> > > Some people on the Nutch list indicated to me that I could full, so I
>> > > remove many things and hdfs is far from full.
>> > > This file & directory was perfectly OK the day before.
>> > > I did a "hadoop fsck"... report says healthy.
>> > >
>> > > What can I do ?
>> > >
>> > > Is is safe to do a Linux FSCK just in case ?
>> > >
>> > > Caused by: java.io.IOException: Could not obtain block:
>> > > blk_8851198258748412820_9031
>> > >
>> >
>> file=/user/nutch/crawl/indexed-segments/20100111233601/part-00000/_103.frq
>> > >
>> > >
>> > > --
>> > > -MilleBii-
>> > >
>> >
>> >
>> >
>> > --
>> > -MilleBii-
>> >
>>
>>
>>
>> --
>> Ken Goodhope
>> Cell: 425-750-5616
>>
>> 362 Bellevue Way NE Apt N415
>> Bellevue WA, 98004
>>
>
>
>
> --
> -MilleBii-
>
--
-MilleBii-
Re: Could not obtain block
Posted by MilleBii <mi...@gmail.com>.
Increased the "ulimit" to 64000 ... same problem
stop/start-all ... same problem but on a different block which of course
present, so it looks like there is nothing wrong with actual data in the
hdfs.
I use the Nutch default hadoop 0.19.x anything related ?
2010/1/30 Ken Goodhope <ke...@gmail.com>
> "Could not obtain block" errors are often caused by running out of
> available
> file handles. You can confirm this by going to the shell and entering
> "ulimit -n". If it says 1024, the default, then you will want to increase
> it to about 64,000.
>
> On Fri, Jan 29, 2010 at 4:06 PM, MilleBii <mi...@gmail.com> wrote:
>
> > X-POST with Nutch mailing list.
> >
> > HEEELP !!!
> >
> > Kind of get stuck on this one.
> > I backed-up my hdfs data, reformated the hdfs, put data back, try to
> merge
> > my segments together and it explodes again.
> >
> > Exception in thread "Lucene Merge Thread #0"
> > org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> > Could not obtain block: blk_4670839132945043210_1585
> > file=/user/nutch/crawl/indexed-segments/20100113003609/part-00000/_ym.frq
> > at
> >
> >
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
> >
> > If I go into the hfds/data directory I DO find the faulty block ????
> > Could it be a synchro problem on the segment merger code ?
> >
> > 2010/1/29 MilleBii <mi...@gmail.com>
> >
> > > I'm looking for some help. I'm Nutch user, everything was working fine,
> > but
> > > now I get the following error when indexing.
> > > I have a single note pseudo distributed set up.
> > > Some people on the Nutch list indicated to me that I could full, so I
> > > remove many things and hdfs is far from full.
> > > This file & directory was perfectly OK the day before.
> > > I did a "hadoop fsck"... report says healthy.
> > >
> > > What can I do ?
> > >
> > > Is is safe to do a Linux FSCK just in case ?
> > >
> > > Caused by: java.io.IOException: Could not obtain block:
> > > blk_8851198258748412820_9031
> > >
> >
> file=/user/nutch/crawl/indexed-segments/20100111233601/part-00000/_103.frq
> > >
> > >
> > > --
> > > -MilleBii-
> > >
> >
> >
> >
> > --
> > -MilleBii-
> >
>
>
>
> --
> Ken Goodhope
> Cell: 425-750-5616
>
> 362 Bellevue Way NE Apt N415
> Bellevue WA, 98004
>
--
-MilleBii-
Re: Could not obtain block
Posted by Ken Goodhope <ke...@gmail.com>.
"Could not obtain block" errors are often caused by running out of available
file handles. You can confirm this by going to the shell and entering
"ulimit -n". If it says 1024, the default, then you will want to increase
it to about 64,000.
On Fri, Jan 29, 2010 at 4:06 PM, MilleBii <mi...@gmail.com> wrote:
> X-POST with Nutch mailing list.
>
> HEEELP !!!
>
> Kind of get stuck on this one.
> I backed-up my hdfs data, reformated the hdfs, put data back, try to merge
> my segments together and it explodes again.
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> Could not obtain block: blk_4670839132945043210_1585
> file=/user/nutch/crawl/indexed-segments/20100113003609/part-00000/_ym.frq
> at
>
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
>
> If I go into the hfds/data directory I DO find the faulty block ????
> Could it be a synchro problem on the segment merger code ?
>
> 2010/1/29 MilleBii <mi...@gmail.com>
>
> > I'm looking for some help. I'm Nutch user, everything was working fine,
> but
> > now I get the following error when indexing.
> > I have a single note pseudo distributed set up.
> > Some people on the Nutch list indicated to me that I could full, so I
> > remove many things and hdfs is far from full.
> > This file & directory was perfectly OK the day before.
> > I did a "hadoop fsck"... report says healthy.
> >
> > What can I do ?
> >
> > Is is safe to do a Linux FSCK just in case ?
> >
> > Caused by: java.io.IOException: Could not obtain block:
> > blk_8851198258748412820_9031
> >
> file=/user/nutch/crawl/indexed-segments/20100111233601/part-00000/_103.frq
> >
> >
> > --
> > -MilleBii-
> >
>
>
>
> --
> -MilleBii-
>
--
Ken Goodhope
Cell: 425-750-5616
362 Bellevue Way NE Apt N415
Bellevue WA, 98004
Re: Could not obtain block
Posted by MilleBii <mi...@gmail.com>.
X-POST with Nutch mailing list.
HEEELP !!!
Kind of get stuck on this one.
I backed-up my hdfs data, reformated the hdfs, put data back, try to merge
my segments together and it explodes again.
Exception in thread "Lucene Merge Thread #0"
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
Could not obtain block: blk_4670839132945043210_1585
file=/user/nutch/crawl/indexed-segments/20100113003609/part-00000/_ym.frq
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
If I go into the hfds/data directory I DO find the faulty block ????
Could it be a synchro problem on the segment merger code ?
2010/1/29 MilleBii <mi...@gmail.com>
> I'm looking for some help. I'm Nutch user, everything was working fine, but
> now I get the following error when indexing.
> I have a single note pseudo distributed set up.
> Some people on the Nutch list indicated to me that I could full, so I
> remove many things and hdfs is far from full.
> This file & directory was perfectly OK the day before.
> I did a "hadoop fsck"... report says healthy.
>
> What can I do ?
>
> Is is safe to do a Linux FSCK just in case ?
>
> Caused by: java.io.IOException: Could not obtain block:
> blk_8851198258748412820_9031
> file=/user/nutch/crawl/indexed-segments/20100111233601/part-00000/_103.frq
>
>
> --
> -MilleBii-
>
--
-MilleBii-