You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Josh Attenberg <jo...@gmail.com> on 2007/11/21 04:24:57 UTC
No space left on device
I had this error when fetching with nutch 0.8.1 there is ~450GB left on the
disk where the crawl db and segments folder. Are there any other settings i
need to make? I know there isnt much space in my home directory, if it was
trying to write there, but there is at least 500M. what are the possible
culprits/fixes?
Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException:
No space left on device
at
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
LocalFileSystem.java:150)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
FSDataOutputStream.java:112)
at java.io.BufferedOutputStream.flushBuffer(
BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(
FSDataOutputStream.java:96)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:154)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
LocalFileSystem.java:311)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:254)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:327)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:443)
at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
LocalFileSystem.java:148)
... 16 more
Re: No space left on device
Posted by Tim Gautier <ti...@gmail.com>.
When you get the "job failed" error, that just means the hadoop job
has failed. It doesn't contain the reason why. For that you'll need
to look in the hadoop.log file where you will find much more
information on what went wrong. As for your space problem, it's been
a while since I set up my config files so I can't recall much off the
top of my head, but my hadoop-site.xml file has these properties in
it:
dfs.data.dir
mapred.local.dir
mapred.system.dir
mapred.temp.dir
Maybe one of them is the culprit.
On Nov 22, 2007 10:55 PM, Susam Pal <su...@gmail.com> wrote:
> It is not required in 'conf/nutch-site.xml'. I made a mistake in
> writing 'conf/nutch-site.xml' in my second post in this thread. It is
> required only in 'conf/hadoop-site.xml'. It seems you are pointing it
> to a partition which doesn't have enough space. My temporary directory
> lies in a partition with about 20 GB free space and I never face a
> problem.
>
> Regards,
> Susam Pal
>
>
> On Nov 23, 2007 4:32 AM, Josh Attenberg <jo...@gmail.com> wrote:
> > i have added
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/opt/tmp</value>
> > <description>Base for Nutch Temporary Directories</description>
> > </property>
> > (with opt/tmp/ changed to an appropriate directory) in both the
> > nutch-site.xml and hadoop-site.xml files. I still get out of space errors
> > right away when trying to crawl. there must be some other configuration
> > property that i am forgetting. can anyone tell me what this is?
> > cheers,
> > Josh
> >
> > On Nov 20, 2007 11:58 PM, Josh Attenberg <jo...@gmail.com> wrote:
> >
> > > i did as you say, and moved the files to a new directory on a big drive,
> > > but now have some additional errors. are there any other pointers i need to
> > > update?
> > >
> > >
> > > On Nov 20, 2007 11:33 PM, Susam Pal < susam.pal@gmail.com> wrote:
> > >
> > > > Is /tmp present in a partition that doesn't have enough space? Does it
> > > > have enough space left when this error occurs? Nutch often needs GBs
> > > > of space for /tmp. If there isn't enough space on the partition having
> > > > /tmp, then you can add the following property in
> > > > 'conf/hadoop- site.xml' to make it use a different directory for
> > > > writing the temporary files.
> > > >
> > > > <property>
> > > > <name>hadoop.tmp.dir</name>
> > > > <value>/opt/tmp</value>
> > > > <description>Base for Nutch Temporary Directories</description>
> > > > </property>
> > > >
> > > > Regards,
> > > > Susam Pal
> > > >
> > > > On Nov 21, 2007 8:54 AM, Josh Attenberg <josh.attenberg@gmail.com >
> > > > wrote:
> > > > > I had this error when fetching with nutch 0.8.1 there is ~450GB left
> > > > on the
> > > > > disk where the crawl db and segments folder. Are there any other
> > > > settings i
> > > > > need to make? I know there isnt much space in my home directory, if it
> > > > was
> > > > > trying to write there, but there is at least 500M. what are the
> > > > possible
> > > > > culprits/fixes?
> > > > >
> > > > >
> > > > > Exception in thread "main" org.apache.hadoop.fs.FSError:
> > > > java.io.IOException:
> > > > > No space left on device
> > > > > at
> > > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > > > LocalFileSystem.java:150)
> > > > > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> > > > > FSDataOutputStream.java:112)
> > > > > at java.io.BufferedOutputStream.flushBuffer(
> > > > > BufferedOutputStream.java:65)
> > > > > at java.io.BufferedOutputStream.flush(
> > > > BufferedOutputStream.java:123)
> > > > > at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> > > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > > > :140)
> > > > > at org.apache.hadoop.fs.FSDataOutputStream$Summer.close (
> > > > > FSDataOutputStream.java:96)
> > > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > > > :143)
> > > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > > > :143)
> > > > > at java.io.FilterOutputStream.close (FilterOutputStream.java
> > > > :143)
> > > > > at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java
> > > > :154)
> > > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> > > > > at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile (
> > > > > LocalFileSystem.java:311)
> > > > > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java
> > > > :254)
> > > > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java
> > > > :327)
> > > > > at org.apache.nutch.fetcher.Fetcher.fetch (Fetcher.java:443)
> > > > > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> > > > > Caused by: java.io.IOException: No space left on device
> > > > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > > > at java.io.FileOutputStream.write(FileOutputStream.java:260)
> > > > > at
> > > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > > > LocalFileSystem.java:148)
> > > > > ... 16 more
> > > > >
> > > >
> > >
> > >
> >
>
Re: No space left on device
Posted by Susam Pal <su...@gmail.com>.
It is not required in 'conf/nutch-site.xml'. I made a mistake in
writing 'conf/nutch-site.xml' in my second post in this thread. It is
required only in 'conf/hadoop-site.xml'. It seems you are pointing it
to a partition which doesn't have enough space. My temporary directory
lies in a partition with about 20 GB free space and I never face a
problem.
Regards,
Susam Pal
On Nov 23, 2007 4:32 AM, Josh Attenberg <jo...@gmail.com> wrote:
> i have added
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/opt/tmp</value>
> <description>Base for Nutch Temporary Directories</description>
> </property>
> (with opt/tmp/ changed to an appropriate directory) in both the
> nutch-site.xml and hadoop-site.xml files. I still get out of space errors
> right away when trying to crawl. there must be some other configuration
> property that i am forgetting. can anyone tell me what this is?
> cheers,
> Josh
>
> On Nov 20, 2007 11:58 PM, Josh Attenberg <jo...@gmail.com> wrote:
>
> > i did as you say, and moved the files to a new directory on a big drive,
> > but now have some additional errors. are there any other pointers i need to
> > update?
> >
> >
> > On Nov 20, 2007 11:33 PM, Susam Pal < susam.pal@gmail.com> wrote:
> >
> > > Is /tmp present in a partition that doesn't have enough space? Does it
> > > have enough space left when this error occurs? Nutch often needs GBs
> > > of space for /tmp. If there isn't enough space on the partition having
> > > /tmp, then you can add the following property in
> > > 'conf/hadoop- site.xml' to make it use a different directory for
> > > writing the temporary files.
> > >
> > > <property>
> > > <name>hadoop.tmp.dir</name>
> > > <value>/opt/tmp</value>
> > > <description>Base for Nutch Temporary Directories</description>
> > > </property>
> > >
> > > Regards,
> > > Susam Pal
> > >
> > > On Nov 21, 2007 8:54 AM, Josh Attenberg <josh.attenberg@gmail.com >
> > > wrote:
> > > > I had this error when fetching with nutch 0.8.1 there is ~450GB left
> > > on the
> > > > disk where the crawl db and segments folder. Are there any other
> > > settings i
> > > > need to make? I know there isnt much space in my home directory, if it
> > > was
> > > > trying to write there, but there is at least 500M. what are the
> > > possible
> > > > culprits/fixes?
> > > >
> > > >
> > > > Exception in thread "main" org.apache.hadoop.fs.FSError:
> > > java.io.IOException:
> > > > No space left on device
> > > > at
> > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > > LocalFileSystem.java:150)
> > > > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> > > > FSDataOutputStream.java:112)
> > > > at java.io.BufferedOutputStream.flushBuffer(
> > > > BufferedOutputStream.java:65)
> > > > at java.io.BufferedOutputStream.flush(
> > > BufferedOutputStream.java:123)
> > > > at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > > :140)
> > > > at org.apache.hadoop.fs.FSDataOutputStream$Summer.close (
> > > > FSDataOutputStream.java:96)
> > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > > :143)
> > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > > :143)
> > > > at java.io.FilterOutputStream.close (FilterOutputStream.java
> > > :143)
> > > > at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java
> > > :154)
> > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> > > > at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile (
> > > > LocalFileSystem.java:311)
> > > > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java
> > > :254)
> > > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java
> > > :327)
> > > > at org.apache.nutch.fetcher.Fetcher.fetch (Fetcher.java:443)
> > > > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> > > > Caused by: java.io.IOException: No space left on device
> > > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > > at java.io.FileOutputStream.write(FileOutputStream.java:260)
> > > > at
> > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > > LocalFileSystem.java:148)
> > > > ... 16 more
> > > >
> > >
> >
> >
>
Re: No space left on device
Posted by Josh Attenberg <jo...@gmail.com>.
i have added
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/tmp</value>
<description>Base for Nutch Temporary Directories</description>
</property>
(with opt/tmp/ changed to an appropriate directory) in both the
nutch-site.xml and hadoop-site.xml files. I still get out of space errors
right away when trying to crawl. there must be some other configuration
property that i am forgetting. can anyone tell me what this is?
cheers,
Josh
On Nov 20, 2007 11:58 PM, Josh Attenberg <jo...@gmail.com> wrote:
> i did as you say, and moved the files to a new directory on a big drive,
> but now have some additional errors. are there any other pointers i need to
> update?
>
>
> On Nov 20, 2007 11:33 PM, Susam Pal < susam.pal@gmail.com> wrote:
>
> > Is /tmp present in a partition that doesn't have enough space? Does it
> > have enough space left when this error occurs? Nutch often needs GBs
> > of space for /tmp. If there isn't enough space on the partition having
> > /tmp, then you can add the following property in
> > 'conf/hadoop- site.xml' to make it use a different directory for
> > writing the temporary files.
> >
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/opt/tmp</value>
> > <description>Base for Nutch Temporary Directories</description>
> > </property>
> >
> > Regards,
> > Susam Pal
> >
> > On Nov 21, 2007 8:54 AM, Josh Attenberg <josh.attenberg@gmail.com >
> > wrote:
> > > I had this error when fetching with nutch 0.8.1 there is ~450GB left
> > on the
> > > disk where the crawl db and segments folder. Are there any other
> > settings i
> > > need to make? I know there isnt much space in my home directory, if it
> > was
> > > trying to write there, but there is at least 500M. what are the
> > possible
> > > culprits/fixes?
> > >
> > >
> > > Exception in thread "main" org.apache.hadoop.fs.FSError:
> > java.io.IOException:
> > > No space left on device
> > > at
> > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > LocalFileSystem.java:150)
> > > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> > > FSDataOutputStream.java:112)
> > > at java.io.BufferedOutputStream.flushBuffer(
> > > BufferedOutputStream.java:65)
> > > at java.io.BufferedOutputStream.flush(
> > BufferedOutputStream.java:123)
> > > at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > :140)
> > > at org.apache.hadoop.fs.FSDataOutputStream$Summer.close (
> > > FSDataOutputStream.java:96)
> > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > :143)
> > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> > :143)
> > > at java.io.FilterOutputStream.close (FilterOutputStream.java
> > :143)
> > > at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java
> > :154)
> > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> > > at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile (
> > > LocalFileSystem.java:311)
> > > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java
> > :254)
> > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java
> > :327)
> > > at org.apache.nutch.fetcher.Fetcher.fetch (Fetcher.java:443)
> > > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> > > Caused by: java.io.IOException: No space left on device
> > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > at java.io.FileOutputStream.write(FileOutputStream.java:260)
> > > at
> > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > LocalFileSystem.java:148)
> > > ... 16 more
> > >
> >
>
>
Re: No space left on device
Posted by Josh Attenberg <jo...@gmail.com>.
I have fixed this problem, seemingly just by clearing my tmp folder, but now
whenever i try to updatedb i get:
CrawlDb update: starting
CrawlDb update: db: db
CrawlDb update: segment: segments/20071121062359
CrawlDb update: Merging segment data into db.
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:357)
at org.apache.nutch.crawl.CrawlDb.update(CrawlDb.java:62)
at org.apache.nutch.crawl.CrawlDb.main(CrawlDb.java:116)
just job failed, no good error message.
On Nov 21, 2007 8:25 AM, Josh Attenberg <jo...@gmail.com> wrote:
> it appears Nutch is still looking for
> /tmp/hadoop/mapred/system/submit_xiq66r/job.jar
> I moved stuff around when i probably shouldnt have, I have copied this
> folder (/tmp/hadoop/) elsewhere, so there still must be a variable to set. i
> get this error during inject:
>
> Injector: Converting injected urls to crawl db entries.
> Exception in thread "main" java.io.FileNotFoundException:
> /tmp/hadoop/mapred/system/submit_xiq66r/job.jar (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
> at org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.<init>(
> LocalFileSystem.java:133)
> at org.apache.hadoop.fs.LocalFileSystem.createRaw(
> LocalFileSystem.java:172)
> at org.apache.hadoop.fs.LocalFileSystem.createRaw(
> LocalFileSystem.java:180)
> at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(
> FSDataOutputStream.java:56)
> at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(
> FSDataOutputStream.java:45)
> at org.apache.hadoop.fs.FSDataOutputStream .<init>(
> FSDataOutputStream.java:146)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:270)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:177)
> at org.apache.hadoop.fs.FileUtil.copy (FileUtil.java:74)
> at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
> LocalFileSystem.java:311)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java
> :254)
> at org.apache.hadoop.mapred.JobClient.runJob (JobClient.java:327)
> at org.apache.nutch.crawl.Injector.inject(Injector.java:138)
> at org.apache.nutch.crawl.Injector.main(Injector.java:164)
>
>
> On Nov 21, 2007 12:09 AM, Susam Pal < susam.pal@gmail.com> wrote:
>
> > I haven't asked you to move the files to a new location. I don't know
> > if moving the files work. My solution was for a fresh crawl. If the
> > partition which contains /tmp doesn't have enough space, you can point
> > Nutch to a different temporary directory by adding this property to
> > your 'conf/nutch-site.xml' and do a new crawl.
> >
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/opt/tmp</value>
> > <description>Base for Nutch Temporary Directories</description>
> > </property>
> >
> > Please note that /opt/tmp is only an example. Change it to whatever is
> > required on your system. Please post the relevant portions of the
> > error logs too when an error occurs.
> >
> > Regards,
> > Susam Pal
> >
> > On Nov 21, 2007 10:28 AM, Josh Attenberg <jo...@gmail.com>
> > wrote:
> > > i did as you say, and moved the files to a new directory on a big
> > drive, but
> > > now have some additional errors. are there any other pointers i need
> > to
> > > update?
> > >
> > >
> > > On Nov 20, 2007 11:33 PM, Susam Pal < susam.pal@gmail.com> wrote:
> > >
> > > > Is /tmp present in a partition that doesn't have enough space? Does
> > it
> > > > have enough space left when this error occurs? Nutch often needs GBs
> >
> > > > of space for /tmp. If there isn't enough space on the partition
> > having
> > > > /tmp, then you can add the following property in
> > > > 'conf/hadoop-site.xml' to make it use a different directory for
> > > > writing the temporary files.
> > > >
> > > > <property>
> > > > <name>hadoop.tmp.dir</name>
> > > > <value>/opt/tmp</value>
> > > > <description>Base for Nutch Temporary Directories</description>
> > > > </property>
> > > >
> > > > Regards,
> > > > Susam Pal
> > > >
> > > > On Nov 21, 2007 8:54 AM, Josh Attenberg <josh.attenberg@gmail.com >
> > wrote:
> > > > > I had this error when fetching with nutch 0.8.1 there is ~450GB
> > left on
> > > > the
> > > > > disk where the crawl db and segments folder. Are there any other
> > > > settings i
> > > > > need to make? I know there isnt much space in my home directory,
> > if it
> > > > was
> > > > > trying to write there, but there is at least 500M. what are the
> > possible
> > > > > culprits/fixes?
> > > > >
> > > > >
> > > > > Exception in thread "main" org.apache.hadoop.fs.FSError:
> > > > java.io.IOException:
> > > > > No space left on device
> > > > > at
> > > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write
> > (
> > > > > LocalFileSystem.java:150)
> > > > > at
> > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> > > > > FSDataOutputStream.java:112)
> > > > > at java.io.BufferedOutputStream.flushBuffer(
> > > > > BufferedOutputStream.java:65)
> > > > > at java.io.BufferedOutputStream.flush (
> > BufferedOutputStream.java
> > > > :123)
> > > > > at java.io.DataOutputStream.flush(DataOutputStream.java
> > :106)
> > > > > at java.io.FilterOutputStream.close(
> > FilterOutputStream.java:140)
> > > > > at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(
> > > > > FSDataOutputStream.java:96)
> > > > > at java.io.FilterOutputStream.close(
> > FilterOutputStream.java:143)
> > > > > at java.io.FilterOutputStream.close(
> > FilterOutputStream.java:143)
> > > > > at java.io.FilterOutputStream.close(
> > FilterOutputStream.java:143)
> > > > > at org.apache.hadoop.fs.FileUtil.copyContent (
> > FileUtil.java:154)
> > > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> > > > > at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
> > > > > LocalFileSystem.java :311)
> > > > > at org.apache.hadoop.mapred.JobClient.submitJob(
> > JobClient.java
> > > > :254)
> > > > > at org.apache.hadoop.mapred.JobClient.runJob(
> > JobClient.java:327)
> > > > > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java
> > :443)
> > > > > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> > > > > Caused by: java.io.IOException: No space left on device
> > > > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > > > at java.io.FileOutputStream.write(FileOutputStream.java
> > :260)
> > > > > at
> > > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > > > LocalFileSystem.java:148)
> > > > > ... 16 more
> > > > >
> > > >
> > >
> >
>
>
Re: No space left on device
Posted by Josh Attenberg <jo...@gmail.com>.
it appears Nutch is still looking for
/tmp/hadoop/mapred/system/submit_xiq66r/job.jar
I moved stuff around when i probably shouldnt have, I have copied this
folder (/tmp/hadoop/) elsewhere, so there still must be a variable to set. i
get this error during inject:
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.io.FileNotFoundException:
/tmp/hadoop/mapred/system/submit_xiq66r/job.jar (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream
.<init>(LocalFileSystem.java:133)
at org.apache.hadoop.fs.LocalFileSystem.createRaw(
LocalFileSystem.java:172)
at org.apache.hadoop.fs.LocalFileSystem.createRaw(
LocalFileSystem.java:180)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(
FSDataOutputStream.java:56)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.<init>(
FSDataOutputStream.java:45)
at org.apache.hadoop.fs.FSDataOutputStream.<init>(
FSDataOutputStream.java:146)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:270)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:177)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
LocalFileSystem.java:311)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:254)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:327)
at org.apache.nutch.crawl.Injector.inject(Injector.java:138)
at org.apache.nutch.crawl.Injector.main(Injector.java:164)
On Nov 21, 2007 12:09 AM, Susam Pal <su...@gmail.com> wrote:
> I haven't asked you to move the files to a new location. I don't know
> if moving the files work. My solution was for a fresh crawl. If the
> partition which contains /tmp doesn't have enough space, you can point
> Nutch to a different temporary directory by adding this property to
> your 'conf/nutch-site.xml' and do a new crawl.
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/opt/tmp</value>
> <description>Base for Nutch Temporary Directories</description>
> </property>
>
> Please note that /opt/tmp is only an example. Change it to whatever is
> required on your system. Please post the relevant portions of the
> error logs too when an error occurs.
>
> Regards,
> Susam Pal
>
> On Nov 21, 2007 10:28 AM, Josh Attenberg <jo...@gmail.com> wrote:
> > i did as you say, and moved the files to a new directory on a big drive,
> but
> > now have some additional errors. are there any other pointers i need to
> > update?
> >
> >
> > On Nov 20, 2007 11:33 PM, Susam Pal <su...@gmail.com> wrote:
> >
> > > Is /tmp present in a partition that doesn't have enough space? Does it
> > > have enough space left when this error occurs? Nutch often needs GBs
> > > of space for /tmp. If there isn't enough space on the partition having
> > > /tmp, then you can add the following property in
> > > 'conf/hadoop-site.xml' to make it use a different directory for
> > > writing the temporary files.
> > >
> > > <property>
> > > <name>hadoop.tmp.dir</name>
> > > <value>/opt/tmp</value>
> > > <description>Base for Nutch Temporary Directories</description>
> > > </property>
> > >
> > > Regards,
> > > Susam Pal
> > >
> > > On Nov 21, 2007 8:54 AM, Josh Attenberg <jo...@gmail.com>
> wrote:
> > > > I had this error when fetching with nutch 0.8.1 there is ~450GB left
> on
> > > the
> > > > disk where the crawl db and segments folder. Are there any other
> > > settings i
> > > > need to make? I know there isnt much space in my home directory, if
> it
> > > was
> > > > trying to write there, but there is at least 500M. what are the
> possible
> > > > culprits/fixes?
> > > >
> > > >
> > > > Exception in thread "main" org.apache.hadoop.fs.FSError:
> > > java.io.IOException:
> > > > No space left on device
> > > > at
> > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > > LocalFileSystem.java:150)
> > > > at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> > > > FSDataOutputStream.java:112)
> > > > at java.io.BufferedOutputStream.flushBuffer(
> > > > BufferedOutputStream.java:65)
> > > > at java.io.BufferedOutputStream.flush(
> BufferedOutputStream.java
> > > :123)
> > > > at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> :140)
> > > > at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(
> > > > FSDataOutputStream.java:96)
> > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> :143)
> > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> :143)
> > > > at java.io.FilterOutputStream.close(FilterOutputStream.java
> :143)
> > > > at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java
> :154)
> > > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> > > > at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
> > > > LocalFileSystem.java:311)
> > > > at org.apache.hadoop.mapred.JobClient.submitJob(
> JobClient.java
> > > :254)
> > > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java
> :327)
> > > > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:443)
> > > > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> > > > Caused by: java.io.IOException: No space left on device
> > > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > > at java.io.FileOutputStream.write(FileOutputStream.java:260)
> > > > at
> > > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > > LocalFileSystem.java:148)
> > > > ... 16 more
> > > >
> > >
> >
>
Re: No space left on device
Posted by Lyndon Maydwell <ma...@gmail.com>.
Hi list.
I've recently moved the location of the temp-files for my crawls to a
20GB partition specifically for this purpose. I'm once again getting
this warning. Is there something that can be done to make Nutch more
efficient with it's temp files?
-- stuff:
Nutch version
Release 0.9 - 2007-04-02
java -version
java version "1.6.0_01"
Java(TM) SE Runtime Environment (build 1.6.0_01-b06)
Java HotSpot(TM) Client VM (build 1.6.0_01-b06, mixed mode, sharing)
uname -a
Linux ceimst1 2.6.16.53-0.16-default #1 Tue Oct 2 16:57:49 UTC 2007
i686 athlon i386 GNU/Linux
Re: No space left on device
Posted by Susam Pal <su...@gmail.com>.
I haven't asked you to move the files to a new location. I don't know
if moving the files work. My solution was for a fresh crawl. If the
partition which contains /tmp doesn't have enough space, you can point
Nutch to a different temporary directory by adding this property to
your 'conf/nutch-site.xml' and do a new crawl.
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/tmp</value>
<description>Base for Nutch Temporary Directories</description>
</property>
Please note that /opt/tmp is only an example. Change it to whatever is
required on your system. Please post the relevant portions of the
error logs too when an error occurs.
Regards,
Susam Pal
On Nov 21, 2007 10:28 AM, Josh Attenberg <jo...@gmail.com> wrote:
> i did as you say, and moved the files to a new directory on a big drive, but
> now have some additional errors. are there any other pointers i need to
> update?
>
>
> On Nov 20, 2007 11:33 PM, Susam Pal <su...@gmail.com> wrote:
>
> > Is /tmp present in a partition that doesn't have enough space? Does it
> > have enough space left when this error occurs? Nutch often needs GBs
> > of space for /tmp. If there isn't enough space on the partition having
> > /tmp, then you can add the following property in
> > 'conf/hadoop-site.xml' to make it use a different directory for
> > writing the temporary files.
> >
> > <property>
> > <name>hadoop.tmp.dir</name>
> > <value>/opt/tmp</value>
> > <description>Base for Nutch Temporary Directories</description>
> > </property>
> >
> > Regards,
> > Susam Pal
> >
> > On Nov 21, 2007 8:54 AM, Josh Attenberg <jo...@gmail.com> wrote:
> > > I had this error when fetching with nutch 0.8.1 there is ~450GB left on
> > the
> > > disk where the crawl db and segments folder. Are there any other
> > settings i
> > > need to make? I know there isnt much space in my home directory, if it
> > was
> > > trying to write there, but there is at least 500M. what are the possible
> > > culprits/fixes?
> > >
> > >
> > > Exception in thread "main" org.apache.hadoop.fs.FSError:
> > java.io.IOException:
> > > No space left on device
> > > at
> > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > LocalFileSystem.java:150)
> > > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> > > FSDataOutputStream.java:112)
> > > at java.io.BufferedOutputStream.flushBuffer(
> > > BufferedOutputStream.java:65)
> > > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java
> > :123)
> > > at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> > > at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
> > > at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(
> > > FSDataOutputStream.java:96)
> > > at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> > > at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> > > at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> > > at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:154)
> > > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> > > at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
> > > LocalFileSystem.java:311)
> > > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java
> > :254)
> > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:327)
> > > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:443)
> > > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> > > Caused by: java.io.IOException: No space left on device
> > > at java.io.FileOutputStream.writeBytes(Native Method)
> > > at java.io.FileOutputStream.write(FileOutputStream.java:260)
> > > at
> > > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > > LocalFileSystem.java:148)
> > > ... 16 more
> > >
> >
>
Re: No space left on device
Posted by Josh Attenberg <jo...@gmail.com>.
i did as you say, and moved the files to a new directory on a big drive, but
now have some additional errors. are there any other pointers i need to
update?
On Nov 20, 2007 11:33 PM, Susam Pal <su...@gmail.com> wrote:
> Is /tmp present in a partition that doesn't have enough space? Does it
> have enough space left when this error occurs? Nutch often needs GBs
> of space for /tmp. If there isn't enough space on the partition having
> /tmp, then you can add the following property in
> 'conf/hadoop-site.xml' to make it use a different directory for
> writing the temporary files.
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/opt/tmp</value>
> <description>Base for Nutch Temporary Directories</description>
> </property>
>
> Regards,
> Susam Pal
>
> On Nov 21, 2007 8:54 AM, Josh Attenberg <jo...@gmail.com> wrote:
> > I had this error when fetching with nutch 0.8.1 there is ~450GB left on
> the
> > disk where the crawl db and segments folder. Are there any other
> settings i
> > need to make? I know there isnt much space in my home directory, if it
> was
> > trying to write there, but there is at least 500M. what are the possible
> > culprits/fixes?
> >
> >
> > Exception in thread "main" org.apache.hadoop.fs.FSError:
> java.io.IOException:
> > No space left on device
> > at
> > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > LocalFileSystem.java:150)
> > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> > FSDataOutputStream.java:112)
> > at java.io.BufferedOutputStream.flushBuffer(
> > BufferedOutputStream.java:65)
> > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java
> :123)
> > at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> > at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
> > at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(
> > FSDataOutputStream.java:96)
> > at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> > at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> > at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> > at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:154)
> > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> > at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
> > LocalFileSystem.java:311)
> > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java
> :254)
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:327)
> > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:443)
> > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> > Caused by: java.io.IOException: No space left on device
> > at java.io.FileOutputStream.writeBytes(Native Method)
> > at java.io.FileOutputStream.write(FileOutputStream.java:260)
> > at
> > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> > LocalFileSystem.java:148)
> > ... 16 more
> >
>
Re: No space left on device
Posted by Susam Pal <su...@gmail.com>.
Is /tmp present in a partition that doesn't have enough space? Does it
have enough space left when this error occurs? Nutch often needs GBs
of space for /tmp. If there isn't enough space on the partition having
/tmp, then you can add the following property in
'conf/hadoop-site.xml' to make it use a different directory for
writing the temporary files.
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/tmp</value>
<description>Base for Nutch Temporary Directories</description>
</property>
Regards,
Susam Pal
On Nov 21, 2007 8:54 AM, Josh Attenberg <jo...@gmail.com> wrote:
> I had this error when fetching with nutch 0.8.1 there is ~450GB left on the
> disk where the crawl db and segments folder. Are there any other settings i
> need to make? I know there isnt much space in my home directory, if it was
> trying to write there, but there is at least 500M. what are the possible
> culprits/fixes?
>
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException:
> No space left on device
> at
> org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> LocalFileSystem.java:150)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> FSDataOutputStream.java:112)
> at java.io.BufferedOutputStream.flushBuffer(
> BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
> at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(
> FSDataOutputStream.java:96)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
> at org.apache.hadoop.fs.FileUtil.copyContent(FileUtil.java:154)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:74)
> at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(
> LocalFileSystem.java:311)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:254)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:327)
> at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:443)
> at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at
> org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> LocalFileSystem.java:148)
> ... 16 more
>