You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by centerqi hu <ce...@gmail.com> on 2012/12/26 08:55:50 UTC
How to estimate hadoop.tmp.dir disk space
hi all
I encountered trouble
Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
device
hadoop dfsadmin -report
Configured Capacity: 44302785945600 (40.29 TB)
Present Capacity: 42020351946752 (38.22 TB)
DFS Remaining: 8124859072512 (7.39 TB)
DFS Used: 33895492874240 (30.83 TB)
DFS Used%: 80.66%
Under replicated blocks: 1687
Blocks with corrupt replicas: 0
Missing blocks: 0
However, my hdfs space is adequate
Change the size of the hadoop.tmp.dir error disappeared
How to estimate hadoop.tmp.dir disk space?
thx
--
centerqi@gmail.com
Re: How to estimate hadoop.tmp.dir disk space
Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?
check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>
--
Nitin Pawar
Re: How to estimate hadoop.tmp.dir disk space
Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx
2012/12/26 Harsh J <ha...@cloudera.com>
> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> > I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> > hadoop dfsadmin -report
> >
> > Configured Capacity: 44302785945600 (40.29 TB)
> > Present Capacity: 42020351946752 (38.22 TB)
> > DFS Remaining: 8124859072512 (7.39 TB)
> > DFS Used: 33895492874240 (30.83 TB)
> > DFS Used%: 80.66%
> > Under replicated blocks: 1687
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>
--
centerqi@gmail.com|齐忠
Re: How to estimate hadoop.tmp.dir disk space
Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx
2012/12/26 Harsh J <ha...@cloudera.com>
> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> > I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> > hadoop dfsadmin -report
> >
> > Configured Capacity: 44302785945600 (40.29 TB)
> > Present Capacity: 42020351946752 (38.22 TB)
> > DFS Remaining: 8124859072512 (7.39 TB)
> > DFS Used: 33895492874240 (30.83 TB)
> > DFS Used%: 80.66%
> > Under replicated blocks: 1687
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>
--
centerqi@gmail.com|齐忠
Re: How to estimate hadoop.tmp.dir disk space
Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx
2012/12/26 Harsh J <ha...@cloudera.com>
> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> > I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> > hadoop dfsadmin -report
> >
> > Configured Capacity: 44302785945600 (40.29 TB)
> > Present Capacity: 42020351946752 (38.22 TB)
> > DFS Remaining: 8124859072512 (7.39 TB)
> > DFS Used: 33895492874240 (30.83 TB)
> > DFS Used%: 80.66%
> > Under replicated blocks: 1687
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>
--
centerqi@gmail.com|齐忠
Re: How to estimate hadoop.tmp.dir disk space
Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx
2012/12/26 Harsh J <ha...@cloudera.com>
> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> > I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> > hadoop dfsadmin -report
> >
> > Configured Capacity: 44302785945600 (40.29 TB)
> > Present Capacity: 42020351946752 (38.22 TB)
> > DFS Remaining: 8124859072512 (7.39 TB)
> > DFS Used: 33895492874240 (30.83 TB)
> > DFS Used%: 80.66%
> > Under replicated blocks: 1687
> > Blocks with corrupt replicas: 0
> > Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>
--
centerqi@gmail.com|齐忠
Re: How to estimate hadoop.tmp.dir disk space
Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
--
Harsh J
Re: How to estimate hadoop.tmp.dir disk space
Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
--
Harsh J
Re: How to estimate hadoop.tmp.dir disk space
Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?
check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>
--
Nitin Pawar
Re: How to estimate hadoop.tmp.dir disk space
Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
--
Harsh J
Re: How to estimate hadoop.tmp.dir disk space
Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
--
Harsh J
Re: How to estimate hadoop.tmp.dir disk space
Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?
check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>
--
Nitin Pawar
Re: How to estimate hadoop.tmp.dir disk space
Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?
check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full
On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
> I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
> hadoop dfsadmin -report
>
> Configured Capacity: 44302785945600 (40.29 TB)
> Present Capacity: 42020351946752 (38.22 TB)
> DFS Remaining: 8124859072512 (7.39 TB)
> DFS Used: 33895492874240 (30.83 TB)
> DFS Used%: 80.66%
> Under replicated blocks: 1687
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>
--
Nitin Pawar