You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by centerqi hu <ce...@gmail.com> on 2012/12/26 08:55:50 UTC

How to estimate hadoop.tmp.dir disk space

hi all
 I encountered trouble

Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
device


 hadoop dfsadmin -report

    Configured Capacity: 44302785945600 (40.29 TB)
    Present Capacity: 42020351946752 (38.22 TB)
    DFS Remaining: 8124859072512 (7.39 TB)
    DFS Used: 33895492874240 (30.83 TB)
    DFS Used%: 80.66%
    Under replicated blocks: 1687
    Blocks with corrupt replicas: 0
    Missing blocks: 0

However, my hdfs space is adequate

Change the size of the hadoop.tmp.dir error disappeared
How to estimate hadoop.tmp.dir disk space?

thx
-- 
centerqi@gmail.com

Re: How to estimate hadoop.tmp.dir disk space

Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?

check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full


On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:

> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>



-- 
Nitin Pawar

Re: How to estimate hadoop.tmp.dir disk space

Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx


2012/12/26 Harsh J <ha...@cloudera.com>

> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> >  I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> >  hadoop dfsadmin -report
> >
> >     Configured Capacity: 44302785945600 (40.29 TB)
> >     Present Capacity: 42020351946752 (38.22 TB)
> >     DFS Remaining: 8124859072512 (7.39 TB)
> >     DFS Used: 33895492874240 (30.83 TB)
> >     DFS Used%: 80.66%
> >     Under replicated blocks: 1687
> >     Blocks with corrupt replicas: 0
> >     Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>



-- 
centerqi@gmail.com|齐忠

Re: How to estimate hadoop.tmp.dir disk space

Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx


2012/12/26 Harsh J <ha...@cloudera.com>

> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> >  I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> >  hadoop dfsadmin -report
> >
> >     Configured Capacity: 44302785945600 (40.29 TB)
> >     Present Capacity: 42020351946752 (38.22 TB)
> >     DFS Remaining: 8124859072512 (7.39 TB)
> >     DFS Used: 33895492874240 (30.83 TB)
> >     DFS Used%: 80.66%
> >     Under replicated blocks: 1687
> >     Blocks with corrupt replicas: 0
> >     Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>



-- 
centerqi@gmail.com|齐忠

Re: How to estimate hadoop.tmp.dir disk space

Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx


2012/12/26 Harsh J <ha...@cloudera.com>

> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> >  I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> >  hadoop dfsadmin -report
> >
> >     Configured Capacity: 44302785945600 (40.29 TB)
> >     Present Capacity: 42020351946752 (38.22 TB)
> >     DFS Remaining: 8124859072512 (7.39 TB)
> >     DFS Used: 33895492874240 (30.83 TB)
> >     DFS Used%: 80.66%
> >     Under replicated blocks: 1687
> >     Blocks with corrupt replicas: 0
> >     Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>



-- 
centerqi@gmail.com|齐忠

Re: How to estimate hadoop.tmp.dir disk space

Posted by centerqi hu <ce...@gmail.com>.
How much disk space to meet the needs of hadoop.tmp.dir?
thx


2012/12/26 Harsh J <ha...@cloudera.com>

> The hadoop.tmp.dir is a local directory, usually defaulting to under
> /tmp/ and is thereby limited by that mount's space, not the HDFS
> space.
>
> On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> > hi all
> >  I encountered trouble
> >
> > Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >
> >
> >
> >  hadoop dfsadmin -report
> >
> >     Configured Capacity: 44302785945600 (40.29 TB)
> >     Present Capacity: 42020351946752 (38.22 TB)
> >     DFS Remaining: 8124859072512 (7.39 TB)
> >     DFS Used: 33895492874240 (30.83 TB)
> >     DFS Used%: 80.66%
> >     Under replicated blocks: 1687
> >     Blocks with corrupt replicas: 0
> >     Missing blocks: 0
> >
> > However, my hdfs space is adequate
> >
> > Change the size of the hadoop.tmp.dir error disappeared
> > How to estimate hadoop.tmp.dir disk space?
> >
> > thx
> > --
> > centerqi@gmail.com
>
>
>
> --
> Harsh J
>



-- 
centerqi@gmail.com|齐忠

Re: How to estimate hadoop.tmp.dir disk space

Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.

On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com



-- 
Harsh J

Re: How to estimate hadoop.tmp.dir disk space

Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.

On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com



-- 
Harsh J

Re: How to estimate hadoop.tmp.dir disk space

Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?

check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full


On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:

> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>



-- 
Nitin Pawar

Re: How to estimate hadoop.tmp.dir disk space

Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.

On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com



-- 
Harsh J

Re: How to estimate hadoop.tmp.dir disk space

Posted by Harsh J <ha...@cloudera.com>.
The hadoop.tmp.dir is a local directory, usually defaulting to under
/tmp/ and is thereby limited by that mount's space, not the HDFS
space.

On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:
> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com



-- 
Harsh J

Re: How to estimate hadoop.tmp.dir disk space

Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?

check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full


On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:

> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>



-- 
Nitin Pawar

Re: How to estimate hadoop.tmp.dir disk space

Posted by Nitin Pawar <ni...@gmail.com>.
Do you have mounted drives on the disk like JBOD setup where you have
allocated few drives to hdfs?

check df -h on all the nodes you may get the mount which holds the logs or
any other information which is outside dfs may be full


On Wed, Dec 26, 2012 at 1:25 PM, centerqi hu <ce...@gmail.com> wrote:

> hi all
>  I encountered trouble
>
> Message: org.apache.hadoop.ipc.RemoteException: java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>
>
>  hadoop dfsadmin -report
>
>     Configured Capacity: 44302785945600 (40.29 TB)
>     Present Capacity: 42020351946752 (38.22 TB)
>     DFS Remaining: 8124859072512 (7.39 TB)
>     DFS Used: 33895492874240 (30.83 TB)
>     DFS Used%: 80.66%
>     Under replicated blocks: 1687
>     Blocks with corrupt replicas: 0
>     Missing blocks: 0
>
> However, my hdfs space is adequate
>
> Change the size of the hadoop.tmp.dir error disappeared
> How to estimate hadoop.tmp.dir disk space?
>
> thx
> --
> centerqi@gmail.com
>



-- 
Nitin Pawar