You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Vikas Jadhav <vi...@gmail.com> on 2013/01/16 08:46:23 UTC
Hadoop NON DFS space
how to remove non dfs space from hadoop cluster
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Vikas,
You might want to check your logs. MR can generate huge logs depending
on what you are logging, and they are not on the DFS. The are on
non-dfs. If it's comming from there, you can change the loglevel to
reduce the size of the output.
On my own cluster, I turned the logs to debug and forgot to remove
that. At the end I ran into disks space issues because of that.
HTH
JM
2013/1/17, Vikas Jadhav <vi...@gmail.com>:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
> GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
> 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> > wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Vikas,
You might want to check your logs. MR can generate huge logs depending
on what you are logging, and they are not on the DFS. The are on
non-dfs. If it's comming from there, you can change the loglevel to
reduce the size of the output.
On my own cluster, I turned the logs to debug and forgot to remove
that. At the end I ran into disks space issues because of that.
HTH
JM
2013/1/17, Vikas Jadhav <vi...@gmail.com>:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
> GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
> 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> > wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Vikas,
You might want to check your logs. MR can generate huge logs depending
on what you are logging, and they are not on the DFS. The are on
non-dfs. If it's comming from there, you can change the loglevel to
reduce the size of the output.
On my own cluster, I turned the logs to debug and forgot to remove
that. At the end I ran into disks space issues because of that.
HTH
JM
2013/1/17, Vikas Jadhav <vi...@gmail.com>:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
> GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
> 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> > wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
FIlesystem counter total space around 20 gb
what is reason behind it ?
I am just writing 700 MB CSV FILE WITH 31 fileds in hbase
CounterMapReduceTotalJob CountersSLOTS_MILLIS_MAPS00592,940Launched reduce
tasks001Launched map tasks0012Data-local map tasks0012File Input Format
CountersBytes Read671,129,6090671,129,609FileSystemCountersFILE_BYTES_READ
6,908,267,48206,908,267,482HDFS_BYTES_READ671,130,7890671,130,789
FILE_BYTES_WRITTEN13,816,870,8846,908,299,38720,725,170,271Map-Reduce
FrameworkMap output materialized bytes6,908,265,47206,908,265,472Map input
records3,902,84903,902,849Reduce shuffle
bytes06,908,265,4726,908,265,472Spilled
Records7,805,69807,805,698Map output bytes6,892,654,01606,892,654,016CPU
time spent (ms)333,000168,450501,450Total committed heap usage (bytes)
2,095,972,352158,728,1922,254,700,544Combine input records000SPLIT_RAW_BYTES
1,18001,180Reduce input records000Reduce input groups000Combine output
records000Physical memory (bytes) snapshot2,354,528,256144,371,712
2,498,899,968Reduce output records000Virtual memory (bytes) snapshot
5,024,333,824514,969,6005,539,303,424Map output records3,902,84903,902,849
On Thu, Jan 17, 2013 at 7:41 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> its 700 mb csv file has 31 colms
> after loading into Hbase its size definately will not be more than 6GB
> (according to me)
>
>
> On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What is the amount of data you are attempting to crunch in one MR job?
>> Note that Map intermediate outputs are written to disk before being sent to
>> reducers and this counts for non-DFS usage. So to say grossly, if your
>> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
>> to do the whole process.
>>
>>
>> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Here is my problem
>>> I am using bulk loading for Hbase using MapReduce Program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>>
>>> But when i run my program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>>
>>> I have disable WAL in hbase still its consuming non-dfs
>>> and my program fails have tried lot times but no luck
>>>
>>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>>
>>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>>> large extent
>>>
>>>
>>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>>> FSError: java.io.IOException: No space left on device
>>> java.lang.Throwable: Child Error
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>>> Caused by: java.io.IOException: Creation of
>>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>>> failed.
>>> at
>>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>>> at
>>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>>
>>>
>>>
>>>
>>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> I think you can still run with the OS on another drive, or on a live
>>>> USB drive, or even on the memory only, loaded from the network while
>>>> the server is booting from the network drive, etc. No?
>>>>
>>>> JM
>>>>
>>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>>> > That would be really cool Chris.
>>>> > +1 for that.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>>> wrote:
>>>> >
>>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>>> >> works
>>>> >> as planned I'll post a nice summary of our approach. :)
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>>> >>
>>>> >>> <kidding> Wipe your OS out. </kidding>
>>>> >>>
>>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>>> >>> <vi...@gmail.com>wrote:
>>>> >>>
>>>> >>>>
>>>> >>>> how to remove non dfs space from hadoop cluster
>>>> >>>>
>>>> >>>> --
>>>> >>>> *
>>>> >>>> *
>>>> >>>> *
>>>> >>>>
>>>> >>>> Thanx and Regards*
>>>> >>>> * Vikas Jadhav*
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
FIlesystem counter total space around 20 gb
what is reason behind it ?
I am just writing 700 MB CSV FILE WITH 31 fileds in hbase
CounterMapReduceTotalJob CountersSLOTS_MILLIS_MAPS00592,940Launched reduce
tasks001Launched map tasks0012Data-local map tasks0012File Input Format
CountersBytes Read671,129,6090671,129,609FileSystemCountersFILE_BYTES_READ
6,908,267,48206,908,267,482HDFS_BYTES_READ671,130,7890671,130,789
FILE_BYTES_WRITTEN13,816,870,8846,908,299,38720,725,170,271Map-Reduce
FrameworkMap output materialized bytes6,908,265,47206,908,265,472Map input
records3,902,84903,902,849Reduce shuffle
bytes06,908,265,4726,908,265,472Spilled
Records7,805,69807,805,698Map output bytes6,892,654,01606,892,654,016CPU
time spent (ms)333,000168,450501,450Total committed heap usage (bytes)
2,095,972,352158,728,1922,254,700,544Combine input records000SPLIT_RAW_BYTES
1,18001,180Reduce input records000Reduce input groups000Combine output
records000Physical memory (bytes) snapshot2,354,528,256144,371,712
2,498,899,968Reduce output records000Virtual memory (bytes) snapshot
5,024,333,824514,969,6005,539,303,424Map output records3,902,84903,902,849
On Thu, Jan 17, 2013 at 7:41 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> its 700 mb csv file has 31 colms
> after loading into Hbase its size definately will not be more than 6GB
> (according to me)
>
>
> On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What is the amount of data you are attempting to crunch in one MR job?
>> Note that Map intermediate outputs are written to disk before being sent to
>> reducers and this counts for non-DFS usage. So to say grossly, if your
>> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
>> to do the whole process.
>>
>>
>> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Here is my problem
>>> I am using bulk loading for Hbase using MapReduce Program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>>
>>> But when i run my program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>>
>>> I have disable WAL in hbase still its consuming non-dfs
>>> and my program fails have tried lot times but no luck
>>>
>>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>>
>>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>>> large extent
>>>
>>>
>>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>>> FSError: java.io.IOException: No space left on device
>>> java.lang.Throwable: Child Error
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>>> Caused by: java.io.IOException: Creation of
>>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>>> failed.
>>> at
>>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>>> at
>>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>>
>>>
>>>
>>>
>>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> I think you can still run with the OS on another drive, or on a live
>>>> USB drive, or even on the memory only, loaded from the network while
>>>> the server is booting from the network drive, etc. No?
>>>>
>>>> JM
>>>>
>>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>>> > That would be really cool Chris.
>>>> > +1 for that.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>>> wrote:
>>>> >
>>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>>> >> works
>>>> >> as planned I'll post a nice summary of our approach. :)
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>>> >>
>>>> >>> <kidding> Wipe your OS out. </kidding>
>>>> >>>
>>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>>> >>> <vi...@gmail.com>wrote:
>>>> >>>
>>>> >>>>
>>>> >>>> how to remove non dfs space from hadoop cluster
>>>> >>>>
>>>> >>>> --
>>>> >>>> *
>>>> >>>> *
>>>> >>>> *
>>>> >>>>
>>>> >>>> Thanx and Regards*
>>>> >>>> * Vikas Jadhav*
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
FIlesystem counter total space around 20 gb
what is reason behind it ?
I am just writing 700 MB CSV FILE WITH 31 fileds in hbase
CounterMapReduceTotalJob CountersSLOTS_MILLIS_MAPS00592,940Launched reduce
tasks001Launched map tasks0012Data-local map tasks0012File Input Format
CountersBytes Read671,129,6090671,129,609FileSystemCountersFILE_BYTES_READ
6,908,267,48206,908,267,482HDFS_BYTES_READ671,130,7890671,130,789
FILE_BYTES_WRITTEN13,816,870,8846,908,299,38720,725,170,271Map-Reduce
FrameworkMap output materialized bytes6,908,265,47206,908,265,472Map input
records3,902,84903,902,849Reduce shuffle
bytes06,908,265,4726,908,265,472Spilled
Records7,805,69807,805,698Map output bytes6,892,654,01606,892,654,016CPU
time spent (ms)333,000168,450501,450Total committed heap usage (bytes)
2,095,972,352158,728,1922,254,700,544Combine input records000SPLIT_RAW_BYTES
1,18001,180Reduce input records000Reduce input groups000Combine output
records000Physical memory (bytes) snapshot2,354,528,256144,371,712
2,498,899,968Reduce output records000Virtual memory (bytes) snapshot
5,024,333,824514,969,6005,539,303,424Map output records3,902,84903,902,849
On Thu, Jan 17, 2013 at 7:41 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> its 700 mb csv file has 31 colms
> after loading into Hbase its size definately will not be more than 6GB
> (according to me)
>
>
> On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What is the amount of data you are attempting to crunch in one MR job?
>> Note that Map intermediate outputs are written to disk before being sent to
>> reducers and this counts for non-DFS usage. So to say grossly, if your
>> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
>> to do the whole process.
>>
>>
>> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Here is my problem
>>> I am using bulk loading for Hbase using MapReduce Program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>>
>>> But when i run my program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>>
>>> I have disable WAL in hbase still its consuming non-dfs
>>> and my program fails have tried lot times but no luck
>>>
>>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>>
>>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>>> large extent
>>>
>>>
>>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>>> FSError: java.io.IOException: No space left on device
>>> java.lang.Throwable: Child Error
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>>> Caused by: java.io.IOException: Creation of
>>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>>> failed.
>>> at
>>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>>> at
>>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>>
>>>
>>>
>>>
>>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> I think you can still run with the OS on another drive, or on a live
>>>> USB drive, or even on the memory only, loaded from the network while
>>>> the server is booting from the network drive, etc. No?
>>>>
>>>> JM
>>>>
>>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>>> > That would be really cool Chris.
>>>> > +1 for that.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>>> wrote:
>>>> >
>>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>>> >> works
>>>> >> as planned I'll post a nice summary of our approach. :)
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>>> >>
>>>> >>> <kidding> Wipe your OS out. </kidding>
>>>> >>>
>>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>>> >>> <vi...@gmail.com>wrote:
>>>> >>>
>>>> >>>>
>>>> >>>> how to remove non dfs space from hadoop cluster
>>>> >>>>
>>>> >>>> --
>>>> >>>> *
>>>> >>>> *
>>>> >>>> *
>>>> >>>>
>>>> >>>> Thanx and Regards*
>>>> >>>> * Vikas Jadhav*
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
FIlesystem counter total space around 20 gb
what is reason behind it ?
I am just writing 700 MB CSV FILE WITH 31 fileds in hbase
CounterMapReduceTotalJob CountersSLOTS_MILLIS_MAPS00592,940Launched reduce
tasks001Launched map tasks0012Data-local map tasks0012File Input Format
CountersBytes Read671,129,6090671,129,609FileSystemCountersFILE_BYTES_READ
6,908,267,48206,908,267,482HDFS_BYTES_READ671,130,7890671,130,789
FILE_BYTES_WRITTEN13,816,870,8846,908,299,38720,725,170,271Map-Reduce
FrameworkMap output materialized bytes6,908,265,47206,908,265,472Map input
records3,902,84903,902,849Reduce shuffle
bytes06,908,265,4726,908,265,472Spilled
Records7,805,69807,805,698Map output bytes6,892,654,01606,892,654,016CPU
time spent (ms)333,000168,450501,450Total committed heap usage (bytes)
2,095,972,352158,728,1922,254,700,544Combine input records000SPLIT_RAW_BYTES
1,18001,180Reduce input records000Reduce input groups000Combine output
records000Physical memory (bytes) snapshot2,354,528,256144,371,712
2,498,899,968Reduce output records000Virtual memory (bytes) snapshot
5,024,333,824514,969,6005,539,303,424Map output records3,902,84903,902,849
On Thu, Jan 17, 2013 at 7:41 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> its 700 mb csv file has 31 colms
> after loading into Hbase its size definately will not be more than 6GB
> (according to me)
>
>
> On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> What is the amount of data you are attempting to crunch in one MR job?
>> Note that Map intermediate outputs are written to disk before being sent to
>> reducers and this counts for non-DFS usage. So to say grossly, if your
>> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
>> to do the whole process.
>>
>>
>> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>> Here is my problem
>>> I am using bulk loading for Hbase using MapReduce Program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>>
>>> But when i run my program
>>>
>>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>>
>>> I have disable WAL in hbase still its consuming non-dfs
>>> and my program fails have tried lot times but no luck
>>>
>>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>>
>>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>>> large extent
>>>
>>>
>>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>>> FSError: java.io.IOException: No space left on device
>>> java.lang.Throwable: Child Error
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>>> Caused by: java.io.IOException: Creation of
>>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>>> failed.
>>> at
>>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>>> at
>>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>>> at
>>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>>
>>>
>>>
>>>
>>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>>> jean-marc@spaggiari.org> wrote:
>>>
>>>> I think you can still run with the OS on another drive, or on a live
>>>> USB drive, or even on the memory only, loaded from the network while
>>>> the server is booting from the network drive, etc. No?
>>>>
>>>> JM
>>>>
>>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>>> > That would be really cool Chris.
>>>> > +1 for that.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>>> wrote:
>>>> >
>>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>>> >> works
>>>> >> as planned I'll post a nice summary of our approach. :)
>>>> >>
>>>> >>
>>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>>> >>
>>>> >>> <kidding> Wipe your OS out. </kidding>
>>>> >>>
>>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>>> >>> <vi...@gmail.com>wrote:
>>>> >>>
>>>> >>>>
>>>> >>>> how to remove non dfs space from hadoop cluster
>>>> >>>>
>>>> >>>> --
>>>> >>>> *
>>>> >>>> *
>>>> >>>> *
>>>> >>>>
>>>> >>>> Thanx and Regards*
>>>> >>>> * Vikas Jadhav*
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
its 700 mb csv file has 31 colms
after loading into Hbase its size definately will not be more than 6GB
(according to me)
On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
> What is the amount of data you are attempting to crunch in one MR job?
> Note that Map intermediate outputs are written to disk before being sent to
> reducers and this counts for non-DFS usage. So to say grossly, if your
> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
> to do the whole process.
>
>
> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Here is my problem
>> I am using bulk loading for Hbase using MapReduce Program
>>
>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>
>> But when i run my program
>>
>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>
>> I have disable WAL in hbase still its consuming non-dfs
>> and my program fails have tried lot times but no luck
>>
>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>
>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>> large extent
>>
>>
>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>> FSError: java.io.IOException: No space left on device
>> java.lang.Throwable: Child Error
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>> Caused by: java.io.IOException: Creation of
>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>> failed.
>> at
>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>> at
>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>
>>
>>
>>
>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> I think you can still run with the OS on another drive, or on a live
>>> USB drive, or even on the memory only, loaded from the network while
>>> the server is booting from the network drive, etc. No?
>>>
>>> JM
>>>
>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>> > That would be really cool Chris.
>>> > +1 for that.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>> wrote:
>>> >
>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>> >> works
>>> >> as planned I'll post a nice summary of our approach. :)
>>> >>
>>> >>
>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>> >>
>>> >>> <kidding> Wipe your OS out. </kidding>
>>> >>>
>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> >>> <vi...@gmail.com>wrote:
>>> >>>
>>> >>>>
>>> >>>> how to remove non dfs space from hadoop cluster
>>> >>>>
>>> >>>> --
>>> >>>> *
>>> >>>> *
>>> >>>> *
>>> >>>>
>>> >>>> Thanx and Regards*
>>> >>>> * Vikas Jadhav*
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>>
>>> >>
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
its 700 mb csv file has 31 colms
after loading into Hbase its size definately will not be more than 6GB
(according to me)
On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
> What is the amount of data you are attempting to crunch in one MR job?
> Note that Map intermediate outputs are written to disk before being sent to
> reducers and this counts for non-DFS usage. So to say grossly, if your
> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
> to do the whole process.
>
>
> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Here is my problem
>> I am using bulk loading for Hbase using MapReduce Program
>>
>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>
>> But when i run my program
>>
>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>
>> I have disable WAL in hbase still its consuming non-dfs
>> and my program fails have tried lot times but no luck
>>
>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>
>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>> large extent
>>
>>
>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>> FSError: java.io.IOException: No space left on device
>> java.lang.Throwable: Child Error
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>> Caused by: java.io.IOException: Creation of
>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>> failed.
>> at
>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>> at
>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>
>>
>>
>>
>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> I think you can still run with the OS on another drive, or on a live
>>> USB drive, or even on the memory only, loaded from the network while
>>> the server is booting from the network drive, etc. No?
>>>
>>> JM
>>>
>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>> > That would be really cool Chris.
>>> > +1 for that.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>> wrote:
>>> >
>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>> >> works
>>> >> as planned I'll post a nice summary of our approach. :)
>>> >>
>>> >>
>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>> >>
>>> >>> <kidding> Wipe your OS out. </kidding>
>>> >>>
>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> >>> <vi...@gmail.com>wrote:
>>> >>>
>>> >>>>
>>> >>>> how to remove non dfs space from hadoop cluster
>>> >>>>
>>> >>>> --
>>> >>>> *
>>> >>>> *
>>> >>>> *
>>> >>>>
>>> >>>> Thanx and Regards*
>>> >>>> * Vikas Jadhav*
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>>
>>> >>
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
its 700 mb csv file has 31 colms
after loading into Hbase its size definately will not be more than 6GB
(according to me)
On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
> What is the amount of data you are attempting to crunch in one MR job?
> Note that Map intermediate outputs are written to disk before being sent to
> reducers and this counts for non-DFS usage. So to say grossly, if your
> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
> to do the whole process.
>
>
> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Here is my problem
>> I am using bulk loading for Hbase using MapReduce Program
>>
>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>
>> But when i run my program
>>
>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>
>> I have disable WAL in hbase still its consuming non-dfs
>> and my program fails have tried lot times but no luck
>>
>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>
>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>> large extent
>>
>>
>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>> FSError: java.io.IOException: No space left on device
>> java.lang.Throwable: Child Error
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>> Caused by: java.io.IOException: Creation of
>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>> failed.
>> at
>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>> at
>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>
>>
>>
>>
>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> I think you can still run with the OS on another drive, or on a live
>>> USB drive, or even on the memory only, loaded from the network while
>>> the server is booting from the network drive, etc. No?
>>>
>>> JM
>>>
>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>> > That would be really cool Chris.
>>> > +1 for that.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>> wrote:
>>> >
>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>> >> works
>>> >> as planned I'll post a nice summary of our approach. :)
>>> >>
>>> >>
>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>> >>
>>> >>> <kidding> Wipe your OS out. </kidding>
>>> >>>
>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> >>> <vi...@gmail.com>wrote:
>>> >>>
>>> >>>>
>>> >>>> how to remove non dfs space from hadoop cluster
>>> >>>>
>>> >>>> --
>>> >>>> *
>>> >>>> *
>>> >>>> *
>>> >>>>
>>> >>>> Thanx and Regards*
>>> >>>> * Vikas Jadhav*
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>>
>>> >>
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
its 700 mb csv file has 31 colms
after loading into Hbase its size definately will not be more than 6GB
(according to me)
On Thu, Jan 17, 2013 at 7:37 PM, Harsh J <ha...@cloudera.com> wrote:
> What is the amount of data you are attempting to crunch in one MR job?
> Note that Map intermediate outputs are written to disk before being sent to
> reducers and this counts for non-DFS usage. So to say grossly, if your
> input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
> to do the whole process.
>
>
> On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Here is my problem
>> I am using bulk loading for Hbase using MapReduce Program
>>
>> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
>> GB DFS Remaining : 13.06 GB DFS Used% : 4.93 % DFS Remaining% : 84.26 %
>>
>> But when i run my program
>>
>> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59
>> GB DFS Remaining : 116.01 MB DFS Used% : 5.16 % DFS Remaining% : 0.73 %
>>
>> I have disable WAL in hbase still its consuming non-dfs
>> and my program fails have tried lot times but no luck
>>
>> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>>
>> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
>> large extent
>>
>>
>> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
>> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
>> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
>> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
>> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
>> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
>> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
>> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
>> attempt_201301170837_0004_m_000009_0, Status : FAILED
>> FSError: java.io.IOException: No space left on device
>> java.lang.Throwable: Child Error
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
>> Caused by: java.io.IOException: Creation of
>> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
>> failed.
>> at
>> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
>> at
>> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
>> at
>> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
>> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
>> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
>> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
>> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
>> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
>> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>>
>>
>>
>>
>> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> I think you can still run with the OS on another drive, or on a live
>>> USB drive, or even on the memory only, loaded from the network while
>>> the server is booting from the network drive, etc. No?
>>>
>>> JM
>>>
>>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>>> > That would be really cool Chris.
>>> > +1 for that.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>>> wrote:
>>> >
>>> >> Ha, you joke, but we're planning on running with no local OS. If it
>>> >> works
>>> >> as planned I'll post a nice summary of our approach. :)
>>> >>
>>> >>
>>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>> >>
>>> >>> <kidding> Wipe your OS out. </kidding>
>>> >>>
>>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>> >>>
>>> >>>
>>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> >>> <vi...@gmail.com>wrote:
>>> >>>
>>> >>>>
>>> >>>> how to remove non dfs space from hadoop cluster
>>> >>>>
>>> >>>> --
>>> >>>> *
>>> >>>> *
>>> >>>> *
>>> >>>>
>>> >>>> Thanx and Regards*
>>> >>>> * Vikas Jadhav*
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>>
>>> >>
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
What is the amount of data you are attempting to crunch in one MR job? Note
that Map intermediate outputs are written to disk before being sent to
reducers and this counts for non-DFS usage. So to say grossly, if your
input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
to do the whole process.
On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68 GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
What is the amount of data you are attempting to crunch in one MR job? Note
that Map intermediate outputs are written to disk before being sent to
reducers and this counts for non-DFS usage. So to say grossly, if your
input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
to do the whole process.
On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68 GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
What is the amount of data you are attempting to crunch in one MR job? Note
that Map intermediate outputs are written to disk before being sent to
reducers and this counts for non-DFS usage. So to say grossly, if your
input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
to do the whole process.
On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68 GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
What is the amount of data you are attempting to crunch in one MR job? Note
that Map intermediate outputs are written to disk before being sent to
reducers and this counts for non-DFS usage. So to say grossly, if your
input is 14 GB, you surely need more than 2 or 3 x 14G free space overall
to do the whole process.
On Thu, Jan 17, 2013 at 7:20 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68 GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used : 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Vikas,
You might want to check your logs. MR can generate huge logs depending
on what you are logging, and they are not on the DFS. The are on
non-dfs. If it's comming from there, you can change the loglevel to
reduce the size of the output.
On my own cluster, I turned the logs to debug and forgot to remove
that. At the end I ran into disks space issues because of that.
HTH
JM
2013/1/17, Vikas Jadhav <vi...@gmail.com>:
> Here is my problem
> I am using bulk loading for Hbase using MapReduce Program
>
> Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
> GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
>
> But when i run my program
>
> Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
> 14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
>
> I have disable WAL in hbase still its consuming non-dfs
> and my program fails have tried lot times but no luck
>
> SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
>
> I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
> large extent
>
>
> 13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
> 13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
> 13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
> 13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
> 13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
> 13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
> 13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
> 13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
> attempt_201301170837_0004_m_000009_0, Status : FAILED
> FSError: java.io.IOException: No space left on device
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Creation of
> /tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
> failed.
> at
> org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
> at
> org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
> at
> org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
> 13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
> rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
> 13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
> 13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
> 13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
>
>
>
>
> On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> I think you can still run with the OS on another drive, or on a live
>> USB drive, or even on the memory only, loaded from the network while
>> the server is booting from the network drive, etc. No?
>>
>> JM
>>
>> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
>> > That would be really cool Chris.
>> > +1 for that.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com>
>> > wrote:
>> >
>> >> Ha, you joke, but we're planning on running with no local OS. If it
>> >> works
>> >> as planned I'll post a nice summary of our approach. :)
>> >>
>> >>
>> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >>> <kidding> Wipe your OS out. </kidding>
>> >>>
>> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>> >>>
>> >>>
>> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>> >>> <vi...@gmail.com>wrote:
>> >>>
>> >>>>
>> >>>> how to remove non dfs space from hadoop cluster
>> >>>>
>> >>>> --
>> >>>> *
>> >>>> *
>> >>>> *
>> >>>>
>> >>>> Thanx and Regards*
>> >>>> * Vikas Jadhav*
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>>
>> >>
>> >>
>> >
>>
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
Here is my problem
I am using bulk loading for Hbase using MapReduce Program
Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
But when i run my program
Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
I have disable WAL in hbase still its consuming non-dfs
and my program fails have tried lot times but no luck
SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
large extent
13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
attempt_201301170837_0004_m_000009_0, Status : FAILED
FSError: java.io.IOException: No space left on device
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of
/tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
failed.
at
org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
at
org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at
org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> I think you can still run with the OS on another drive, or on a live
> USB drive, or even on the memory only, loaded from the network while
> the server is booting from the network drive, etc. No?
>
> JM
>
> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
> > That would be really cool Chris.
> > +1 for that.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> >
> >> Ha, you joke, but we're planning on running with no local OS. If it
> >> works
> >> as planned I'll post a nice summary of our approach. :)
> >>
> >>
> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >>> <kidding> Wipe your OS out. </kidding>
> >>>
> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
> >>>
> >>>
> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
> >>> <vi...@gmail.com>wrote:
> >>>
> >>>>
> >>>> how to remove non dfs space from hadoop cluster
> >>>>
> >>>> --
> >>>> *
> >>>> *
> >>>> *
> >>>>
> >>>> Thanx and Regards*
> >>>> * Vikas Jadhav*
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>>
> >>
> >>
> >
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
Here is my problem
I am using bulk loading for Hbase using MapReduce Program
Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
But when i run my program
Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
I have disable WAL in hbase still its consuming non-dfs
and my program fails have tried lot times but no luck
SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
large extent
13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
attempt_201301170837_0004_m_000009_0, Status : FAILED
FSError: java.io.IOException: No space left on device
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of
/tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
failed.
at
org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
at
org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at
org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> I think you can still run with the OS on another drive, or on a live
> USB drive, or even on the memory only, loaded from the network while
> the server is booting from the network drive, etc. No?
>
> JM
>
> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
> > That would be really cool Chris.
> > +1 for that.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> >
> >> Ha, you joke, but we're planning on running with no local OS. If it
> >> works
> >> as planned I'll post a nice summary of our approach. :)
> >>
> >>
> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >>> <kidding> Wipe your OS out. </kidding>
> >>>
> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
> >>>
> >>>
> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
> >>> <vi...@gmail.com>wrote:
> >>>
> >>>>
> >>>> how to remove non dfs space from hadoop cluster
> >>>>
> >>>> --
> >>>> *
> >>>> *
> >>>> *
> >>>>
> >>>> Thanx and Regards*
> >>>> * Vikas Jadhav*
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>>
> >>
> >>
> >
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
Here is my problem
I am using bulk loading for Hbase using MapReduce Program
Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
But when i run my program
Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
I have disable WAL in hbase still its consuming non-dfs
and my program fails have tried lot times but no luck
SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
large extent
13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
attempt_201301170837_0004_m_000009_0, Status : FAILED
FSError: java.io.IOException: No space left on device
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of
/tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
failed.
at
org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
at
org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at
org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> I think you can still run with the OS on another drive, or on a live
> USB drive, or even on the memory only, loaded from the network while
> the server is booting from the network drive, etc. No?
>
> JM
>
> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
> > That would be really cool Chris.
> > +1 for that.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> >
> >> Ha, you joke, but we're planning on running with no local OS. If it
> >> works
> >> as planned I'll post a nice summary of our approach. :)
> >>
> >>
> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >>> <kidding> Wipe your OS out. </kidding>
> >>>
> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
> >>>
> >>>
> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
> >>> <vi...@gmail.com>wrote:
> >>>
> >>>>
> >>>> how to remove non dfs space from hadoop cluster
> >>>>
> >>>> --
> >>>> *
> >>>> *
> >>>> *
> >>>>
> >>>> Thanx and Regards*
> >>>> * Vikas Jadhav*
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>>
> >>
> >>
> >
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Vikas Jadhav <vi...@gmail.com>.
Here is my problem
I am using bulk loading for Hbase using MapReduce Program
Configured Capacity : 15.5 GB DFS Used : 781.91 MB Non DFS Used : 1.68
GBDFS Remaining:13.06 GBDFS Used%:4.93 %DFS Remaining%:84.26 %
But when i run my program
Configured Capacity : 15.5 GB DFS Used : 819.69 MB Non DFS Used :
14.59 GBDFS Remaining:116.01 MBDFS Used%:5.16 %DFS Remaining%:0.73 %
I have disable WAL in hbase still its consuming non-dfs
and my program fails have tried lot times but no luck
SO WHAT SHLOULD I DO SO THAT NON DFS WILL NOT CONSUME WHOLE SPACE
I AM ALSO NOT ABLE TO FIND REASON BEHIND usage of non-dfs space to this
large extent
13/01/17 08:44:07 INFO mapred.JobClient: map 83% reduce 22%
13/01/17 08:44:09 INFO mapred.JobClient: map 84% reduce 22%
13/01/17 08:44:12 INFO mapred.JobClient: map 85% reduce 22%
13/01/17 08:44:15 INFO mapred.JobClient: map 86% reduce 22%
13/01/17 08:44:18 INFO mapred.JobClient: map 87% reduce 22%
13/01/17 08:44:22 INFO mapred.JobClient: map 79% reduce 22%
13/01/17 08:44:25 INFO mapred.JobClient: map 80% reduce 25%
13/01/17 08:44:27 INFO mapred.JobClient: Task Id :
attempt_201301170837_0004_m_000009_0, Status : FAILED
FSError: java.io.IOException: No space left on device
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Creation of
/tmp/hadoop-cfgsas1/mapred/local/userlogs/job_201301170837_0004/attempt_201301170837_0004_m_000009_0.cleanup
failed.
at
org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:104)
at
org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
at
org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stdout
13/01/17 08:44:27 WARN mapred.JobClient: Error reading task outputhttp://
rdcesx12078.race.sas.com:50060/tasklog?plaintext=true&attemptid=attempt_201301170837_0004_m_000009_0&filter=stderr
13/01/17 08:44:28 INFO mapred.JobClient: map 82% reduce 25%
13/01/17 08:44:31 INFO mapred.JobClient: map 83% reduce 25%
13/01/17 08:45:07 INFO mapred.JobClient: map 83% reduce 27%
On Wed, Jan 16, 2013 at 6:43 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> I think you can still run with the OS on another drive, or on a live
> USB drive, or even on the memory only, loaded from the network while
> the server is booting from the network drive, etc. No?
>
> JM
>
> 2013/1/16, Mohammad Tariq <do...@gmail.com>:
> > That would be really cool Chris.
> > +1 for that.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> >
> >> Ha, you joke, but we're planning on running with no local OS. If it
> >> works
> >> as planned I'll post a nice summary of our approach. :)
> >>
> >>
> >> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >>> <kidding> Wipe your OS out. </kidding>
> >>>
> >>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
> >>>
> >>>
> >>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
> >>> <vi...@gmail.com>wrote:
> >>>
> >>>>
> >>>> how to remove non dfs space from hadoop cluster
> >>>>
> >>>> --
> >>>> *
> >>>> *
> >>>> *
> >>>>
> >>>> Thanx and Regards*
> >>>> * Vikas Jadhav*
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>>
> >>
> >>
> >
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I think you can still run with the OS on another drive, or on a live
USB drive, or even on the memory only, loaded from the network while
the server is booting from the network drive, etc. No?
JM
2013/1/16, Mohammad Tariq <do...@gmail.com>:
> That would be really cool Chris.
> +1 for that.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
>
>> Ha, you joke, but we're planning on running with no local OS. If it
>> works
>> as planned I'll post a nice summary of our approach. :)
>>
>>
>> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> <kidding> Wipe your OS out. </kidding>
>>>
>>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>
>>>
>>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> <vi...@gmail.com>wrote:
>>>
>>>>
>>>> how to remove non dfs space from hadoop cluster
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I think you can still run with the OS on another drive, or on a live
USB drive, or even on the memory only, loaded from the network while
the server is booting from the network drive, etc. No?
JM
2013/1/16, Mohammad Tariq <do...@gmail.com>:
> That would be really cool Chris.
> +1 for that.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
>
>> Ha, you joke, but we're planning on running with no local OS. If it
>> works
>> as planned I'll post a nice summary of our approach. :)
>>
>>
>> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> <kidding> Wipe your OS out. </kidding>
>>>
>>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>
>>>
>>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> <vi...@gmail.com>wrote:
>>>
>>>>
>>>> how to remove non dfs space from hadoop cluster
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I think you can still run with the OS on another drive, or on a live
USB drive, or even on the memory only, loaded from the network while
the server is booting from the network drive, etc. No?
JM
2013/1/16, Mohammad Tariq <do...@gmail.com>:
> That would be really cool Chris.
> +1 for that.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
>
>> Ha, you joke, but we're planning on running with no local OS. If it
>> works
>> as planned I'll post a nice summary of our approach. :)
>>
>>
>> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> <kidding> Wipe your OS out. </kidding>
>>>
>>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>
>>>
>>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> <vi...@gmail.com>wrote:
>>>
>>>>
>>>> how to remove non dfs space from hadoop cluster
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: Hadoop NON DFS space
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I think you can still run with the OS on another drive, or on a live
USB drive, or even on the memory only, loaded from the network while
the server is booting from the network drive, etc. No?
JM
2013/1/16, Mohammad Tariq <do...@gmail.com>:
> That would be really cool Chris.
> +1 for that.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
>
>> Ha, you joke, but we're planning on running with no local OS. If it
>> works
>> as planned I'll post a nice summary of our approach. :)
>>
>>
>> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> <kidding> Wipe your OS out. </kidding>
>>>
>>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>>
>>>
>>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav
>>> <vi...@gmail.com>wrote:
>>>
>>>>
>>>> how to remove non dfs space from hadoop cluster
>>>>
>>>> --
>>>> *
>>>> *
>>>> *
>>>>
>>>> Thanx and Regards*
>>>> * Vikas Jadhav*
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
Re: Hadoop NON DFS space
Posted by Mohammad Tariq <do...@gmail.com>.
That would be really cool Chris.
+1 for that.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> Ha, you joke, but we're planning on running with no local OS. If it works
> as planned I'll post a nice summary of our approach. :)
>
>
> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> <kidding> Wipe your OS out. </kidding>
>>
>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>
>>
>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> how to remove non dfs space from hadoop cluster
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop NON DFS space
Posted by Mohammad Tariq <do...@gmail.com>.
That would be really cool Chris.
+1 for that.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> Ha, you joke, but we're planning on running with no local OS. If it works
> as planned I'll post a nice summary of our approach. :)
>
>
> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> <kidding> Wipe your OS out. </kidding>
>>
>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>
>>
>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> how to remove non dfs space from hadoop cluster
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop NON DFS space
Posted by Mohammad Tariq <do...@gmail.com>.
That would be really cool Chris.
+1 for that.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> Ha, you joke, but we're planning on running with no local OS. If it works
> as planned I'll post a nice summary of our approach. :)
>
>
> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> <kidding> Wipe your OS out. </kidding>
>>
>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>
>>
>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> how to remove non dfs space from hadoop cluster
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop NON DFS space
Posted by Mohammad Tariq <do...@gmail.com>.
That would be really cool Chris.
+1 for that.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Jan 16, 2013 at 6:15 PM, Chris Embree <ce...@gmail.com> wrote:
> Ha, you joke, but we're planning on running with no local OS. If it works
> as planned I'll post a nice summary of our approach. :)
>
>
> On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> <kidding> Wipe your OS out. </kidding>
>>
>> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>>
>>
>> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>>
>>>
>>> how to remove non dfs space from hadoop cluster
>>>
>>> --
>>> *
>>> *
>>> *
>>>
>>> Thanx and Regards*
>>> * Vikas Jadhav*
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop NON DFS space
Posted by Chris Embree <ce...@gmail.com>.
Ha, you joke, but we're planning on running with no local OS. If it works
as planned I'll post a nice summary of our approach. :)
On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Jagat Singh <ja...@gmail.com>.
:)
Check whats consuming space other then core OS files.
Check temp spaces and other areas
On Wed, Jan 16, 2013 at 6:53 PM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Jagat Singh <ja...@gmail.com>.
:)
Check whats consuming space other then core OS files.
Check temp spaces and other areas
On Wed, Jan 16, 2013 at 6:53 PM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Chris Embree <ce...@gmail.com>.
Ha, you joke, but we're planning on running with no local OS. If it works
as planned I'll post a nice summary of our approach. :)
On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Jagat Singh <ja...@gmail.com>.
:)
Check whats consuming space other then core OS files.
Check temp spaces and other areas
On Wed, Jan 16, 2013 at 6:53 PM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Chris Embree <ce...@gmail.com>.
Ha, you joke, but we're planning on running with no local OS. If it works
as planned I'll post a nice summary of our approach. :)
On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Jagat Singh <ja...@gmail.com>.
:)
Check whats consuming space other then core OS files.
Check temp spaces and other areas
On Wed, Jan 16, 2013 at 6:53 PM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Chris Embree <ce...@gmail.com>.
Ha, you joke, but we're planning on running with no local OS. If it works
as planned I'll post a nice summary of our approach. :)
On Wed, Jan 16, 2013 at 2:53 AM, Harsh J <ha...@cloudera.com> wrote:
> <kidding> Wipe your OS out. </kidding>
>
> Please read: http://search-hadoop.com/m/9Qwi9UgMOe
>
>
> On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>>
>> how to remove non dfs space from hadoop cluster
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> Harsh J
>
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
<kidding> Wipe your OS out. </kidding>
Please read: http://search-hadoop.com/m/9Qwi9UgMOe
On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> how to remove non dfs space from hadoop cluster
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
<kidding> Wipe your OS out. </kidding>
Please read: http://search-hadoop.com/m/9Qwi9UgMOe
On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> how to remove non dfs space from hadoop cluster
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
<kidding> Wipe your OS out. </kidding>
Please read: http://search-hadoop.com/m/9Qwi9UgMOe
On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> how to remove non dfs space from hadoop cluster
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J
Re: Hadoop NON DFS space
Posted by Harsh J <ha...@cloudera.com>.
<kidding> Wipe your OS out. </kidding>
Please read: http://search-hadoop.com/m/9Qwi9UgMOe
On Wed, Jan 16, 2013 at 1:16 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
> how to remove non dfs space from hadoop cluster
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
Harsh J