You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by sagar arlekar <sa...@gmail.com> on 2008/12/31 15:16:18 UTC

cannot allocate memory error

Hello,

I am new to hadoop. I am running hapdoop 0.17 in a Eucalyptus cloud
instance (its a centos image on xen)

bin/hadoop dfs -ls /
gives the following Exception

08/12/31 08:58:10 WARN fs.FileSystem: "localhost:9000" is a deprecated
filesystem name. Use "hdfs://localhost:9000/" instead.
08/12/31 08:58:10 WARN fs.FileSystem: uri=hdfs://localhost:9000
javax.security.auth.login.LoginException: Login failed: Cannot run
program "whoami": java.io.IOException: error=12, Cannot allocate
memory
        at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
        at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
        at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
        at org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1353)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1289)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
        at org.apache.hadoop.fs.FsShell.init(FsShell.java:87)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:1717)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1866)
Bad connection to FS. command aborted.

Running the command again gives.
bin/hadoop dfs -ls /
Error occurred during initialization of VM
Could not reserve enough space for object heap

Changing value of 'mapred.child.java.opts' property in hadoop-site.xml
did not help.

Kindly help me. What could I do to give more memory to hadoop?

BTW is there a way to search through the mail archive? I only saw the
mails listed according to year and months.

Regards,
Sagar

Re: cannot allocate memory error

Posted by Brian Bockelman <bb...@cse.unl.edu>.
How much memory is on the system, how much memory is each Java process  
allocated, and how many Java processes are running?

If you have overcommit_mem turned on, Java 1.5 will lock *all* of its  
maximum heap size into RAM (ignores swap!) upon startup.  Earlier  
versions of 1.5 also allocate 1GB of RAM for code compilation.  I've  
seen situations where there was enough RAM to start up, but not enough  
to fork other processes.

Brian

On Dec 31, 2008, at 11:43 AM, Sagar Naik wrote:

> {HADOOP_HOME}/conf/hadoop-env.sh
> export HADOOP_HEAPSIZE
>
> the default is 1M, so I think that there could be another issue
>
> -Sagar
> sagar arlekar wrote:
>> Hello,
>>
>> I am new to hadoop. I am running hapdoop 0.17 in a Eucalyptus cloud
>> instance (its a centos image on xen)
>>
>> bin/hadoop dfs -ls /
>> gives the following Exception
>>
>> 08/12/31 08:58:10 WARN fs.FileSystem: "localhost:9000" is a  
>> deprecated
>> filesystem name. Use "hdfs://localhost:9000/" instead.
>> 08/12/31 08:58:10 WARN fs.FileSystem: uri=hdfs://localhost:9000
>> javax.security.auth.login.LoginException: Login failed: Cannot run
>> program "whoami": java.io.IOException: error=12, Cannot allocate
>> memory
>>        at  
>> org 
>> .apache 
>> .hadoop 
>> .security 
>> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
>>        at  
>> org 
>> .apache 
>> .hadoop 
>> .security 
>> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>>        at  
>> org 
>> .apache 
>> .hadoop 
>> .security 
>> .UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
>>        at  
>> org 
>> .apache 
>> .hadoop 
>> .security.UserGroupInformation.login(UserGroupInformation.java:67)
>>        at org.apache.hadoop.fs.FileSystem$Cache 
>> $Key.<init>(FileSystem.java:1353)
>>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
>> 1289)
>>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>>        at org.apache.hadoop.fs.FsShell.init(FsShell.java:87)
>>        at org.apache.hadoop.fs.FsShell.run(FsShell.java:1717)
>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>        at org.apache.hadoop.fs.FsShell.main(FsShell.java:1866)
>> Bad connection to FS. command aborted.
>>
>> Running the command again gives.
>> bin/hadoop dfs -ls /
>> Error occurred during initialization of VM
>> Could not reserve enough space for object heap
>>
>> Changing value of 'mapred.child.java.opts' property in hadoop- 
>> site.xml
>> did not help.
>>
>> Kindly help me. What could I do to give more memory to hadoop?
>>
>> BTW is there a way to search through the mail archive? I only saw the
>> mails listed according to year and months.
>>
>> Regards,
>> Sagar
>>

Re: cannot allocate memory error

Posted by Sagar Naik <sn...@attributor.com>.
{HADOOP_HOME}/conf/hadoop-env.sh
export HADOOP_HEAPSIZE

the default is 1M, so I think that there could be another issue

-Sagar
sagar arlekar wrote:
> Hello,
>
> I am new to hadoop. I am running hapdoop 0.17 in a Eucalyptus cloud
> instance (its a centos image on xen)
>
> bin/hadoop dfs -ls /
> gives the following Exception
>
> 08/12/31 08:58:10 WARN fs.FileSystem: "localhost:9000" is a deprecated
> filesystem name. Use "hdfs://localhost:9000/" instead.
> 08/12/31 08:58:10 WARN fs.FileSystem: uri=hdfs://localhost:9000
> javax.security.auth.login.LoginException: Login failed: Cannot run
> program "whoami": java.io.IOException: error=12, Cannot allocate
> memory
>         at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
>         at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>         at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
>         at org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1353)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1289)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>         at org.apache.hadoop.fs.FsShell.init(FsShell.java:87)
>         at org.apache.hadoop.fs.FsShell.run(FsShell.java:1717)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1866)
> Bad connection to FS. command aborted.
>
> Running the command again gives.
> bin/hadoop dfs -ls /
> Error occurred during initialization of VM
> Could not reserve enough space for object heap
>
> Changing value of 'mapred.child.java.opts' property in hadoop-site.xml
> did not help.
>
> Kindly help me. What could I do to give more memory to hadoop?
>
> BTW is there a way to search through the mail archive? I only saw the
> mails listed according to year and months.
>
> Regards,
> Sagar
>

Re: cannot allocate memory error

Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Your OS is running out of memory. Usually a sign of too many processes 
(or threads) on the machine. Check what else is happening on the system.

Raghu.

sagar arlekar wrote:
> Hello,
> 
> I am new to hadoop. I am running hapdoop 0.17 in a Eucalyptus cloud
> instance (its a centos image on xen)
> 
> bin/hadoop dfs -ls /
> gives the following Exception
> 
> 08/12/31 08:58:10 WARN fs.FileSystem: "localhost:9000" is a deprecated
> filesystem name. Use "hdfs://localhost:9000/" instead.
> 08/12/31 08:58:10 WARN fs.FileSystem: uri=hdfs://localhost:9000
> javax.security.auth.login.LoginException: Login failed: Cannot run
> program "whoami": java.io.IOException: error=12, Cannot allocate
> memory
>         at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
>         at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
>         at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
>         at org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67)
>         at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1353)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1289)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
>         at org.apache.hadoop.fs.FsShell.init(FsShell.java:87)
>         at org.apache.hadoop.fs.FsShell.run(FsShell.java:1717)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:1866)
> Bad connection to FS. command aborted.
> 
> Running the command again gives.
> bin/hadoop dfs -ls /
> Error occurred during initialization of VM
> Could not reserve enough space for object heap
> 
> Changing value of 'mapred.child.java.opts' property in hadoop-site.xml
> did not help.
> 
> Kindly help me. What could I do to give more memory to hadoop?
> 
> BTW is there a way to search through the mail archive? I only saw the
> mails listed according to year and months.
> 
> Regards,
> Sagar