You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Jyothish Soman <jy...@gmail.com> on 2010/05/10 20:00:34 UTC
Is MapReduce NFS friendly
I have a distributed system on NFS, and wanted to use MapReduce on it, but
the system keeps spawning errors related to inability to allocate temporary
space.
Though sufficient is available, hence my question.
Is HDFS and NFS compatible?.
Re: Is MapReduce NFS friendly
Posted by Jyothish Soman <jy...@gmail.com>.
Forgot to thank you for this, it did work fine. Did the trick for me.
Thanks a lot
On Tue, May 11, 2010 at 12:09 AM, Marcin Sieniek <
msieniek@wirtualne-tatry.pl> wrote:
> Hi Jyothish,
>
> I had exactly the same problem and I solved it. To answer your question: as
> for me, HDFS and NFS are totally incompatible;) However, you may configure
> MadReduce to run on NFS only, without HDFS. See the last but one post here:
> http://old.nabble.com/Hadoop-over-Lustre--td19092864.html
> I did it and it works very well for NFS too (note that old hadoop-site.xml
> was splited to core-site.xml, mapred-site.xml and hdfs-site.xml in newer
> releases). Let me know if you have any problems with this configuration.
>
> Marcin
>
> Le 2010-05-10 20:16, alex kamil a écrit :
>
> Jyothish,
>
> as far as i know it is not recommended to run Hadoop on NFS, you suppose
> to use use local volumes for all mapred and dfs directories
>
> Alex
>
> On Mon, May 10, 2010 at 2:00 PM, Jyothish Soman <jy...@gmail.com>wrote:
>
>> I have a distributed system on NFS, and wanted to use MapReduce on it, but
>> the system keeps spawning errors related to inability to allocate temporary
>> space.
>> Though sufficient is available, hence my question.
>> Is HDFS and NFS compatible?.
>>
>
>
>
MapReduce and Heap Size
Posted by Mohamed Riadh Trad <Mo...@inria.fr>.
Hi,
I am running hadoop over 100 million files from a local file system.
Each split contains a subset of the file collection. The recordReader provides a file as a record.
The problem is that I get java.lang.outofmemoryerror: Java heap space.
I tried to increase the heap size to 16Gb, enougth to handle 20 millions files.
Up to 100 millions, I get the same problem and the job fails. Moreover, the job loading time exceeds 40 minutes while handling 20million files.
Any suggestions in order to avoid this delay and the Heap Space releated problems?
Regards,
Le 10 mai 2010 à 20:39, Marcin Sieniek a écrit :
> Hi Jyothish,
>
> I had exactly the same problem and I solved it. To answer your question: as for me, HDFS and NFS are totally incompatible;) However, you may configure MadReduce to run on NFS only, without HDFS. See the last but one post here:
> http://old.nabble.com/Hadoop-over-Lustre--td19092864.html
> I did it and it works very well for NFS too (note that old hadoop-site.xml was splited to core-site.xml, mapred-site.xml and hdfs-site.xml in newer releases). Let me know if you have any problems with this configuration.
>
> Marcin
>
> Le 2010-05-10 20:16, alex kamil a écrit :
>>
>> Jyothish,
>>
>> as far as i know it is not recommended to run Hadoop on NFS, you suppose to use use local volumes for all mapred and dfs directories
>>
>> Alex
>>
>> On Mon, May 10, 2010 at 2:00 PM, Jyothish Soman <jy...@gmail.com> wrote:
>> I have a distributed system on NFS, and wanted to use MapReduce on it, but the system keeps spawning errors related to inability to allocate temporary space.
>> Though sufficient is available, hence my question.
>> Is HDFS and NFS compatible?.
>>
>
Re: Is MapReduce NFS friendly
Posted by Marcin Sieniek <ms...@wirtualne-tatry.pl>.
Hi Jyothish,
I had exactly the same problem and I solved it. To answer your question:
as for me, HDFS and NFS are totally incompatible;) However, you may
configure MadReduce to run on NFS only, without HDFS. See the last but
one post here:
http://old.nabble.com/Hadoop-over-Lustre--td19092864.html
I did it and it works very well for NFS too (note that old
hadoop-site.xml was splited to core-site.xml, mapred-site.xml and
hdfs-site.xml in newer releases). Let me know if you have any problems
with this configuration.
Marcin
Le 2010-05-10 20:16, alex kamil a écrit :
> Jyothish,
>
> as far as i know it is not recommended to run Hadoop on NFS, you
> suppose to use use local volumes for all mapred and dfs directories
>
> Alex
>
> On Mon, May 10, 2010 at 2:00 PM, Jyothish Soman
> <jyothish.soman@gmail.com <ma...@gmail.com>> wrote:
>
> I have a distributed system on NFS, and wanted to use MapReduce on
> it, but the system keeps spawning errors related to inability to
> allocate temporary space.
> Though sufficient is available, hence my question.
> Is HDFS and NFS compatible?.
>
>
Re: Is MapReduce NFS friendly
Posted by alex kamil <al...@gmail.com>.
Jyothish,
as far as i know it is not recommended to run Hadoop on NFS, you suppose to
use use local volumes for all mapred and dfs directories
Alex
On Mon, May 10, 2010 at 2:00 PM, Jyothish Soman <jy...@gmail.com>wrote:
> I have a distributed system on NFS, and wanted to use MapReduce on it, but
> the system keeps spawning errors related to inability to allocate temporary
> space.
> Though sufficient is available, hence my question.
> Is HDFS and NFS compatible?.
>