You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Atish Kathpal <at...@gmail.com> on 2014/01/08 11:00:45 UTC
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Hi
By giving the complete URI, the MR jobs worked across both nodes. Thanks a
lot for the advice.
*Two issues though*:
1. On completion of the MR job, I see only the "_SUCCESS" file in the
output directory, but no part-r file containing the actual results of the
wordcount job. However I am seeing the correct output on running MR over
HDFS. What is going wrong? Any place I can find logs for the MR job. I see
no errors on the console.
Command used:
hadoop jar
/home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
2. I am observing that the mappers seem to be accessing files sequentially,
splitting the files across mappers, and then reading data in parallelel,
then moving on to the next file. What I want instead is that, files
themselves should be accessed in parallel, that is, if there are 10 files
to be MRed, then MR should ask for each of these files in parallel in one
go, and then work on the splits of these files in parallel.
*Why do I need this?* Some of the data coming from the NFS mount point is
coming from offline media (which takes ~5-10 seconds of time before first
bytes are received). So I would like all required files to be asked at the
onset itself from the NFS mount point. This way several offline media will
be spun up parallely and as the data from these media gets available MR can
process them.
Would be glad to get inputs on these points!
Thanks
Atish
Tip for those who are trying similar stuff::
In my case. after a while the jobs would fail, complaining of
"java.lang.OutOfMemoryError:
Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
but I was able to rectify this with help from:
http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
> test the suggestions made by Yong and Devin and get back to you guys.
>
> As on the bottlenecking issue, I agree, but I am trying to run few MR
> jobs on a traditional NAS server. I can live with a few bottlenecks, so
> long as I don't have to move the data to a dedicated HDFS cluster.
>
>
> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>
>> Yong raises an important issue: You have thrown out the I/O advantages
>> of HDFS and also thrown out the advantages of data locality. It would be
>> interesting to know why you are taking this approach.
>> Chris
>>
>>
>> On 12/20/2013 9:28 AM, java8964 wrote:
>>
>> I believe the "-fs local" should be removed too. The reason is that even
>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>> local", I believe that all the mappers will be run sequentially.
>>
>> "-fs local" will force the mapreducer run in "local" mode, which is
>> really a test mode.
>>
>> What you can do is to remove both "-fs local -jt local", but give the
>> FULL URI of the input and output path, to tell Hadoop that they are local
>> filesystem instead of HDFS.
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///hduser/mount_point file:///results"
>>
>> Keep in mind followings:
>>
>> 1) The NFS mount need to be available in all your Task Nodes, and
>> mounted in the same way.
>> 2) Even you can do that, but your sharing storage will be your
>> bottleneck. NFS won't work well for scalability.
>>
>> Yong
>>
>> ------------------------------
>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> From: dsuiter@rdx.com
>> To: user@hadoop.apache.org
>>
>> I think most of your problem is coming from the options you are setting:
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>
>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>> *-jt* argument and it should run in distributed mode if your cluster is
>> set up right. You don't need to do anything special to point Hadoop towards
>> a NFS location, other than set up the NFS location properly and make sure
>> if you are directing to it by name that it will resolve to the right
>> address. Hadoop doesn't care where it is, as long as it can read from and
>> write to it. The fact that you are telling it to read/write from/to a NFS
>> location that happens to be mounted as a local filesystem object doesn't
>> matter - you could direct it to the local /hduser/ path and set the -fs
>> local option, and it would end up on the NFS mount, because that's where
>> the NFS mount actually exists, or you could direct it to the absolute
>> network location of the folder that you want, it shouldn't make a
>> difference.
>>
>> *Devin Suiter*
>> Jr. Data Solutions Software Engineer
>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>> Google Voice: 412-256-8556 | www.rdx.com
>>
>>
>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hello
>>
>> The picture below describes the deployment architecture I am trying to
>> achieve.
>> However, when I run the wordcount example code with the below
>> configuration, by issuing the command from the master node, I notice only
>> the master node spawning map tasks and completing the submitted job. Below
>> is the command I used:
>>
>> *hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>
>> *Question: How can I leverage both the hadoop nodes for running MR,
>> while serving my data from the common NFS mount point running my filesystem
>> at the backend? Has any one tried such a setup before?*
>> [image: Inline image 1]
>>
>> Thanks!
>>
>>
>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Have Look on
BlockReaderLocal.java : Enables local short circuited reads.
The client gets the path to the file where block is stored using
getBlockLocalPathInfo()
Used to read file BlockReaderLocal()
http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/
On Fri, Jan 17, 2014 at 1:24 PM, Atish Kathpal <at...@gmail.com>wrote:
> Thanks Sudhakara. That explains the behavior I am observing.
>
> What I need: I would like files submitted as part of a MR job, to be asked
> from the underlying storage in a batched form immediately after submitting
> a job. This way the files that would be accessed as part of the MR job,
> will get cached in the underlying storage, leading to improved MR
> performance. I need to get this behavior out of MR as some of the files in
> the underlying storage (NFS mount point) I am using, are not available
> online and need to be fetched with some latency (~5-10secs or more). When
> all such offline files are asked from the underlying store by MR in one go,
> the underlying storage can make them available in parallel hence speeding
> up the entire process.
>
> Any clues on which classes I could start looking at in order to get this
> behavior from MR would be helpful.
>
> *TL;DR*
> How to make MR perform batch IO for files to be accessed as part of
> submitted job, before actual mappers have started processing those files?
>
> Thanks
> Atish
>
>
>
>
>
> On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Hello Atish,
>> Number of splits parallel processed at any instance is
>> dependance upon the your cluster capacity i,e. How many mapper and reduce
>> slots are configure in the cluster.
>> The default each tasktracker run maximum 2 map tasks that will be run
>> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
>> single task tacker it can simultaneously two mapper and process to input
>> splits. In your case your processing >1GB file, it as more than 8 input
>> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
>> mapper slots, maximum will process 4 input split any instance. Hence it
>> looking like sequential behavior.
>> For more info please have look on:
>>
>> mapred.tasktracker.map.tasks.maximum( for map slots)
>> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> Basically what I really need is that if I am submitting a job (say
>>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>>> issue immediate IO for reads on each of these files, such that the
>>> underlying storage system can know that it has to cache those files and
>>> present them back to Hadoop for processing. What I observe through by logs
>>> currently is that, if a particular file is very large, then Hadoop first
>>> operates on all splits of this large file, before asking for the next file.
>>>
>>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>>> where I should make any changes to. Any other suggestions for this will be
>>> welcome too.
>>>
>>>
>>>
>>>
>>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>>> I took logs at the underlying storage system and noticed the sequential
>>>> behavior across files. However, most of the files I am testing with are big
>>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>>> possible that this is just the case as I have just 2 nodes and the files
>>>> being processed are huge, Hence I am observing a sequential behaviour
>>>> across files.
>>>>
>>>> Within the files however, the splits seems to be getting processed in
>>>> parallel as I see random offsets within the file being accessed. That said,
>>>> let me try logging at the NFS level to map IOs coming from different IP
>>>> addresses (nodes) and analyze the pattern.
>>>>
>>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>>> take a look and get back accordingly.
>>>>
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>>
>>>>> When you said that the mappers seem to be accessing file sequentially,
>>>>> why do you think so?
>>>>>
>>>>> NFS maybe changes something, but mappers shouldn't access file
>>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>>> test to verify it.
>>>>>
>>>>> The class you want to check out is the
>>>>> org.apache.hadoop.mapred.FileInputFormat, especially method
>>>>> getSplits().
>>>>>
>>>>> The above code is the key how the split list is generated. If it
>>>>> doesn't performance well for your underline storage system, you can always
>>>>> write your own InputFormat to utilize your own storage system.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> From: atish.kathpal@gmail.com
>>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>>
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>>> Need guidance for 2. though!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>>
>>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>>> > wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>>> Thanks a lot for the advice.
>>>>>
>>>>> *Two issues though*:
>>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>>> output directory, but no part-r file containing the actual results of
>>>>> the wordcount job. However I am seeing the correct output on running MR
>>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>>> see no errors on the console.
>>>>> Command used:
>>>>> hadoop jar
>>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>>
>>>>>
>>>>> 2. I am observing that the mappers seem to be accessing files
>>>>> sequentially, splitting the files across mappers, and then reading data in
>>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>>> one go, and then work on the splits of these files in parallel.
>>>>> *Why do I need this?* Some of the data coming from the NFS mount
>>>>> point is coming from offline media (which takes ~5-10 seconds of time
>>>>> before first bytes are received). So I would like all required files to be
>>>>> asked at the onset itself from the NFS mount point. This way several
>>>>> offline media will be spun up parallely and as the data from these media
>>>>> gets available MR can process them.
>>>>>
>>>>> Would be glad to get inputs on these points!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>> Tip for those who are trying similar stuff::
>>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>>> but I was able to rectify this with help from:
>>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>>
>>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>>
>>>>>
>>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>>
>>>>> Yong raises an important issue: You have thrown out the I/O
>>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>>> would be interesting to know why you are taking this approach.
>>>>> Chris
>>>>>
>>>>>
>>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>>
>>>>> I believe the "-fs local" should be removed too. The reason is that
>>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>>
>>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>>> really a test mode.
>>>>>
>>>>> What you can do is to remove both "-fs local -jt local", but give
>>>>> the FULL URI of the input and output path, to tell Hadoop that they are
>>>>> local filesystem instead of HDFS.
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///hduser/mount_point file:///results"
>>>>>
>>>>> Keep in mind followings:
>>>>>
>>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>>> mounted in the same way.
>>>>> 2) Even you can do that, but your sharing storage will be your
>>>>> bottleneck. NFS won't work well for scalability.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> From: dsuiter@rdx.com
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>> I think most of your problem is coming from the options you are
>>>>> setting:
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>>
>>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>>> is set up right. You don't need to do anything special to point Hadoop
>>>>> towards a NFS location, other than set up the NFS location properly and
>>>>> make sure if you are directing to it by name that it will resolve to the
>>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>>> NFS location that happens to be mounted as a local filesystem object
>>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>>> network location of the folder that you want, it shouldn't make a
>>>>> difference.
>>>>>
>>>>> *Devin Suiter*
>>>>> Jr. Data Solutions Software Engineer
>>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>>
>>>>>
>>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Hello
>>>>>
>>>>> The picture below describes the deployment architecture I am trying
>>>>> to achieve.
>>>>> However, when I run the wordcount example code with the below
>>>>> configuration, by issuing the command from the master node, I notice only
>>>>> the master node spawning map tasks and completing the submitted job. Below
>>>>> is the command I used:
>>>>>
>>>>> *hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>>
>>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>>> while serving my data from the common NFS mount point running my filesystem
>>>>> at the backend? Has any one tried such a setup before?*
>>>>> [image: Inline image 1]
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> ...Sudhakara.st
>>
>>
>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Have Look on
BlockReaderLocal.java : Enables local short circuited reads.
The client gets the path to the file where block is stored using
getBlockLocalPathInfo()
Used to read file BlockReaderLocal()
http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/
On Fri, Jan 17, 2014 at 1:24 PM, Atish Kathpal <at...@gmail.com>wrote:
> Thanks Sudhakara. That explains the behavior I am observing.
>
> What I need: I would like files submitted as part of a MR job, to be asked
> from the underlying storage in a batched form immediately after submitting
> a job. This way the files that would be accessed as part of the MR job,
> will get cached in the underlying storage, leading to improved MR
> performance. I need to get this behavior out of MR as some of the files in
> the underlying storage (NFS mount point) I am using, are not available
> online and need to be fetched with some latency (~5-10secs or more). When
> all such offline files are asked from the underlying store by MR in one go,
> the underlying storage can make them available in parallel hence speeding
> up the entire process.
>
> Any clues on which classes I could start looking at in order to get this
> behavior from MR would be helpful.
>
> *TL;DR*
> How to make MR perform batch IO for files to be accessed as part of
> submitted job, before actual mappers have started processing those files?
>
> Thanks
> Atish
>
>
>
>
>
> On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Hello Atish,
>> Number of splits parallel processed at any instance is
>> dependance upon the your cluster capacity i,e. How many mapper and reduce
>> slots are configure in the cluster.
>> The default each tasktracker run maximum 2 map tasks that will be run
>> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
>> single task tacker it can simultaneously two mapper and process to input
>> splits. In your case your processing >1GB file, it as more than 8 input
>> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
>> mapper slots, maximum will process 4 input split any instance. Hence it
>> looking like sequential behavior.
>> For more info please have look on:
>>
>> mapred.tasktracker.map.tasks.maximum( for map slots)
>> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> Basically what I really need is that if I am submitting a job (say
>>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>>> issue immediate IO for reads on each of these files, such that the
>>> underlying storage system can know that it has to cache those files and
>>> present them back to Hadoop for processing. What I observe through by logs
>>> currently is that, if a particular file is very large, then Hadoop first
>>> operates on all splits of this large file, before asking for the next file.
>>>
>>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>>> where I should make any changes to. Any other suggestions for this will be
>>> welcome too.
>>>
>>>
>>>
>>>
>>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>>> I took logs at the underlying storage system and noticed the sequential
>>>> behavior across files. However, most of the files I am testing with are big
>>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>>> possible that this is just the case as I have just 2 nodes and the files
>>>> being processed are huge, Hence I am observing a sequential behaviour
>>>> across files.
>>>>
>>>> Within the files however, the splits seems to be getting processed in
>>>> parallel as I see random offsets within the file being accessed. That said,
>>>> let me try logging at the NFS level to map IOs coming from different IP
>>>> addresses (nodes) and analyze the pattern.
>>>>
>>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>>> take a look and get back accordingly.
>>>>
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>>
>>>>> When you said that the mappers seem to be accessing file sequentially,
>>>>> why do you think so?
>>>>>
>>>>> NFS maybe changes something, but mappers shouldn't access file
>>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>>> test to verify it.
>>>>>
>>>>> The class you want to check out is the
>>>>> org.apache.hadoop.mapred.FileInputFormat, especially method
>>>>> getSplits().
>>>>>
>>>>> The above code is the key how the split list is generated. If it
>>>>> doesn't performance well for your underline storage system, you can always
>>>>> write your own InputFormat to utilize your own storage system.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> From: atish.kathpal@gmail.com
>>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>>
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>>> Need guidance for 2. though!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>>
>>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>>> > wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>>> Thanks a lot for the advice.
>>>>>
>>>>> *Two issues though*:
>>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>>> output directory, but no part-r file containing the actual results of
>>>>> the wordcount job. However I am seeing the correct output on running MR
>>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>>> see no errors on the console.
>>>>> Command used:
>>>>> hadoop jar
>>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>>
>>>>>
>>>>> 2. I am observing that the mappers seem to be accessing files
>>>>> sequentially, splitting the files across mappers, and then reading data in
>>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>>> one go, and then work on the splits of these files in parallel.
>>>>> *Why do I need this?* Some of the data coming from the NFS mount
>>>>> point is coming from offline media (which takes ~5-10 seconds of time
>>>>> before first bytes are received). So I would like all required files to be
>>>>> asked at the onset itself from the NFS mount point. This way several
>>>>> offline media will be spun up parallely and as the data from these media
>>>>> gets available MR can process them.
>>>>>
>>>>> Would be glad to get inputs on these points!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>> Tip for those who are trying similar stuff::
>>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>>> but I was able to rectify this with help from:
>>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>>
>>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>>
>>>>>
>>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>>
>>>>> Yong raises an important issue: You have thrown out the I/O
>>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>>> would be interesting to know why you are taking this approach.
>>>>> Chris
>>>>>
>>>>>
>>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>>
>>>>> I believe the "-fs local" should be removed too. The reason is that
>>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>>
>>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>>> really a test mode.
>>>>>
>>>>> What you can do is to remove both "-fs local -jt local", but give
>>>>> the FULL URI of the input and output path, to tell Hadoop that they are
>>>>> local filesystem instead of HDFS.
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///hduser/mount_point file:///results"
>>>>>
>>>>> Keep in mind followings:
>>>>>
>>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>>> mounted in the same way.
>>>>> 2) Even you can do that, but your sharing storage will be your
>>>>> bottleneck. NFS won't work well for scalability.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> From: dsuiter@rdx.com
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>> I think most of your problem is coming from the options you are
>>>>> setting:
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>>
>>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>>> is set up right. You don't need to do anything special to point Hadoop
>>>>> towards a NFS location, other than set up the NFS location properly and
>>>>> make sure if you are directing to it by name that it will resolve to the
>>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>>> NFS location that happens to be mounted as a local filesystem object
>>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>>> network location of the folder that you want, it shouldn't make a
>>>>> difference.
>>>>>
>>>>> *Devin Suiter*
>>>>> Jr. Data Solutions Software Engineer
>>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>>
>>>>>
>>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Hello
>>>>>
>>>>> The picture below describes the deployment architecture I am trying
>>>>> to achieve.
>>>>> However, when I run the wordcount example code with the below
>>>>> configuration, by issuing the command from the master node, I notice only
>>>>> the master node spawning map tasks and completing the submitted job. Below
>>>>> is the command I used:
>>>>>
>>>>> *hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>>
>>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>>> while serving my data from the common NFS mount point running my filesystem
>>>>> at the backend? Has any one tried such a setup before?*
>>>>> [image: Inline image 1]
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> ...Sudhakara.st
>>
>>
>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Have Look on
BlockReaderLocal.java : Enables local short circuited reads.
The client gets the path to the file where block is stored using
getBlockLocalPathInfo()
Used to read file BlockReaderLocal()
http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/
On Fri, Jan 17, 2014 at 1:24 PM, Atish Kathpal <at...@gmail.com>wrote:
> Thanks Sudhakara. That explains the behavior I am observing.
>
> What I need: I would like files submitted as part of a MR job, to be asked
> from the underlying storage in a batched form immediately after submitting
> a job. This way the files that would be accessed as part of the MR job,
> will get cached in the underlying storage, leading to improved MR
> performance. I need to get this behavior out of MR as some of the files in
> the underlying storage (NFS mount point) I am using, are not available
> online and need to be fetched with some latency (~5-10secs or more). When
> all such offline files are asked from the underlying store by MR in one go,
> the underlying storage can make them available in parallel hence speeding
> up the entire process.
>
> Any clues on which classes I could start looking at in order to get this
> behavior from MR would be helpful.
>
> *TL;DR*
> How to make MR perform batch IO for files to be accessed as part of
> submitted job, before actual mappers have started processing those files?
>
> Thanks
> Atish
>
>
>
>
>
> On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Hello Atish,
>> Number of splits parallel processed at any instance is
>> dependance upon the your cluster capacity i,e. How many mapper and reduce
>> slots are configure in the cluster.
>> The default each tasktracker run maximum 2 map tasks that will be run
>> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
>> single task tacker it can simultaneously two mapper and process to input
>> splits. In your case your processing >1GB file, it as more than 8 input
>> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
>> mapper slots, maximum will process 4 input split any instance. Hence it
>> looking like sequential behavior.
>> For more info please have look on:
>>
>> mapred.tasktracker.map.tasks.maximum( for map slots)
>> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> Basically what I really need is that if I am submitting a job (say
>>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>>> issue immediate IO for reads on each of these files, such that the
>>> underlying storage system can know that it has to cache those files and
>>> present them back to Hadoop for processing. What I observe through by logs
>>> currently is that, if a particular file is very large, then Hadoop first
>>> operates on all splits of this large file, before asking for the next file.
>>>
>>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>>> where I should make any changes to. Any other suggestions for this will be
>>> welcome too.
>>>
>>>
>>>
>>>
>>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>>> I took logs at the underlying storage system and noticed the sequential
>>>> behavior across files. However, most of the files I am testing with are big
>>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>>> possible that this is just the case as I have just 2 nodes and the files
>>>> being processed are huge, Hence I am observing a sequential behaviour
>>>> across files.
>>>>
>>>> Within the files however, the splits seems to be getting processed in
>>>> parallel as I see random offsets within the file being accessed. That said,
>>>> let me try logging at the NFS level to map IOs coming from different IP
>>>> addresses (nodes) and analyze the pattern.
>>>>
>>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>>> take a look and get back accordingly.
>>>>
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>>
>>>>> When you said that the mappers seem to be accessing file sequentially,
>>>>> why do you think so?
>>>>>
>>>>> NFS maybe changes something, but mappers shouldn't access file
>>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>>> test to verify it.
>>>>>
>>>>> The class you want to check out is the
>>>>> org.apache.hadoop.mapred.FileInputFormat, especially method
>>>>> getSplits().
>>>>>
>>>>> The above code is the key how the split list is generated. If it
>>>>> doesn't performance well for your underline storage system, you can always
>>>>> write your own InputFormat to utilize your own storage system.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> From: atish.kathpal@gmail.com
>>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>>
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>>> Need guidance for 2. though!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>>
>>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>>> > wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>>> Thanks a lot for the advice.
>>>>>
>>>>> *Two issues though*:
>>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>>> output directory, but no part-r file containing the actual results of
>>>>> the wordcount job. However I am seeing the correct output on running MR
>>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>>> see no errors on the console.
>>>>> Command used:
>>>>> hadoop jar
>>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>>
>>>>>
>>>>> 2. I am observing that the mappers seem to be accessing files
>>>>> sequentially, splitting the files across mappers, and then reading data in
>>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>>> one go, and then work on the splits of these files in parallel.
>>>>> *Why do I need this?* Some of the data coming from the NFS mount
>>>>> point is coming from offline media (which takes ~5-10 seconds of time
>>>>> before first bytes are received). So I would like all required files to be
>>>>> asked at the onset itself from the NFS mount point. This way several
>>>>> offline media will be spun up parallely and as the data from these media
>>>>> gets available MR can process them.
>>>>>
>>>>> Would be glad to get inputs on these points!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>> Tip for those who are trying similar stuff::
>>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>>> but I was able to rectify this with help from:
>>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>>
>>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>>
>>>>>
>>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>>
>>>>> Yong raises an important issue: You have thrown out the I/O
>>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>>> would be interesting to know why you are taking this approach.
>>>>> Chris
>>>>>
>>>>>
>>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>>
>>>>> I believe the "-fs local" should be removed too. The reason is that
>>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>>
>>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>>> really a test mode.
>>>>>
>>>>> What you can do is to remove both "-fs local -jt local", but give
>>>>> the FULL URI of the input and output path, to tell Hadoop that they are
>>>>> local filesystem instead of HDFS.
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///hduser/mount_point file:///results"
>>>>>
>>>>> Keep in mind followings:
>>>>>
>>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>>> mounted in the same way.
>>>>> 2) Even you can do that, but your sharing storage will be your
>>>>> bottleneck. NFS won't work well for scalability.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> From: dsuiter@rdx.com
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>> I think most of your problem is coming from the options you are
>>>>> setting:
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>>
>>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>>> is set up right. You don't need to do anything special to point Hadoop
>>>>> towards a NFS location, other than set up the NFS location properly and
>>>>> make sure if you are directing to it by name that it will resolve to the
>>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>>> NFS location that happens to be mounted as a local filesystem object
>>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>>> network location of the folder that you want, it shouldn't make a
>>>>> difference.
>>>>>
>>>>> *Devin Suiter*
>>>>> Jr. Data Solutions Software Engineer
>>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>>
>>>>>
>>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Hello
>>>>>
>>>>> The picture below describes the deployment architecture I am trying
>>>>> to achieve.
>>>>> However, when I run the wordcount example code with the below
>>>>> configuration, by issuing the command from the master node, I notice only
>>>>> the master node spawning map tasks and completing the submitted job. Below
>>>>> is the command I used:
>>>>>
>>>>> *hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>>
>>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>>> while serving my data from the common NFS mount point running my filesystem
>>>>> at the backend? Has any one tried such a setup before?*
>>>>> [image: Inline image 1]
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> ...Sudhakara.st
>>
>>
>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Have Look on
BlockReaderLocal.java : Enables local short circuited reads.
The client gets the path to the file where block is stored using
getBlockLocalPathInfo()
Used to read file BlockReaderLocal()
http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/
On Fri, Jan 17, 2014 at 1:24 PM, Atish Kathpal <at...@gmail.com>wrote:
> Thanks Sudhakara. That explains the behavior I am observing.
>
> What I need: I would like files submitted as part of a MR job, to be asked
> from the underlying storage in a batched form immediately after submitting
> a job. This way the files that would be accessed as part of the MR job,
> will get cached in the underlying storage, leading to improved MR
> performance. I need to get this behavior out of MR as some of the files in
> the underlying storage (NFS mount point) I am using, are not available
> online and need to be fetched with some latency (~5-10secs or more). When
> all such offline files are asked from the underlying store by MR in one go,
> the underlying storage can make them available in parallel hence speeding
> up the entire process.
>
> Any clues on which classes I could start looking at in order to get this
> behavior from MR would be helpful.
>
> *TL;DR*
> How to make MR perform batch IO for files to be accessed as part of
> submitted job, before actual mappers have started processing those files?
>
> Thanks
> Atish
>
>
>
>
>
> On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
>
>> Hello Atish,
>> Number of splits parallel processed at any instance is
>> dependance upon the your cluster capacity i,e. How many mapper and reduce
>> slots are configure in the cluster.
>> The default each tasktracker run maximum 2 map tasks that will be run
>> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
>> single task tacker it can simultaneously two mapper and process to input
>> splits. In your case your processing >1GB file, it as more than 8 input
>> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
>> mapper slots, maximum will process 4 input split any instance. Hence it
>> looking like sequential behavior.
>> For more info please have look on:
>>
>> mapred.tasktracker.map.tasks.maximum( for map slots)
>> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> Basically what I really need is that if I am submitting a job (say
>>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>>> issue immediate IO for reads on each of these files, such that the
>>> underlying storage system can know that it has to cache those files and
>>> present them back to Hadoop for processing. What I observe through by logs
>>> currently is that, if a particular file is very large, then Hadoop first
>>> operates on all splits of this large file, before asking for the next file.
>>>
>>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>>> where I should make any changes to. Any other suggestions for this will be
>>> welcome too.
>>>
>>>
>>>
>>>
>>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>>> I took logs at the underlying storage system and noticed the sequential
>>>> behavior across files. However, most of the files I am testing with are big
>>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>>> possible that this is just the case as I have just 2 nodes and the files
>>>> being processed are huge, Hence I am observing a sequential behaviour
>>>> across files.
>>>>
>>>> Within the files however, the splits seems to be getting processed in
>>>> parallel as I see random offsets within the file being accessed. That said,
>>>> let me try logging at the NFS level to map IOs coming from different IP
>>>> addresses (nodes) and analyze the pattern.
>>>>
>>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>>> take a look and get back accordingly.
>>>>
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>>
>>>>> When you said that the mappers seem to be accessing file sequentially,
>>>>> why do you think so?
>>>>>
>>>>> NFS maybe changes something, but mappers shouldn't access file
>>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>>> test to verify it.
>>>>>
>>>>> The class you want to check out is the
>>>>> org.apache.hadoop.mapred.FileInputFormat, especially method
>>>>> getSplits().
>>>>>
>>>>> The above code is the key how the split list is generated. If it
>>>>> doesn't performance well for your underline storage system, you can always
>>>>> write your own InputFormat to utilize your own storage system.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> From: atish.kathpal@gmail.com
>>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>>
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>>
>>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>>> Need guidance for 2. though!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>>
>>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>>> > wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>>> Thanks a lot for the advice.
>>>>>
>>>>> *Two issues though*:
>>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>>> output directory, but no part-r file containing the actual results of
>>>>> the wordcount job. However I am seeing the correct output on running MR
>>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>>> see no errors on the console.
>>>>> Command used:
>>>>> hadoop jar
>>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>>
>>>>>
>>>>> 2. I am observing that the mappers seem to be accessing files
>>>>> sequentially, splitting the files across mappers, and then reading data in
>>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>>> one go, and then work on the splits of these files in parallel.
>>>>> *Why do I need this?* Some of the data coming from the NFS mount
>>>>> point is coming from offline media (which takes ~5-10 seconds of time
>>>>> before first bytes are received). So I would like all required files to be
>>>>> asked at the onset itself from the NFS mount point. This way several
>>>>> offline media will be spun up parallely and as the data from these media
>>>>> gets available MR can process them.
>>>>>
>>>>> Would be glad to get inputs on these points!
>>>>>
>>>>> Thanks
>>>>> Atish
>>>>>
>>>>> Tip for those who are trying similar stuff::
>>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>>> but I was able to rectify this with help from:
>>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>>
>>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>>
>>>>>
>>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>>
>>>>> Yong raises an important issue: You have thrown out the I/O
>>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>>> would be interesting to know why you are taking this approach.
>>>>> Chris
>>>>>
>>>>>
>>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>>
>>>>> I believe the "-fs local" should be removed too. The reason is that
>>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>>
>>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>>> really a test mode.
>>>>>
>>>>> What you can do is to remove both "-fs local -jt local", but give
>>>>> the FULL URI of the input and output path, to tell Hadoop that they are
>>>>> local filesystem instead of HDFS.
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount file:///hduser/mount_point file:///results"
>>>>>
>>>>> Keep in mind followings:
>>>>>
>>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>>> mounted in the same way.
>>>>> 2) Even you can do that, but your sharing storage will be your
>>>>> bottleneck. NFS won't work well for scalability.
>>>>>
>>>>> Yong
>>>>>
>>>>> ------------------------------
>>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>>> filesystem
>>>>> From: dsuiter@rdx.com
>>>>> To: user@hadoop.apache.org
>>>>>
>>>>> I think most of your problem is coming from the options you are
>>>>> setting:
>>>>>
>>>>> "hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>>
>>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>>> is set up right. You don't need to do anything special to point Hadoop
>>>>> towards a NFS location, other than set up the NFS location properly and
>>>>> make sure if you are directing to it by name that it will resolve to the
>>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>>> NFS location that happens to be mounted as a local filesystem object
>>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>>> network location of the folder that you want, it shouldn't make a
>>>>> difference.
>>>>>
>>>>> *Devin Suiter*
>>>>> Jr. Data Solutions Software Engineer
>>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>>
>>>>>
>>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <
>>>>> atish.kathpal@gmail.com> wrote:
>>>>>
>>>>> Hello
>>>>>
>>>>> The picture below describes the deployment architecture I am trying
>>>>> to achieve.
>>>>> However, when I run the wordcount example code with the below
>>>>> configuration, by issuing the command from the master node, I notice only
>>>>> the master node spawning map tasks and completing the submitted job. Below
>>>>> is the command I used:
>>>>>
>>>>> *hadoop jar
>>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>>
>>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>>> while serving my data from the common NFS mount point running my filesystem
>>>>> at the backend? Has any one tried such a setup before?*
>>>>> [image: Inline image 1]
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> ...Sudhakara.st
>>
>>
>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Thanks Sudhakara. That explains the behavior I am observing.
What I need: I would like files submitted as part of a MR job, to be asked
from the underlying storage in a batched form immediately after submitting
a job. This way the files that would be accessed as part of the MR job,
will get cached in the underlying storage, leading to improved MR
performance. I need to get this behavior out of MR as some of the files in
the underlying storage (NFS mount point) I am using, are not available
online and need to be fetched with some latency (~5-10secs or more). When
all such offline files are asked from the underlying store by MR in one go,
the underlying storage can make them available in parallel hence speeding
up the entire process.
Any clues on which classes I could start looking at in order to get this
behavior from MR would be helpful.
*TL;DR*
How to make MR perform batch IO for files to be accessed as part of
submitted job, before actual mappers have started processing those files?
Thanks
Atish
On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
> Hello Atish,
> Number of splits parallel processed at any instance is
> dependance upon the your cluster capacity i,e. How many mapper and reduce
> slots are configure in the cluster.
> The default each tasktracker run maximum 2 map tasks that will be run
> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
> single task tacker it can simultaneously two mapper and process to input
> splits. In your case your processing >1GB file, it as more than 8 input
> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
> mapper slots, maximum will process 4 input split any instance. Hence it
> looking like sequential behavior.
> For more info please have look on:
>
> mapred.tasktracker.map.tasks.maximum( for map slots)
> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>
>
>
>
> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Basically what I really need is that if I am submitting a job (say
>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>> issue immediate IO for reads on each of these files, such that the
>> underlying storage system can know that it has to cache those files and
>> present them back to Hadoop for processing. What I observe through by logs
>> currently is that, if a particular file is very large, then Hadoop first
>> operates on all splits of this large file, before asking for the next file.
>>
>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>> where I should make any changes to. Any other suggestions for this will be
>> welcome too.
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> I took logs at the underlying storage system and noticed the sequential
>>> behavior across files. However, most of the files I am testing with are big
>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>> possible that this is just the case as I have just 2 nodes and the files
>>> being processed are huge, Hence I am observing a sequential behaviour
>>> across files.
>>>
>>> Within the files however, the splits seems to be getting processed in
>>> parallel as I see random offsets within the file being accessed. That said,
>>> let me try logging at the NFS level to map IOs coming from different IP
>>> addresses (nodes) and analyze the pattern.
>>>
>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>> take a look and get back accordingly.
>>>
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>
>>>> When you said that the mappers seem to be accessing file sequentially,
>>>> why do you think so?
>>>>
>>>> NFS maybe changes something, but mappers shouldn't access file
>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>> test to verify it.
>>>>
>>>> The class you want to check out is the
>>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>>
>>>> The above code is the key how the split list is generated. If it
>>>> doesn't performance well for your underline storage system, you can always
>>>> write your own InputFormat to utilize your own storage system.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> From: atish.kathpal@gmail.com
>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>> Need guidance for 2. though!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>>
>>>> Hi
>>>>
>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>> Thanks a lot for the advice.
>>>>
>>>> *Two issues though*:
>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>> output directory, but no part-r file containing the actual results of
>>>> the wordcount job. However I am seeing the correct output on running MR
>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>> see no errors on the console.
>>>> Command used:
>>>> hadoop jar
>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>
>>>>
>>>> 2. I am observing that the mappers seem to be accessing files
>>>> sequentially, splitting the files across mappers, and then reading data in
>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>> one go, and then work on the splits of these files in parallel.
>>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>>> is coming from offline media (which takes ~5-10 seconds of time before
>>>> first bytes are received). So I would like all required files to be asked
>>>> at the onset itself from the NFS mount point. This way several offline
>>>> media will be spun up parallely and as the data from these media gets
>>>> available MR can process them.
>>>>
>>>> Would be glad to get inputs on these points!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>> Tip for those who are trying similar stuff::
>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>> but I was able to rectify this with help from:
>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>
>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>
>>>>
>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>
>>>> Yong raises an important issue: You have thrown out the I/O
>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>> would be interesting to know why you are taking this approach.
>>>> Chris
>>>>
>>>>
>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>
>>>> I believe the "-fs local" should be removed too. The reason is that
>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>
>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>> really a test mode.
>>>>
>>>> What you can do is to remove both "-fs local -jt local", but give the
>>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>>> filesystem instead of HDFS.
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///hduser/mount_point file:///results"
>>>>
>>>> Keep in mind followings:
>>>>
>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>> mounted in the same way.
>>>> 2) Even you can do that, but your sharing storage will be your
>>>> bottleneck. NFS won't work well for scalability.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> From: dsuiter@rdx.com
>>>> To: user@hadoop.apache.org
>>>>
>>>> I think most of your problem is coming from the options you are
>>>> setting:
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>
>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>> is set up right. You don't need to do anything special to point Hadoop
>>>> towards a NFS location, other than set up the NFS location properly and
>>>> make sure if you are directing to it by name that it will resolve to the
>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>> NFS location that happens to be mounted as a local filesystem object
>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>> network location of the folder that you want, it shouldn't make a
>>>> difference.
>>>>
>>>> *Devin Suiter*
>>>> Jr. Data Solutions Software Engineer
>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>
>>>>
>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Hello
>>>>
>>>> The picture below describes the deployment architecture I am trying
>>>> to achieve.
>>>> However, when I run the wordcount example code with the below
>>>> configuration, by issuing the command from the master node, I notice only
>>>> the master node spawning map tasks and completing the submitted job. Below
>>>> is the command I used:
>>>>
>>>> *hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>
>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>> while serving my data from the common NFS mount point running my filesystem
>>>> at the backend? Has any one tried such a setup before?*
>>>> [image: Inline image 1]
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
>
> Regards,
> ...Sudhakara.st
>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Thanks Sudhakara. That explains the behavior I am observing.
What I need: I would like files submitted as part of a MR job, to be asked
from the underlying storage in a batched form immediately after submitting
a job. This way the files that would be accessed as part of the MR job,
will get cached in the underlying storage, leading to improved MR
performance. I need to get this behavior out of MR as some of the files in
the underlying storage (NFS mount point) I am using, are not available
online and need to be fetched with some latency (~5-10secs or more). When
all such offline files are asked from the underlying store by MR in one go,
the underlying storage can make them available in parallel hence speeding
up the entire process.
Any clues on which classes I could start looking at in order to get this
behavior from MR would be helpful.
*TL;DR*
How to make MR perform batch IO for files to be accessed as part of
submitted job, before actual mappers have started processing those files?
Thanks
Atish
On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
> Hello Atish,
> Number of splits parallel processed at any instance is
> dependance upon the your cluster capacity i,e. How many mapper and reduce
> slots are configure in the cluster.
> The default each tasktracker run maximum 2 map tasks that will be run
> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
> single task tacker it can simultaneously two mapper and process to input
> splits. In your case your processing >1GB file, it as more than 8 input
> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
> mapper slots, maximum will process 4 input split any instance. Hence it
> looking like sequential behavior.
> For more info please have look on:
>
> mapred.tasktracker.map.tasks.maximum( for map slots)
> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>
>
>
>
> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Basically what I really need is that if I am submitting a job (say
>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>> issue immediate IO for reads on each of these files, such that the
>> underlying storage system can know that it has to cache those files and
>> present them back to Hadoop for processing. What I observe through by logs
>> currently is that, if a particular file is very large, then Hadoop first
>> operates on all splits of this large file, before asking for the next file.
>>
>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>> where I should make any changes to. Any other suggestions for this will be
>> welcome too.
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> I took logs at the underlying storage system and noticed the sequential
>>> behavior across files. However, most of the files I am testing with are big
>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>> possible that this is just the case as I have just 2 nodes and the files
>>> being processed are huge, Hence I am observing a sequential behaviour
>>> across files.
>>>
>>> Within the files however, the splits seems to be getting processed in
>>> parallel as I see random offsets within the file being accessed. That said,
>>> let me try logging at the NFS level to map IOs coming from different IP
>>> addresses (nodes) and analyze the pattern.
>>>
>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>> take a look and get back accordingly.
>>>
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>
>>>> When you said that the mappers seem to be accessing file sequentially,
>>>> why do you think so?
>>>>
>>>> NFS maybe changes something, but mappers shouldn't access file
>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>> test to verify it.
>>>>
>>>> The class you want to check out is the
>>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>>
>>>> The above code is the key how the split list is generated. If it
>>>> doesn't performance well for your underline storage system, you can always
>>>> write your own InputFormat to utilize your own storage system.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> From: atish.kathpal@gmail.com
>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>> Need guidance for 2. though!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>>
>>>> Hi
>>>>
>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>> Thanks a lot for the advice.
>>>>
>>>> *Two issues though*:
>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>> output directory, but no part-r file containing the actual results of
>>>> the wordcount job. However I am seeing the correct output on running MR
>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>> see no errors on the console.
>>>> Command used:
>>>> hadoop jar
>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>
>>>>
>>>> 2. I am observing that the mappers seem to be accessing files
>>>> sequentially, splitting the files across mappers, and then reading data in
>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>> one go, and then work on the splits of these files in parallel.
>>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>>> is coming from offline media (which takes ~5-10 seconds of time before
>>>> first bytes are received). So I would like all required files to be asked
>>>> at the onset itself from the NFS mount point. This way several offline
>>>> media will be spun up parallely and as the data from these media gets
>>>> available MR can process them.
>>>>
>>>> Would be glad to get inputs on these points!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>> Tip for those who are trying similar stuff::
>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>> but I was able to rectify this with help from:
>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>
>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>
>>>>
>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>
>>>> Yong raises an important issue: You have thrown out the I/O
>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>> would be interesting to know why you are taking this approach.
>>>> Chris
>>>>
>>>>
>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>
>>>> I believe the "-fs local" should be removed too. The reason is that
>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>
>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>> really a test mode.
>>>>
>>>> What you can do is to remove both "-fs local -jt local", but give the
>>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>>> filesystem instead of HDFS.
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///hduser/mount_point file:///results"
>>>>
>>>> Keep in mind followings:
>>>>
>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>> mounted in the same way.
>>>> 2) Even you can do that, but your sharing storage will be your
>>>> bottleneck. NFS won't work well for scalability.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> From: dsuiter@rdx.com
>>>> To: user@hadoop.apache.org
>>>>
>>>> I think most of your problem is coming from the options you are
>>>> setting:
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>
>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>> is set up right. You don't need to do anything special to point Hadoop
>>>> towards a NFS location, other than set up the NFS location properly and
>>>> make sure if you are directing to it by name that it will resolve to the
>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>> NFS location that happens to be mounted as a local filesystem object
>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>> network location of the folder that you want, it shouldn't make a
>>>> difference.
>>>>
>>>> *Devin Suiter*
>>>> Jr. Data Solutions Software Engineer
>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>
>>>>
>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Hello
>>>>
>>>> The picture below describes the deployment architecture I am trying
>>>> to achieve.
>>>> However, when I run the wordcount example code with the below
>>>> configuration, by issuing the command from the master node, I notice only
>>>> the master node spawning map tasks and completing the submitted job. Below
>>>> is the command I used:
>>>>
>>>> *hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>
>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>> while serving my data from the common NFS mount point running my filesystem
>>>> at the backend? Has any one tried such a setup before?*
>>>> [image: Inline image 1]
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
>
> Regards,
> ...Sudhakara.st
>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Thanks Sudhakara. That explains the behavior I am observing.
What I need: I would like files submitted as part of a MR job, to be asked
from the underlying storage in a batched form immediately after submitting
a job. This way the files that would be accessed as part of the MR job,
will get cached in the underlying storage, leading to improved MR
performance. I need to get this behavior out of MR as some of the files in
the underlying storage (NFS mount point) I am using, are not available
online and need to be fetched with some latency (~5-10secs or more). When
all such offline files are asked from the underlying store by MR in one go,
the underlying storage can make them available in parallel hence speeding
up the entire process.
Any clues on which classes I could start looking at in order to get this
behavior from MR would be helpful.
*TL;DR*
How to make MR perform batch IO for files to be accessed as part of
submitted job, before actual mappers have started processing those files?
Thanks
Atish
On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
> Hello Atish,
> Number of splits parallel processed at any instance is
> dependance upon the your cluster capacity i,e. How many mapper and reduce
> slots are configure in the cluster.
> The default each tasktracker run maximum 2 map tasks that will be run
> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
> single task tacker it can simultaneously two mapper and process to input
> splits. In your case your processing >1GB file, it as more than 8 input
> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
> mapper slots, maximum will process 4 input split any instance. Hence it
> looking like sequential behavior.
> For more info please have look on:
>
> mapred.tasktracker.map.tasks.maximum( for map slots)
> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>
>
>
>
> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Basically what I really need is that if I am submitting a job (say
>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>> issue immediate IO for reads on each of these files, such that the
>> underlying storage system can know that it has to cache those files and
>> present them back to Hadoop for processing. What I observe through by logs
>> currently is that, if a particular file is very large, then Hadoop first
>> operates on all splits of this large file, before asking for the next file.
>>
>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>> where I should make any changes to. Any other suggestions for this will be
>> welcome too.
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> I took logs at the underlying storage system and noticed the sequential
>>> behavior across files. However, most of the files I am testing with are big
>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>> possible that this is just the case as I have just 2 nodes and the files
>>> being processed are huge, Hence I am observing a sequential behaviour
>>> across files.
>>>
>>> Within the files however, the splits seems to be getting processed in
>>> parallel as I see random offsets within the file being accessed. That said,
>>> let me try logging at the NFS level to map IOs coming from different IP
>>> addresses (nodes) and analyze the pattern.
>>>
>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>> take a look and get back accordingly.
>>>
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>
>>>> When you said that the mappers seem to be accessing file sequentially,
>>>> why do you think so?
>>>>
>>>> NFS maybe changes something, but mappers shouldn't access file
>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>> test to verify it.
>>>>
>>>> The class you want to check out is the
>>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>>
>>>> The above code is the key how the split list is generated. If it
>>>> doesn't performance well for your underline storage system, you can always
>>>> write your own InputFormat to utilize your own storage system.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> From: atish.kathpal@gmail.com
>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>> Need guidance for 2. though!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>>
>>>> Hi
>>>>
>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>> Thanks a lot for the advice.
>>>>
>>>> *Two issues though*:
>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>> output directory, but no part-r file containing the actual results of
>>>> the wordcount job. However I am seeing the correct output on running MR
>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>> see no errors on the console.
>>>> Command used:
>>>> hadoop jar
>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>
>>>>
>>>> 2. I am observing that the mappers seem to be accessing files
>>>> sequentially, splitting the files across mappers, and then reading data in
>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>> one go, and then work on the splits of these files in parallel.
>>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>>> is coming from offline media (which takes ~5-10 seconds of time before
>>>> first bytes are received). So I would like all required files to be asked
>>>> at the onset itself from the NFS mount point. This way several offline
>>>> media will be spun up parallely and as the data from these media gets
>>>> available MR can process them.
>>>>
>>>> Would be glad to get inputs on these points!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>> Tip for those who are trying similar stuff::
>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>> but I was able to rectify this with help from:
>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>
>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>
>>>>
>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>
>>>> Yong raises an important issue: You have thrown out the I/O
>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>> would be interesting to know why you are taking this approach.
>>>> Chris
>>>>
>>>>
>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>
>>>> I believe the "-fs local" should be removed too. The reason is that
>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>
>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>> really a test mode.
>>>>
>>>> What you can do is to remove both "-fs local -jt local", but give the
>>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>>> filesystem instead of HDFS.
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///hduser/mount_point file:///results"
>>>>
>>>> Keep in mind followings:
>>>>
>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>> mounted in the same way.
>>>> 2) Even you can do that, but your sharing storage will be your
>>>> bottleneck. NFS won't work well for scalability.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> From: dsuiter@rdx.com
>>>> To: user@hadoop.apache.org
>>>>
>>>> I think most of your problem is coming from the options you are
>>>> setting:
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>
>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>> is set up right. You don't need to do anything special to point Hadoop
>>>> towards a NFS location, other than set up the NFS location properly and
>>>> make sure if you are directing to it by name that it will resolve to the
>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>> NFS location that happens to be mounted as a local filesystem object
>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>> network location of the folder that you want, it shouldn't make a
>>>> difference.
>>>>
>>>> *Devin Suiter*
>>>> Jr. Data Solutions Software Engineer
>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>
>>>>
>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Hello
>>>>
>>>> The picture below describes the deployment architecture I am trying
>>>> to achieve.
>>>> However, when I run the wordcount example code with the below
>>>> configuration, by issuing the command from the master node, I notice only
>>>> the master node spawning map tasks and completing the submitted job. Below
>>>> is the command I used:
>>>>
>>>> *hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>
>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>> while serving my data from the common NFS mount point running my filesystem
>>>> at the backend? Has any one tried such a setup before?*
>>>> [image: Inline image 1]
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
>
> Regards,
> ...Sudhakara.st
>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Thanks Sudhakara. That explains the behavior I am observing.
What I need: I would like files submitted as part of a MR job, to be asked
from the underlying storage in a batched form immediately after submitting
a job. This way the files that would be accessed as part of the MR job,
will get cached in the underlying storage, leading to improved MR
performance. I need to get this behavior out of MR as some of the files in
the underlying storage (NFS mount point) I am using, are not available
online and need to be fetched with some latency (~5-10secs or more). When
all such offline files are asked from the underlying store by MR in one go,
the underlying storage can make them available in parallel hence speeding
up the entire process.
Any clues on which classes I could start looking at in order to get this
behavior from MR would be helpful.
*TL;DR*
How to make MR perform batch IO for files to be accessed as part of
submitted job, before actual mappers have started processing those files?
Thanks
Atish
On Tue, Jan 14, 2014 at 12:23 PM, sudhakara st <su...@gmail.com>wrote:
> Hello Atish,
> Number of splits parallel processed at any instance is
> dependance upon the your cluster capacity i,e. How many mapper and reduce
> slots are configure in the cluster.
> The default each tasktracker run maximum 2 map tasks that will be run
> simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
> single task tacker it can simultaneously two mapper and process to input
> splits. In your case your processing >1GB file, it as more than 8 input
> splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
> mapper slots, maximum will process 4 input split any instance. Hence it
> looking like sequential behavior.
> For more info please have look on:
>
> mapred.tasktracker.map.tasks.maximum( for map slots)
> mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
>
>
>
>
> On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Basically what I really need is that if I am submitting a job (say
>> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
>> issue immediate IO for reads on each of these files, such that the
>> underlying storage system can know that it has to cache those files and
>> present them back to Hadoop for processing. What I observe through by logs
>> currently is that, if a particular file is very large, then Hadoop first
>> operates on all splits of this large file, before asking for the next file.
>>
>> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
>> where I should make any changes to. Any other suggestions for this will be
>> welcome too.
>>
>>
>>
>>
>> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>>> I took logs at the underlying storage system and noticed the sequential
>>> behavior across files. However, most of the files I am testing with are big
>>> > 1GB in size, hence leading to several splits. From the logs I notice that
>>> File2 is not being accessed until all blocks of File1 have been read. It is
>>> possible that this is just the case as I have just 2 nodes and the files
>>> being processed are huge, Hence I am observing a sequential behaviour
>>> across files.
>>>
>>> Within the files however, the splits seems to be getting processed in
>>> parallel as I see random offsets within the file being accessed. That said,
>>> let me try logging at the NFS level to map IOs coming from different IP
>>> addresses (nodes) and analyze the pattern.
>>>
>>> Thanks for your suggestion on looking at the getSplits() function. Will
>>> take a look and get back accordingly.
>>>
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>>
>>>> When you said that the mappers seem to be accessing file sequentially,
>>>> why do you think so?
>>>>
>>>> NFS maybe changes something, but mappers shouldn't access file
>>>> sequentially. NFS could make the file unsplittable, but you need to more
>>>> test to verify it.
>>>>
>>>> The class you want to check out is the
>>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>>
>>>> The above code is the key how the split list is generated. If it
>>>> doesn't performance well for your underline storage system, you can always
>>>> write your own InputFormat to utilize your own storage system.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> From: atish.kathpal@gmail.com
>>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>>
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> To: user@hadoop.apache.org
>>>>
>>>>
>>>> Figured out 1. The output of the reduce was going to the slave node,
>>>> while I was looking for it in the master node. Which is perfectly fine.
>>>> Need guidance for 2. though!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>>
>>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>>
>>>> Hi
>>>>
>>>> By giving the complete URI, the MR jobs worked across both nodes.
>>>> Thanks a lot for the advice.
>>>>
>>>> *Two issues though*:
>>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>>> output directory, but no part-r file containing the actual results of
>>>> the wordcount job. However I am seeing the correct output on running MR
>>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>>> see no errors on the console.
>>>> Command used:
>>>> hadoop jar
>>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>>
>>>>
>>>> 2. I am observing that the mappers seem to be accessing files
>>>> sequentially, splitting the files across mappers, and then reading data in
>>>> parallelel, then moving on to the next file. What I want instead is that,
>>>> files themselves should be accessed in parallel, that is, if there are 10
>>>> files to be MRed, then MR should ask for each of these files in parallel in
>>>> one go, and then work on the splits of these files in parallel.
>>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>>> is coming from offline media (which takes ~5-10 seconds of time before
>>>> first bytes are received). So I would like all required files to be asked
>>>> at the onset itself from the NFS mount point. This way several offline
>>>> media will be spun up parallely and as the data from these media gets
>>>> available MR can process them.
>>>>
>>>> Would be glad to get inputs on these points!
>>>>
>>>> Thanks
>>>> Atish
>>>>
>>>> Tip for those who are trying similar stuff::
>>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>>> but I was able to rectify this with help from:
>>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>>
>>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>>
>>>>
>>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>>
>>>> Yong raises an important issue: You have thrown out the I/O
>>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>>> would be interesting to know why you are taking this approach.
>>>> Chris
>>>>
>>>>
>>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>>
>>>> I believe the "-fs local" should be removed too. The reason is that
>>>> even you have a dedicated JobTracker after removing "-jt local", but with
>>>> "-fs local", I believe that all the mappers will be run sequentially.
>>>>
>>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>>> really a test mode.
>>>>
>>>> What you can do is to remove both "-fs local -jt local", but give the
>>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>>> filesystem instead of HDFS.
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount file:///hduser/mount_point file:///results"
>>>>
>>>> Keep in mind followings:
>>>>
>>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>>> mounted in the same way.
>>>> 2) Even you can do that, but your sharing storage will be your
>>>> bottleneck. NFS won't work well for scalability.
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>>> filesystem
>>>> From: dsuiter@rdx.com
>>>> To: user@hadoop.apache.org
>>>>
>>>> I think most of your problem is coming from the options you are
>>>> setting:
>>>>
>>>> "hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>>
>>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>>> *-jt* argument and it should run in distributed mode if your cluster
>>>> is set up right. You don't need to do anything special to point Hadoop
>>>> towards a NFS location, other than set up the NFS location properly and
>>>> make sure if you are directing to it by name that it will resolve to the
>>>> right address. Hadoop doesn't care where it is, as long as it can read from
>>>> and write to it. The fact that you are telling it to read/write from/to a
>>>> NFS location that happens to be mounted as a local filesystem object
>>>> doesn't matter - you could direct it to the local /hduser/ path and set the
>>>> -fs local option, and it would end up on the NFS mount, because that's
>>>> where the NFS mount actually exists, or you could direct it to the absolute
>>>> network location of the folder that you want, it shouldn't make a
>>>> difference.
>>>>
>>>> *Devin Suiter*
>>>> Jr. Data Solutions Software Engineer
>>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>>> Google Voice: 412-256-8556 | www.rdx.com
>>>>
>>>>
>>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <atish.kathpal@gmail.com
>>>> > wrote:
>>>>
>>>> Hello
>>>>
>>>> The picture below describes the deployment architecture I am trying
>>>> to achieve.
>>>> However, when I run the wordcount example code with the below
>>>> configuration, by issuing the command from the master node, I notice only
>>>> the master node spawning map tasks and completing the submitted job. Below
>>>> is the command I used:
>>>>
>>>> *hadoop jar
>>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>>
>>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>>> while serving my data from the common NFS mount point running my filesystem
>>>> at the backend? Has any one tried such a setup before?*
>>>> [image: Inline image 1]
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
>
> Regards,
> ...Sudhakara.st
>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Hello Atish,
Number of splits parallel processed at any instance is dependance
upon the your cluster capacity i,e. How many mapper and reduce slots are
configure in the cluster.
The default each tasktracker run maximum 2 map tasks that will be run
simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
single task tacker it can simultaneously two mapper and process to input
splits. In your case your processing >1GB file, it as more than 8 input
splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
mapper slots, maximum will process 4 input split any instance. Hence it
looking like sequential behavior.
For more info please have look on:
mapred.tasktracker.map.tasks.maximum( for map slots)
mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
> Basically what I really need is that if I am submitting a job (say
> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
> issue immediate IO for reads on each of these files, such that the
> underlying storage system can know that it has to cache those files and
> present them back to Hadoop for processing. What I observe through by logs
> currently is that, if a particular file is very large, then Hadoop first
> operates on all splits of this large file, before asking for the next file.
>
> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
> where I should make any changes to. Any other suggestions for this will be
> welcome too.
>
>
>
>
> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> I took logs at the underlying storage system and noticed the sequential
>> behavior across files. However, most of the files I am testing with are big
>> > 1GB in size, hence leading to several splits. From the logs I notice that
>> File2 is not being accessed until all blocks of File1 have been read. It is
>> possible that this is just the case as I have just 2 nodes and the files
>> being processed are huge, Hence I am observing a sequential behaviour
>> across files.
>>
>> Within the files however, the splits seems to be getting processed in
>> parallel as I see random offsets within the file being accessed. That said,
>> let me try logging at the NFS level to map IOs coming from different IP
>> addresses (nodes) and analyze the pattern.
>>
>> Thanks for your suggestion on looking at the getSplits() function. Will
>> take a look and get back accordingly.
>>
>>
>> Thanks
>> Atish
>>
>>
>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>
>>> When you said that the mappers seem to be accessing file sequentially,
>>> why do you think so?
>>>
>>> NFS maybe changes something, but mappers shouldn't access file
>>> sequentially. NFS could make the file unsplittable, but you need to more
>>> test to verify it.
>>>
>>> The class you want to check out is the
>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>
>>> The above code is the key how the split list is generated. If it doesn't
>>> performance well for your underline storage system, you can always write
>>> your own InputFormat to utilize your own storage system.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> From: atish.kathpal@gmail.com
>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> To: user@hadoop.apache.org
>>>
>>>
>>> Figured out 1. The output of the reduce was going to the slave node,
>>> while I was looking for it in the master node. Which is perfectly fine.
>>> Need guidance for 2. though!
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hi
>>>
>>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>>> a lot for the advice.
>>>
>>> *Two issues though*:
>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>> output directory, but no part-r file containing the actual results of
>>> the wordcount job. However I am seeing the correct output on running MR
>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>> see no errors on the console.
>>> Command used:
>>> hadoop jar
>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>
>>>
>>> 2. I am observing that the mappers seem to be accessing files
>>> sequentially, splitting the files across mappers, and then reading data in
>>> parallelel, then moving on to the next file. What I want instead is that,
>>> files themselves should be accessed in parallel, that is, if there are 10
>>> files to be MRed, then MR should ask for each of these files in parallel in
>>> one go, and then work on the splits of these files in parallel.
>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>> is coming from offline media (which takes ~5-10 seconds of time before
>>> first bytes are received). So I would like all required files to be asked
>>> at the onset itself from the NFS mount point. This way several offline
>>> media will be spun up parallely and as the data from these media gets
>>> available MR can process them.
>>>
>>> Would be glad to get inputs on these points!
>>>
>>> Thanks
>>> Atish
>>>
>>> Tip for those who are trying similar stuff::
>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>> but I was able to rectify this with help from:
>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>
>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>
>>>
>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Hello Atish,
Number of splits parallel processed at any instance is dependance
upon the your cluster capacity i,e. How many mapper and reduce slots are
configure in the cluster.
The default each tasktracker run maximum 2 map tasks that will be run
simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
single task tacker it can simultaneously two mapper and process to input
splits. In your case your processing >1GB file, it as more than 8 input
splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
mapper slots, maximum will process 4 input split any instance. Hence it
looking like sequential behavior.
For more info please have look on:
mapred.tasktracker.map.tasks.maximum( for map slots)
mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
> Basically what I really need is that if I am submitting a job (say
> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
> issue immediate IO for reads on each of these files, such that the
> underlying storage system can know that it has to cache those files and
> present them back to Hadoop for processing. What I observe through by logs
> currently is that, if a particular file is very large, then Hadoop first
> operates on all splits of this large file, before asking for the next file.
>
> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
> where I should make any changes to. Any other suggestions for this will be
> welcome too.
>
>
>
>
> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> I took logs at the underlying storage system and noticed the sequential
>> behavior across files. However, most of the files I am testing with are big
>> > 1GB in size, hence leading to several splits. From the logs I notice that
>> File2 is not being accessed until all blocks of File1 have been read. It is
>> possible that this is just the case as I have just 2 nodes and the files
>> being processed are huge, Hence I am observing a sequential behaviour
>> across files.
>>
>> Within the files however, the splits seems to be getting processed in
>> parallel as I see random offsets within the file being accessed. That said,
>> let me try logging at the NFS level to map IOs coming from different IP
>> addresses (nodes) and analyze the pattern.
>>
>> Thanks for your suggestion on looking at the getSplits() function. Will
>> take a look and get back accordingly.
>>
>>
>> Thanks
>> Atish
>>
>>
>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>
>>> When you said that the mappers seem to be accessing file sequentially,
>>> why do you think so?
>>>
>>> NFS maybe changes something, but mappers shouldn't access file
>>> sequentially. NFS could make the file unsplittable, but you need to more
>>> test to verify it.
>>>
>>> The class you want to check out is the
>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>
>>> The above code is the key how the split list is generated. If it doesn't
>>> performance well for your underline storage system, you can always write
>>> your own InputFormat to utilize your own storage system.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> From: atish.kathpal@gmail.com
>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> To: user@hadoop.apache.org
>>>
>>>
>>> Figured out 1. The output of the reduce was going to the slave node,
>>> while I was looking for it in the master node. Which is perfectly fine.
>>> Need guidance for 2. though!
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hi
>>>
>>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>>> a lot for the advice.
>>>
>>> *Two issues though*:
>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>> output directory, but no part-r file containing the actual results of
>>> the wordcount job. However I am seeing the correct output on running MR
>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>> see no errors on the console.
>>> Command used:
>>> hadoop jar
>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>
>>>
>>> 2. I am observing that the mappers seem to be accessing files
>>> sequentially, splitting the files across mappers, and then reading data in
>>> parallelel, then moving on to the next file. What I want instead is that,
>>> files themselves should be accessed in parallel, that is, if there are 10
>>> files to be MRed, then MR should ask for each of these files in parallel in
>>> one go, and then work on the splits of these files in parallel.
>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>> is coming from offline media (which takes ~5-10 seconds of time before
>>> first bytes are received). So I would like all required files to be asked
>>> at the onset itself from the NFS mount point. This way several offline
>>> media will be spun up parallely and as the data from these media gets
>>> available MR can process them.
>>>
>>> Would be glad to get inputs on these points!
>>>
>>> Thanks
>>> Atish
>>>
>>> Tip for those who are trying similar stuff::
>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>> but I was able to rectify this with help from:
>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>
>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>
>>>
>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Hello Atish,
Number of splits parallel processed at any instance is dependance
upon the your cluster capacity i,e. How many mapper and reduce slots are
configure in the cluster.
The default each tasktracker run maximum 2 map tasks that will be run
simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
single task tacker it can simultaneously two mapper and process to input
splits. In your case your processing >1GB file, it as more than 8 input
splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
mapper slots, maximum will process 4 input split any instance. Hence it
looking like sequential behavior.
For more info please have look on:
mapred.tasktracker.map.tasks.maximum( for map slots)
mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
> Basically what I really need is that if I am submitting a job (say
> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
> issue immediate IO for reads on each of these files, such that the
> underlying storage system can know that it has to cache those files and
> present them back to Hadoop for processing. What I observe through by logs
> currently is that, if a particular file is very large, then Hadoop first
> operates on all splits of this large file, before asking for the next file.
>
> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
> where I should make any changes to. Any other suggestions for this will be
> welcome too.
>
>
>
>
> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> I took logs at the underlying storage system and noticed the sequential
>> behavior across files. However, most of the files I am testing with are big
>> > 1GB in size, hence leading to several splits. From the logs I notice that
>> File2 is not being accessed until all blocks of File1 have been read. It is
>> possible that this is just the case as I have just 2 nodes and the files
>> being processed are huge, Hence I am observing a sequential behaviour
>> across files.
>>
>> Within the files however, the splits seems to be getting processed in
>> parallel as I see random offsets within the file being accessed. That said,
>> let me try logging at the NFS level to map IOs coming from different IP
>> addresses (nodes) and analyze the pattern.
>>
>> Thanks for your suggestion on looking at the getSplits() function. Will
>> take a look and get back accordingly.
>>
>>
>> Thanks
>> Atish
>>
>>
>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>
>>> When you said that the mappers seem to be accessing file sequentially,
>>> why do you think so?
>>>
>>> NFS maybe changes something, but mappers shouldn't access file
>>> sequentially. NFS could make the file unsplittable, but you need to more
>>> test to verify it.
>>>
>>> The class you want to check out is the
>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>
>>> The above code is the key how the split list is generated. If it doesn't
>>> performance well for your underline storage system, you can always write
>>> your own InputFormat to utilize your own storage system.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> From: atish.kathpal@gmail.com
>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> To: user@hadoop.apache.org
>>>
>>>
>>> Figured out 1. The output of the reduce was going to the slave node,
>>> while I was looking for it in the master node. Which is perfectly fine.
>>> Need guidance for 2. though!
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hi
>>>
>>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>>> a lot for the advice.
>>>
>>> *Two issues though*:
>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>> output directory, but no part-r file containing the actual results of
>>> the wordcount job. However I am seeing the correct output on running MR
>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>> see no errors on the console.
>>> Command used:
>>> hadoop jar
>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>
>>>
>>> 2. I am observing that the mappers seem to be accessing files
>>> sequentially, splitting the files across mappers, and then reading data in
>>> parallelel, then moving on to the next file. What I want instead is that,
>>> files themselves should be accessed in parallel, that is, if there are 10
>>> files to be MRed, then MR should ask for each of these files in parallel in
>>> one go, and then work on the splits of these files in parallel.
>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>> is coming from offline media (which takes ~5-10 seconds of time before
>>> first bytes are received). So I would like all required files to be asked
>>> at the onset itself from the NFS mount point. This way several offline
>>> media will be spun up parallely and as the data from these media gets
>>> available MR can process them.
>>>
>>> Would be glad to get inputs on these points!
>>>
>>> Thanks
>>> Atish
>>>
>>> Tip for those who are trying similar stuff::
>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>> but I was able to rectify this with help from:
>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>
>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>
>>>
>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by sudhakara st <su...@gmail.com>.
Hello Atish,
Number of splits parallel processed at any instance is dependance
upon the your cluster capacity i,e. How many mapper and reduce slots are
configure in the cluster.
The default each tasktracker run maximum 2 map tasks that will be run
simultaneously( mapred.tasktracker.map.tasks.maximum). That mean if have
single task tacker it can simultaneously two mapper and process to input
splits. In your case your processing >1GB file, it as more than 8 input
splits(assumed 128MB block size), you have 2 task tackers mean defualt 4
mapper slots, maximum will process 4 input split any instance. Hence it
looking like sequential behavior.
For more info please have look on:
mapred.tasktracker.map.tasks.maximum( for map slots)
mapred.tasktracker.reduce.tasks.maximum (for reduce slot)
On Sun, Jan 12, 2014 at 4:14 PM, Atish Kathpal <at...@gmail.com>wrote:
> Basically what I really need is that if I am submitting a job (say
> wordcount) consisting of 1000s (or even more) of big files, Hadoop should
> issue immediate IO for reads on each of these files, such that the
> underlying storage system can know that it has to cache those files and
> present them back to Hadoop for processing. What I observe through by logs
> currently is that, if a particular file is very large, then Hadoop first
> operates on all splits of this large file, before asking for the next file.
>
> I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is
> where I should make any changes to. Any other suggestions for this will be
> welcome too.
>
>
>
>
> On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> I took logs at the underlying storage system and noticed the sequential
>> behavior across files. However, most of the files I am testing with are big
>> > 1GB in size, hence leading to several splits. From the logs I notice that
>> File2 is not being accessed until all blocks of File1 have been read. It is
>> possible that this is just the case as I have just 2 nodes and the files
>> being processed are huge, Hence I am observing a sequential behaviour
>> across files.
>>
>> Within the files however, the splits seems to be getting processed in
>> parallel as I see random offsets within the file being accessed. That said,
>> let me try logging at the NFS level to map IOs coming from different IP
>> addresses (nodes) and analyze the pattern.
>>
>> Thanks for your suggestion on looking at the getSplits() function. Will
>> take a look and get back accordingly.
>>
>>
>> Thanks
>> Atish
>>
>>
>> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>>
>>> When you said that the mappers seem to be accessing file sequentially,
>>> why do you think so?
>>>
>>> NFS maybe changes something, but mappers shouldn't access file
>>> sequentially. NFS could make the file unsplittable, but you need to more
>>> test to verify it.
>>>
>>> The class you want to check out is the
>>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>>
>>> The above code is the key how the split list is generated. If it doesn't
>>> performance well for your underline storage system, you can always write
>>> your own InputFormat to utilize your own storage system.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> From: atish.kathpal@gmail.com
>>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>>
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> To: user@hadoop.apache.org
>>>
>>>
>>> Figured out 1. The output of the reduce was going to the slave node,
>>> while I was looking for it in the master node. Which is perfectly fine.
>>> Need guidance for 2. though!
>>>
>>> Thanks
>>> Atish
>>>
>>>
>>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hi
>>>
>>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>>> a lot for the advice.
>>>
>>> *Two issues though*:
>>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>>> output directory, but no part-r file containing the actual results of
>>> the wordcount job. However I am seeing the correct output on running MR
>>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>>> see no errors on the console.
>>> Command used:
>>> hadoop jar
>>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>>
>>>
>>> 2. I am observing that the mappers seem to be accessing files
>>> sequentially, splitting the files across mappers, and then reading data in
>>> parallelel, then moving on to the next file. What I want instead is that,
>>> files themselves should be accessed in parallel, that is, if there are 10
>>> files to be MRed, then MR should ask for each of these files in parallel in
>>> one go, and then work on the splits of these files in parallel.
>>> *Why do I need this?* Some of the data coming from the NFS mount point
>>> is coming from offline media (which takes ~5-10 seconds of time before
>>> first bytes are received). So I would like all required files to be asked
>>> at the onset itself from the NFS mount point. This way several offline
>>> media will be spun up parallely and as the data from these media gets
>>> available MR can process them.
>>>
>>> Would be glad to get inputs on these points!
>>>
>>> Thanks
>>> Atish
>>>
>>> Tip for those who are trying similar stuff::
>>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>>> but I was able to rectify this with help from:
>>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>>> test the suggestions made by Yong and Devin and get back to you guys.
>>>
>>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>>> long as I don't have to move the data to a dedicated HDFS cluster.
>>>
>>>
>>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
--
Regards,
...Sudhakara.st
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Basically what I really need is that if I am submitting a job (say
wordcount) consisting of 1000s (or even more) of big files, Hadoop should
issue immediate IO for reads on each of these files, such that the
underlying storage system can know that it has to cache those files and
present them back to Hadoop for processing. What I observe through by logs
currently is that, if a particular file is very large, then Hadoop first
operates on all splits of this large file, before asking for the next file.
I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is where
I should make any changes to. Any other suggestions for this will be
welcome too.
On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
> I took logs at the underlying storage system and noticed the sequential
> behavior across files. However, most of the files I am testing with are big
> > 1GB in size, hence leading to several splits. From the logs I notice that
> File2 is not being accessed until all blocks of File1 have been read. It is
> possible that this is just the case as I have just 2 nodes and the files
> being processed are huge, Hence I am observing a sequential behaviour
> across files.
>
> Within the files however, the splits seems to be getting processed in
> parallel as I see random offsets within the file being accessed. That said,
> let me try logging at the NFS level to map IOs coming from different IP
> addresses (nodes) and analyze the pattern.
>
> Thanks for your suggestion on looking at the getSplits() function. Will
> take a look and get back accordingly.
>
>
> Thanks
> Atish
>
>
> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>
>> When you said that the mappers seem to be accessing file sequentially,
>> why do you think so?
>>
>> NFS maybe changes something, but mappers shouldn't access file
>> sequentially. NFS could make the file unsplittable, but you need to more
>> test to verify it.
>>
>> The class you want to check out is the
>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>
>> The above code is the key how the split list is generated. If it doesn't
>> performance well for your underline storage system, you can always write
>> your own InputFormat to utilize your own storage system.
>>
>> Yong
>>
>> ------------------------------
>> From: atish.kathpal@gmail.com
>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> To: user@hadoop.apache.org
>>
>>
>> Figured out 1. The output of the reduce was going to the slave node,
>> while I was looking for it in the master node. Which is perfectly fine.
>> Need guidance for 2. though!
>>
>> Thanks
>> Atish
>>
>>
>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hi
>>
>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>> a lot for the advice.
>>
>> *Two issues though*:
>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>> output directory, but no part-r file containing the actual results of
>> the wordcount job. However I am seeing the correct output on running MR
>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>> see no errors on the console.
>> Command used:
>> hadoop jar
>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>
>>
>> 2. I am observing that the mappers seem to be accessing files
>> sequentially, splitting the files across mappers, and then reading data in
>> parallelel, then moving on to the next file. What I want instead is that,
>> files themselves should be accessed in parallel, that is, if there are 10
>> files to be MRed, then MR should ask for each of these files in parallel in
>> one go, and then work on the splits of these files in parallel.
>> *Why do I need this?* Some of the data coming from the NFS mount point
>> is coming from offline media (which takes ~5-10 seconds of time before
>> first bytes are received). So I would like all required files to be asked
>> at the onset itself from the NFS mount point. This way several offline
>> media will be spun up parallely and as the data from these media gets
>> available MR can process them.
>>
>> Would be glad to get inputs on these points!
>>
>> Thanks
>> Atish
>>
>> Tip for those who are trying similar stuff::
>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>> but I was able to rectify this with help from:
>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>
>>
>>
>>
>>
>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>> Yong raises an important issue: You have thrown out the I/O advantages
>> of HDFS and also thrown out the advantages of data locality. It would be
>> interesting to know why you are taking this approach.
>> Chris
>>
>>
>> On 12/20/2013 9:28 AM, java8964 wrote:
>>
>> I believe the "-fs local" should be removed too. The reason is that even
>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>> local", I believe that all the mappers will be run sequentially.
>>
>> "-fs local" will force the mapreducer run in "local" mode, which is
>> really a test mode.
>>
>> What you can do is to remove both "-fs local -jt local", but give the
>> FULL URI of the input and output path, to tell Hadoop that they are local
>> filesystem instead of HDFS.
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///hduser/mount_point file:///results"
>>
>> Keep in mind followings:
>>
>> 1) The NFS mount need to be available in all your Task Nodes, and
>> mounted in the same way.
>> 2) Even you can do that, but your sharing storage will be your
>> bottleneck. NFS won't work well for scalability.
>>
>> Yong
>>
>> ------------------------------
>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> From: dsuiter@rdx.com
>> To: user@hadoop.apache.org
>>
>> I think most of your problem is coming from the options you are setting:
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>
>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>> *-jt* argument and it should run in distributed mode if your cluster is
>> set up right. You don't need to do anything special to point Hadoop towards
>> a NFS location, other than set up the NFS location properly and make sure
>> if you are directing to it by name that it will resolve to the right
>> address. Hadoop doesn't care where it is, as long as it can read from and
>> write to it. The fact that you are telling it to read/write from/to a NFS
>> location that happens to be mounted as a local filesystem object doesn't
>> matter - you could direct it to the local /hduser/ path and set the -fs
>> local option, and it would end up on the NFS mount, because that's where
>> the NFS mount actually exists, or you could direct it to the absolute
>> network location of the folder that you want, it shouldn't make a
>> difference.
>>
>> *Devin Suiter*
>> Jr. Data Solutions Software Engineer
>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>> Google Voice: 412-256-8556 | www.rdx.com
>>
>>
>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hello
>>
>> The picture below describes the deployment architecture I am trying to
>> achieve.
>> However, when I run the wordcount example code with the below
>> configuration, by issuing the command from the master node, I notice only
>> the master node spawning map tasks and completing the submitted job. Below
>> is the command I used:
>>
>> *hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>
>> *Question: How can I leverage both the hadoop nodes for running MR,
>> while serving my data from the common NFS mount point running my filesystem
>> at the backend? Has any one tried such a setup before?*
>> [image: Inline image 1]
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Basically what I really need is that if I am submitting a job (say
wordcount) consisting of 1000s (or even more) of big files, Hadoop should
issue immediate IO for reads on each of these files, such that the
underlying storage system can know that it has to cache those files and
present them back to Hadoop for processing. What I observe through by logs
currently is that, if a particular file is very large, then Hadoop first
operates on all splits of this large file, before asking for the next file.
I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is where
I should make any changes to. Any other suggestions for this will be
welcome too.
On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
> I took logs at the underlying storage system and noticed the sequential
> behavior across files. However, most of the files I am testing with are big
> > 1GB in size, hence leading to several splits. From the logs I notice that
> File2 is not being accessed until all blocks of File1 have been read. It is
> possible that this is just the case as I have just 2 nodes and the files
> being processed are huge, Hence I am observing a sequential behaviour
> across files.
>
> Within the files however, the splits seems to be getting processed in
> parallel as I see random offsets within the file being accessed. That said,
> let me try logging at the NFS level to map IOs coming from different IP
> addresses (nodes) and analyze the pattern.
>
> Thanks for your suggestion on looking at the getSplits() function. Will
> take a look and get back accordingly.
>
>
> Thanks
> Atish
>
>
> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>
>> When you said that the mappers seem to be accessing file sequentially,
>> why do you think so?
>>
>> NFS maybe changes something, but mappers shouldn't access file
>> sequentially. NFS could make the file unsplittable, but you need to more
>> test to verify it.
>>
>> The class you want to check out is the
>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>
>> The above code is the key how the split list is generated. If it doesn't
>> performance well for your underline storage system, you can always write
>> your own InputFormat to utilize your own storage system.
>>
>> Yong
>>
>> ------------------------------
>> From: atish.kathpal@gmail.com
>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> To: user@hadoop.apache.org
>>
>>
>> Figured out 1. The output of the reduce was going to the slave node,
>> while I was looking for it in the master node. Which is perfectly fine.
>> Need guidance for 2. though!
>>
>> Thanks
>> Atish
>>
>>
>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hi
>>
>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>> a lot for the advice.
>>
>> *Two issues though*:
>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>> output directory, but no part-r file containing the actual results of
>> the wordcount job. However I am seeing the correct output on running MR
>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>> see no errors on the console.
>> Command used:
>> hadoop jar
>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>
>>
>> 2. I am observing that the mappers seem to be accessing files
>> sequentially, splitting the files across mappers, and then reading data in
>> parallelel, then moving on to the next file. What I want instead is that,
>> files themselves should be accessed in parallel, that is, if there are 10
>> files to be MRed, then MR should ask for each of these files in parallel in
>> one go, and then work on the splits of these files in parallel.
>> *Why do I need this?* Some of the data coming from the NFS mount point
>> is coming from offline media (which takes ~5-10 seconds of time before
>> first bytes are received). So I would like all required files to be asked
>> at the onset itself from the NFS mount point. This way several offline
>> media will be spun up parallely and as the data from these media gets
>> available MR can process them.
>>
>> Would be glad to get inputs on these points!
>>
>> Thanks
>> Atish
>>
>> Tip for those who are trying similar stuff::
>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>> but I was able to rectify this with help from:
>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>
>>
>>
>>
>>
>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>> Yong raises an important issue: You have thrown out the I/O advantages
>> of HDFS and also thrown out the advantages of data locality. It would be
>> interesting to know why you are taking this approach.
>> Chris
>>
>>
>> On 12/20/2013 9:28 AM, java8964 wrote:
>>
>> I believe the "-fs local" should be removed too. The reason is that even
>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>> local", I believe that all the mappers will be run sequentially.
>>
>> "-fs local" will force the mapreducer run in "local" mode, which is
>> really a test mode.
>>
>> What you can do is to remove both "-fs local -jt local", but give the
>> FULL URI of the input and output path, to tell Hadoop that they are local
>> filesystem instead of HDFS.
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///hduser/mount_point file:///results"
>>
>> Keep in mind followings:
>>
>> 1) The NFS mount need to be available in all your Task Nodes, and
>> mounted in the same way.
>> 2) Even you can do that, but your sharing storage will be your
>> bottleneck. NFS won't work well for scalability.
>>
>> Yong
>>
>> ------------------------------
>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> From: dsuiter@rdx.com
>> To: user@hadoop.apache.org
>>
>> I think most of your problem is coming from the options you are setting:
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>
>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>> *-jt* argument and it should run in distributed mode if your cluster is
>> set up right. You don't need to do anything special to point Hadoop towards
>> a NFS location, other than set up the NFS location properly and make sure
>> if you are directing to it by name that it will resolve to the right
>> address. Hadoop doesn't care where it is, as long as it can read from and
>> write to it. The fact that you are telling it to read/write from/to a NFS
>> location that happens to be mounted as a local filesystem object doesn't
>> matter - you could direct it to the local /hduser/ path and set the -fs
>> local option, and it would end up on the NFS mount, because that's where
>> the NFS mount actually exists, or you could direct it to the absolute
>> network location of the folder that you want, it shouldn't make a
>> difference.
>>
>> *Devin Suiter*
>> Jr. Data Solutions Software Engineer
>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>> Google Voice: 412-256-8556 | www.rdx.com
>>
>>
>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hello
>>
>> The picture below describes the deployment architecture I am trying to
>> achieve.
>> However, when I run the wordcount example code with the below
>> configuration, by issuing the command from the master node, I notice only
>> the master node spawning map tasks and completing the submitted job. Below
>> is the command I used:
>>
>> *hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>
>> *Question: How can I leverage both the hadoop nodes for running MR,
>> while serving my data from the common NFS mount point running my filesystem
>> at the backend? Has any one tried such a setup before?*
>> [image: Inline image 1]
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Basically what I really need is that if I am submitting a job (say
wordcount) consisting of 1000s (or even more) of big files, Hadoop should
issue immediate IO for reads on each of these files, such that the
underlying storage system can know that it has to cache those files and
present them back to Hadoop for processing. What I observe through by logs
currently is that, if a particular file is very large, then Hadoop first
operates on all splits of this large file, before asking for the next file.
I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is where
I should make any changes to. Any other suggestions for this will be
welcome too.
On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
> I took logs at the underlying storage system and noticed the sequential
> behavior across files. However, most of the files I am testing with are big
> > 1GB in size, hence leading to several splits. From the logs I notice that
> File2 is not being accessed until all blocks of File1 have been read. It is
> possible that this is just the case as I have just 2 nodes and the files
> being processed are huge, Hence I am observing a sequential behaviour
> across files.
>
> Within the files however, the splits seems to be getting processed in
> parallel as I see random offsets within the file being accessed. That said,
> let me try logging at the NFS level to map IOs coming from different IP
> addresses (nodes) and analyze the pattern.
>
> Thanks for your suggestion on looking at the getSplits() function. Will
> take a look and get back accordingly.
>
>
> Thanks
> Atish
>
>
> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>
>> When you said that the mappers seem to be accessing file sequentially,
>> why do you think so?
>>
>> NFS maybe changes something, but mappers shouldn't access file
>> sequentially. NFS could make the file unsplittable, but you need to more
>> test to verify it.
>>
>> The class you want to check out is the
>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>
>> The above code is the key how the split list is generated. If it doesn't
>> performance well for your underline storage system, you can always write
>> your own InputFormat to utilize your own storage system.
>>
>> Yong
>>
>> ------------------------------
>> From: atish.kathpal@gmail.com
>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> To: user@hadoop.apache.org
>>
>>
>> Figured out 1. The output of the reduce was going to the slave node,
>> while I was looking for it in the master node. Which is perfectly fine.
>> Need guidance for 2. though!
>>
>> Thanks
>> Atish
>>
>>
>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hi
>>
>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>> a lot for the advice.
>>
>> *Two issues though*:
>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>> output directory, but no part-r file containing the actual results of
>> the wordcount job. However I am seeing the correct output on running MR
>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>> see no errors on the console.
>> Command used:
>> hadoop jar
>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>
>>
>> 2. I am observing that the mappers seem to be accessing files
>> sequentially, splitting the files across mappers, and then reading data in
>> parallelel, then moving on to the next file. What I want instead is that,
>> files themselves should be accessed in parallel, that is, if there are 10
>> files to be MRed, then MR should ask for each of these files in parallel in
>> one go, and then work on the splits of these files in parallel.
>> *Why do I need this?* Some of the data coming from the NFS mount point
>> is coming from offline media (which takes ~5-10 seconds of time before
>> first bytes are received). So I would like all required files to be asked
>> at the onset itself from the NFS mount point. This way several offline
>> media will be spun up parallely and as the data from these media gets
>> available MR can process them.
>>
>> Would be glad to get inputs on these points!
>>
>> Thanks
>> Atish
>>
>> Tip for those who are trying similar stuff::
>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>> but I was able to rectify this with help from:
>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>
>>
>>
>>
>>
>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>> Yong raises an important issue: You have thrown out the I/O advantages
>> of HDFS and also thrown out the advantages of data locality. It would be
>> interesting to know why you are taking this approach.
>> Chris
>>
>>
>> On 12/20/2013 9:28 AM, java8964 wrote:
>>
>> I believe the "-fs local" should be removed too. The reason is that even
>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>> local", I believe that all the mappers will be run sequentially.
>>
>> "-fs local" will force the mapreducer run in "local" mode, which is
>> really a test mode.
>>
>> What you can do is to remove both "-fs local -jt local", but give the
>> FULL URI of the input and output path, to tell Hadoop that they are local
>> filesystem instead of HDFS.
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///hduser/mount_point file:///results"
>>
>> Keep in mind followings:
>>
>> 1) The NFS mount need to be available in all your Task Nodes, and
>> mounted in the same way.
>> 2) Even you can do that, but your sharing storage will be your
>> bottleneck. NFS won't work well for scalability.
>>
>> Yong
>>
>> ------------------------------
>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> From: dsuiter@rdx.com
>> To: user@hadoop.apache.org
>>
>> I think most of your problem is coming from the options you are setting:
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>
>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>> *-jt* argument and it should run in distributed mode if your cluster is
>> set up right. You don't need to do anything special to point Hadoop towards
>> a NFS location, other than set up the NFS location properly and make sure
>> if you are directing to it by name that it will resolve to the right
>> address. Hadoop doesn't care where it is, as long as it can read from and
>> write to it. The fact that you are telling it to read/write from/to a NFS
>> location that happens to be mounted as a local filesystem object doesn't
>> matter - you could direct it to the local /hduser/ path and set the -fs
>> local option, and it would end up on the NFS mount, because that's where
>> the NFS mount actually exists, or you could direct it to the absolute
>> network location of the folder that you want, it shouldn't make a
>> difference.
>>
>> *Devin Suiter*
>> Jr. Data Solutions Software Engineer
>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>> Google Voice: 412-256-8556 | www.rdx.com
>>
>>
>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hello
>>
>> The picture below describes the deployment architecture I am trying to
>> achieve.
>> However, when I run the wordcount example code with the below
>> configuration, by issuing the command from the master node, I notice only
>> the master node spawning map tasks and completing the submitted job. Below
>> is the command I used:
>>
>> *hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>
>> *Question: How can I leverage both the hadoop nodes for running MR,
>> while serving my data from the common NFS mount point running my filesystem
>> at the backend? Has any one tried such a setup before?*
>> [image: Inline image 1]
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Basically what I really need is that if I am submitting a job (say
wordcount) consisting of 1000s (or even more) of big files, Hadoop should
issue immediate IO for reads on each of these files, such that the
underlying storage system can know that it has to cache those files and
present them back to Hadoop for processing. What I observe through by logs
currently is that, if a particular file is very large, then Hadoop first
operates on all splits of this large file, before asking for the next file.
I will take a look if *org.apache.hadoop.mapred.**FileInputFormat* is where
I should make any changes to. Any other suggestions for this will be
welcome too.
On Sun, Jan 12, 2014 at 4:06 PM, Atish Kathpal <at...@gmail.com>wrote:
> I took logs at the underlying storage system and noticed the sequential
> behavior across files. However, most of the files I am testing with are big
> > 1GB in size, hence leading to several splits. From the logs I notice that
> File2 is not being accessed until all blocks of File1 have been read. It is
> possible that this is just the case as I have just 2 nodes and the files
> being processed are huge, Hence I am observing a sequential behaviour
> across files.
>
> Within the files however, the splits seems to be getting processed in
> parallel as I see random offsets within the file being accessed. That said,
> let me try logging at the NFS level to map IOs coming from different IP
> addresses (nodes) and analyze the pattern.
>
> Thanks for your suggestion on looking at the getSplits() function. Will
> take a look and get back accordingly.
>
>
> Thanks
> Atish
>
>
> On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
>
>> When you said that the mappers seem to be accessing file sequentially,
>> why do you think so?
>>
>> NFS maybe changes something, but mappers shouldn't access file
>> sequentially. NFS could make the file unsplittable, but you need to more
>> test to verify it.
>>
>> The class you want to check out is the
>> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>>
>> The above code is the key how the split list is generated. If it doesn't
>> performance well for your underline storage system, you can always write
>> your own InputFormat to utilize your own storage system.
>>
>> Yong
>>
>> ------------------------------
>> From: atish.kathpal@gmail.com
>> Date: Wed, 8 Jan 2014 15:48:12 +0530
>>
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> To: user@hadoop.apache.org
>>
>>
>> Figured out 1. The output of the reduce was going to the slave node,
>> while I was looking for it in the master node. Which is perfectly fine.
>> Need guidance for 2. though!
>>
>> Thanks
>> Atish
>>
>>
>> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hi
>>
>> By giving the complete URI, the MR jobs worked across both nodes. Thanks
>> a lot for the advice.
>>
>> *Two issues though*:
>> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
>> output directory, but no part-r file containing the actual results of
>> the wordcount job. However I am seeing the correct output on running MR
>> over HDFS. What is going wrong? Any place I can find logs for the MR job. I
>> see no errors on the console.
>> Command used:
>> hadoop jar
>> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>>
>>
>> 2. I am observing that the mappers seem to be accessing files
>> sequentially, splitting the files across mappers, and then reading data in
>> parallelel, then moving on to the next file. What I want instead is that,
>> files themselves should be accessed in parallel, that is, if there are 10
>> files to be MRed, then MR should ask for each of these files in parallel in
>> one go, and then work on the splits of these files in parallel.
>> *Why do I need this?* Some of the data coming from the NFS mount point
>> is coming from offline media (which takes ~5-10 seconds of time before
>> first bytes are received). So I would like all required files to be asked
>> at the onset itself from the NFS mount point. This way several offline
>> media will be spun up parallely and as the data from these media gets
>> available MR can process them.
>>
>> Would be glad to get inputs on these points!
>>
>> Thanks
>> Atish
>>
>> Tip for those who are trying similar stuff::
>> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
>> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
>> but I was able to rectify this with help from:
>> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>>
>>
>>
>>
>>
>> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>> Yong raises an important issue: You have thrown out the I/O advantages
>> of HDFS and also thrown out the advantages of data locality. It would be
>> interesting to know why you are taking this approach.
>> Chris
>>
>>
>> On 12/20/2013 9:28 AM, java8964 wrote:
>>
>> I believe the "-fs local" should be removed too. The reason is that even
>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>> local", I believe that all the mappers will be run sequentially.
>>
>> "-fs local" will force the mapreducer run in "local" mode, which is
>> really a test mode.
>>
>> What you can do is to remove both "-fs local -jt local", but give the
>> FULL URI of the input and output path, to tell Hadoop that they are local
>> filesystem instead of HDFS.
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount file:///hduser/mount_point file:///results"
>>
>> Keep in mind followings:
>>
>> 1) The NFS mount need to be available in all your Task Nodes, and
>> mounted in the same way.
>> 2) Even you can do that, but your sharing storage will be your
>> bottleneck. NFS won't work well for scalability.
>>
>> Yong
>>
>> ------------------------------
>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>> filesystem
>> From: dsuiter@rdx.com
>> To: user@hadoop.apache.org
>>
>> I think most of your problem is coming from the options you are setting:
>>
>> "hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>
>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>> *-jt* argument and it should run in distributed mode if your cluster is
>> set up right. You don't need to do anything special to point Hadoop towards
>> a NFS location, other than set up the NFS location properly and make sure
>> if you are directing to it by name that it will resolve to the right
>> address. Hadoop doesn't care where it is, as long as it can read from and
>> write to it. The fact that you are telling it to read/write from/to a NFS
>> location that happens to be mounted as a local filesystem object doesn't
>> matter - you could direct it to the local /hduser/ path and set the -fs
>> local option, and it would end up on the NFS mount, because that's where
>> the NFS mount actually exists, or you could direct it to the absolute
>> network location of the folder that you want, it shouldn't make a
>> difference.
>>
>> *Devin Suiter*
>> Jr. Data Solutions Software Engineer
>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>> Google Voice: 412-256-8556 | www.rdx.com
>>
>>
>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>
>> Hello
>>
>> The picture below describes the deployment architecture I am trying to
>> achieve.
>> However, when I run the wordcount example code with the below
>> configuration, by issuing the command from the master node, I notice only
>> the master node spawning map tasks and completing the submitted job. Below
>> is the command I used:
>>
>> *hadoop jar
>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>
>> *Question: How can I leverage both the hadoop nodes for running MR,
>> while serving my data from the common NFS mount point running my filesystem
>> at the backend? Has any one tried such a setup before?*
>> [image: Inline image 1]
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
I took logs at the underlying storage system and noticed the sequential
behavior across files. However, most of the files I am testing with are big
> 1GB in size, hence leading to several splits. From the logs I notice that
File2 is not being accessed until all blocks of File1 have been read. It is
possible that this is just the case as I have just 2 nodes and the files
being processed are huge, Hence I am observing a sequential behaviour
across files.
Within the files however, the splits seems to be getting processed in
parallel as I see random offsets within the file being accessed. That said,
let me try logging at the NFS level to map IOs coming from different IP
addresses (nodes) and analyze the pattern.
Thanks for your suggestion on looking at the getSplits() function. Will
take a look and get back accordingly.
Thanks
Atish
On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
> When you said that the mappers seem to be accessing file sequentially, why
> do you think so?
>
> NFS maybe changes something, but mappers shouldn't access file
> sequentially. NFS could make the file unsplittable, but you need to more
> test to verify it.
>
> The class you want to check out is the
> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>
> The above code is the key how the split list is generated. If it doesn't
> performance well for your underline storage system, you can always write
> your own InputFormat to utilize your own storage system.
>
> Yong
>
> ------------------------------
> From: atish.kathpal@gmail.com
> Date: Wed, 8 Jan 2014 15:48:12 +0530
>
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> To: user@hadoop.apache.org
>
>
> Figured out 1. The output of the reduce was going to the slave node, while
> I was looking for it in the master node. Which is perfectly fine.
> Need guidance for 2. though!
>
> Thanks
> Atish
>
>
> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
> test the suggestions made by Yong and Devin and get back to you guys.
>
> As on the bottlenecking issue, I agree, but I am trying to run few MR
> jobs on a traditional NAS server. I can live with a few bottlenecks, so
> long as I don't have to move the data to a dedicated HDFS cluster.
>
>
> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>
> Yong raises an important issue: You have thrown out the I/O advantages
> of HDFS and also thrown out the advantages of data locality. It would be
> interesting to know why you are taking this approach.
> Chris
>
>
> On 12/20/2013 9:28 AM, java8964 wrote:
>
> I believe the "-fs local" should be removed too. The reason is that even
> you have a dedicated JobTracker after removing "-jt local", but with "-fs
> local", I believe that all the mappers will be run sequentially.
>
> "-fs local" will force the mapreducer run in "local" mode, which is
> really a test mode.
>
> What you can do is to remove both "-fs local -jt local", but give the
> FULL URI of the input and output path, to tell Hadoop that they are local
> filesystem instead of HDFS.
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///hduser/mount_point file:///results"
>
> Keep in mind followings:
>
> 1) The NFS mount need to be available in all your Task Nodes, and
> mounted in the same way.
> 2) Even you can do that, but your sharing storage will be your bottleneck.
> NFS won't work well for scalability.
>
> Yong
>
> ------------------------------
> Date: Fri, 20 Dec 2013 09:01:32 -0500
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> From: dsuiter@rdx.com
> To: user@hadoop.apache.org
>
> I think most of your problem is coming from the options you are setting:
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>
> You appear to be directing your namenode to run jobs in the *LOCAL* job
> runner and directing it to read from the *LOCAL* filesystem. Drop the
> *-jt* argument and it should run in distributed mode if your cluster is
> set up right. You don't need to do anything special to point Hadoop towards
> a NFS location, other than set up the NFS location properly and make sure
> if you are directing to it by name that it will resolve to the right
> address. Hadoop doesn't care where it is, as long as it can read from and
> write to it. The fact that you are telling it to read/write from/to a NFS
> location that happens to be mounted as a local filesystem object doesn't
> matter - you could direct it to the local /hduser/ path and set the -fs
> local option, and it would end up on the NFS mount, because that's where
> the NFS mount actually exists, or you could direct it to the absolute
> network location of the folder that you want, it shouldn't make a
> difference.
>
> *Devin Suiter*
> Jr. Data Solutions Software Engineer
> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
> Google Voice: 412-256-8556 | www.rdx.com
>
>
> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hello
>
> The picture below describes the deployment architecture I am trying to
> achieve.
> However, when I run the wordcount example code with the below
> configuration, by issuing the command from the master node, I notice only
> the master node spawning map tasks and completing the submitted job. Below
> is the command I used:
>
> *hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount -fs local -jt local /hduser/mount_point/ /results*
>
> *Question: How can I leverage both the hadoop nodes for running MR,
> while serving my data from the common NFS mount point running my filesystem
> at the backend? Has any one tried such a setup before?*
> [image: Inline image 1]
>
> Thanks!
>
>
>
>
>
>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
I took logs at the underlying storage system and noticed the sequential
behavior across files. However, most of the files I am testing with are big
> 1GB in size, hence leading to several splits. From the logs I notice that
File2 is not being accessed until all blocks of File1 have been read. It is
possible that this is just the case as I have just 2 nodes and the files
being processed are huge, Hence I am observing a sequential behaviour
across files.
Within the files however, the splits seems to be getting processed in
parallel as I see random offsets within the file being accessed. That said,
let me try logging at the NFS level to map IOs coming from different IP
addresses (nodes) and analyze the pattern.
Thanks for your suggestion on looking at the getSplits() function. Will
take a look and get back accordingly.
Thanks
Atish
On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
> When you said that the mappers seem to be accessing file sequentially, why
> do you think so?
>
> NFS maybe changes something, but mappers shouldn't access file
> sequentially. NFS could make the file unsplittable, but you need to more
> test to verify it.
>
> The class you want to check out is the
> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>
> The above code is the key how the split list is generated. If it doesn't
> performance well for your underline storage system, you can always write
> your own InputFormat to utilize your own storage system.
>
> Yong
>
> ------------------------------
> From: atish.kathpal@gmail.com
> Date: Wed, 8 Jan 2014 15:48:12 +0530
>
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> To: user@hadoop.apache.org
>
>
> Figured out 1. The output of the reduce was going to the slave node, while
> I was looking for it in the master node. Which is perfectly fine.
> Need guidance for 2. though!
>
> Thanks
> Atish
>
>
> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
> test the suggestions made by Yong and Devin and get back to you guys.
>
> As on the bottlenecking issue, I agree, but I am trying to run few MR
> jobs on a traditional NAS server. I can live with a few bottlenecks, so
> long as I don't have to move the data to a dedicated HDFS cluster.
>
>
> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>
> Yong raises an important issue: You have thrown out the I/O advantages
> of HDFS and also thrown out the advantages of data locality. It would be
> interesting to know why you are taking this approach.
> Chris
>
>
> On 12/20/2013 9:28 AM, java8964 wrote:
>
> I believe the "-fs local" should be removed too. The reason is that even
> you have a dedicated JobTracker after removing "-jt local", but with "-fs
> local", I believe that all the mappers will be run sequentially.
>
> "-fs local" will force the mapreducer run in "local" mode, which is
> really a test mode.
>
> What you can do is to remove both "-fs local -jt local", but give the
> FULL URI of the input and output path, to tell Hadoop that they are local
> filesystem instead of HDFS.
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///hduser/mount_point file:///results"
>
> Keep in mind followings:
>
> 1) The NFS mount need to be available in all your Task Nodes, and
> mounted in the same way.
> 2) Even you can do that, but your sharing storage will be your bottleneck.
> NFS won't work well for scalability.
>
> Yong
>
> ------------------------------
> Date: Fri, 20 Dec 2013 09:01:32 -0500
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> From: dsuiter@rdx.com
> To: user@hadoop.apache.org
>
> I think most of your problem is coming from the options you are setting:
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>
> You appear to be directing your namenode to run jobs in the *LOCAL* job
> runner and directing it to read from the *LOCAL* filesystem. Drop the
> *-jt* argument and it should run in distributed mode if your cluster is
> set up right. You don't need to do anything special to point Hadoop towards
> a NFS location, other than set up the NFS location properly and make sure
> if you are directing to it by name that it will resolve to the right
> address. Hadoop doesn't care where it is, as long as it can read from and
> write to it. The fact that you are telling it to read/write from/to a NFS
> location that happens to be mounted as a local filesystem object doesn't
> matter - you could direct it to the local /hduser/ path and set the -fs
> local option, and it would end up on the NFS mount, because that's where
> the NFS mount actually exists, or you could direct it to the absolute
> network location of the folder that you want, it shouldn't make a
> difference.
>
> *Devin Suiter*
> Jr. Data Solutions Software Engineer
> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
> Google Voice: 412-256-8556 | www.rdx.com
>
>
> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hello
>
> The picture below describes the deployment architecture I am trying to
> achieve.
> However, when I run the wordcount example code with the below
> configuration, by issuing the command from the master node, I notice only
> the master node spawning map tasks and completing the submitted job. Below
> is the command I used:
>
> *hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount -fs local -jt local /hduser/mount_point/ /results*
>
> *Question: How can I leverage both the hadoop nodes for running MR,
> while serving my data from the common NFS mount point running my filesystem
> at the backend? Has any one tried such a setup before?*
> [image: Inline image 1]
>
> Thanks!
>
>
>
>
>
>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
I took logs at the underlying storage system and noticed the sequential
behavior across files. However, most of the files I am testing with are big
> 1GB in size, hence leading to several splits. From the logs I notice that
File2 is not being accessed until all blocks of File1 have been read. It is
possible that this is just the case as I have just 2 nodes and the files
being processed are huge, Hence I am observing a sequential behaviour
across files.
Within the files however, the splits seems to be getting processed in
parallel as I see random offsets within the file being accessed. That said,
let me try logging at the NFS level to map IOs coming from different IP
addresses (nodes) and analyze the pattern.
Thanks for your suggestion on looking at the getSplits() function. Will
take a look and get back accordingly.
Thanks
Atish
On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
> When you said that the mappers seem to be accessing file sequentially, why
> do you think so?
>
> NFS maybe changes something, but mappers shouldn't access file
> sequentially. NFS could make the file unsplittable, but you need to more
> test to verify it.
>
> The class you want to check out is the
> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>
> The above code is the key how the split list is generated. If it doesn't
> performance well for your underline storage system, you can always write
> your own InputFormat to utilize your own storage system.
>
> Yong
>
> ------------------------------
> From: atish.kathpal@gmail.com
> Date: Wed, 8 Jan 2014 15:48:12 +0530
>
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> To: user@hadoop.apache.org
>
>
> Figured out 1. The output of the reduce was going to the slave node, while
> I was looking for it in the master node. Which is perfectly fine.
> Need guidance for 2. though!
>
> Thanks
> Atish
>
>
> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
> test the suggestions made by Yong and Devin and get back to you guys.
>
> As on the bottlenecking issue, I agree, but I am trying to run few MR
> jobs on a traditional NAS server. I can live with a few bottlenecks, so
> long as I don't have to move the data to a dedicated HDFS cluster.
>
>
> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>
> Yong raises an important issue: You have thrown out the I/O advantages
> of HDFS and also thrown out the advantages of data locality. It would be
> interesting to know why you are taking this approach.
> Chris
>
>
> On 12/20/2013 9:28 AM, java8964 wrote:
>
> I believe the "-fs local" should be removed too. The reason is that even
> you have a dedicated JobTracker after removing "-jt local", but with "-fs
> local", I believe that all the mappers will be run sequentially.
>
> "-fs local" will force the mapreducer run in "local" mode, which is
> really a test mode.
>
> What you can do is to remove both "-fs local -jt local", but give the
> FULL URI of the input and output path, to tell Hadoop that they are local
> filesystem instead of HDFS.
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///hduser/mount_point file:///results"
>
> Keep in mind followings:
>
> 1) The NFS mount need to be available in all your Task Nodes, and
> mounted in the same way.
> 2) Even you can do that, but your sharing storage will be your bottleneck.
> NFS won't work well for scalability.
>
> Yong
>
> ------------------------------
> Date: Fri, 20 Dec 2013 09:01:32 -0500
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> From: dsuiter@rdx.com
> To: user@hadoop.apache.org
>
> I think most of your problem is coming from the options you are setting:
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>
> You appear to be directing your namenode to run jobs in the *LOCAL* job
> runner and directing it to read from the *LOCAL* filesystem. Drop the
> *-jt* argument and it should run in distributed mode if your cluster is
> set up right. You don't need to do anything special to point Hadoop towards
> a NFS location, other than set up the NFS location properly and make sure
> if you are directing to it by name that it will resolve to the right
> address. Hadoop doesn't care where it is, as long as it can read from and
> write to it. The fact that you are telling it to read/write from/to a NFS
> location that happens to be mounted as a local filesystem object doesn't
> matter - you could direct it to the local /hduser/ path and set the -fs
> local option, and it would end up on the NFS mount, because that's where
> the NFS mount actually exists, or you could direct it to the absolute
> network location of the folder that you want, it shouldn't make a
> difference.
>
> *Devin Suiter*
> Jr. Data Solutions Software Engineer
> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
> Google Voice: 412-256-8556 | www.rdx.com
>
>
> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hello
>
> The picture below describes the deployment architecture I am trying to
> achieve.
> However, when I run the wordcount example code with the below
> configuration, by issuing the command from the master node, I notice only
> the master node spawning map tasks and completing the submitted job. Below
> is the command I used:
>
> *hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount -fs local -jt local /hduser/mount_point/ /results*
>
> *Question: How can I leverage both the hadoop nodes for running MR,
> while serving my data from the common NFS mount point running my filesystem
> at the backend? Has any one tried such a setup before?*
> [image: Inline image 1]
>
> Thanks!
>
>
>
>
>
>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
I took logs at the underlying storage system and noticed the sequential
behavior across files. However, most of the files I am testing with are big
> 1GB in size, hence leading to several splits. From the logs I notice that
File2 is not being accessed until all blocks of File1 have been read. It is
possible that this is just the case as I have just 2 nodes and the files
being processed are huge, Hence I am observing a sequential behaviour
across files.
Within the files however, the splits seems to be getting processed in
parallel as I see random offsets within the file being accessed. That said,
let me try logging at the NFS level to map IOs coming from different IP
addresses (nodes) and analyze the pattern.
Thanks for your suggestion on looking at the getSplits() function. Will
take a look and get back accordingly.
Thanks
Atish
On Fri, Jan 10, 2014 at 9:12 PM, java8964 <ja...@hotmail.com> wrote:
> When you said that the mappers seem to be accessing file sequentially, why
> do you think so?
>
> NFS maybe changes something, but mappers shouldn't access file
> sequentially. NFS could make the file unsplittable, but you need to more
> test to verify it.
>
> The class you want to check out is the
> org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
>
> The above code is the key how the split list is generated. If it doesn't
> performance well for your underline storage system, you can always write
> your own InputFormat to utilize your own storage system.
>
> Yong
>
> ------------------------------
> From: atish.kathpal@gmail.com
> Date: Wed, 8 Jan 2014 15:48:12 +0530
>
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> To: user@hadoop.apache.org
>
>
> Figured out 1. The output of the reduce was going to the slave node, while
> I was looking for it in the master node. Which is perfectly fine.
> Need guidance for 2. though!
>
> Thanks
> Atish
>
>
> On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
> test the suggestions made by Yong and Devin and get back to you guys.
>
> As on the bottlenecking issue, I agree, but I am trying to run few MR
> jobs on a traditional NAS server. I can live with a few bottlenecks, so
> long as I don't have to move the data to a dedicated HDFS cluster.
>
>
> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>
> Yong raises an important issue: You have thrown out the I/O advantages
> of HDFS and also thrown out the advantages of data locality. It would be
> interesting to know why you are taking this approach.
> Chris
>
>
> On 12/20/2013 9:28 AM, java8964 wrote:
>
> I believe the "-fs local" should be removed too. The reason is that even
> you have a dedicated JobTracker after removing "-jt local", but with "-fs
> local", I believe that all the mappers will be run sequentially.
>
> "-fs local" will force the mapreducer run in "local" mode, which is
> really a test mode.
>
> What you can do is to remove both "-fs local -jt local", but give the
> FULL URI of the input and output path, to tell Hadoop that they are local
> filesystem instead of HDFS.
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///hduser/mount_point file:///results"
>
> Keep in mind followings:
>
> 1) The NFS mount need to be available in all your Task Nodes, and
> mounted in the same way.
> 2) Even you can do that, but your sharing storage will be your bottleneck.
> NFS won't work well for scalability.
>
> Yong
>
> ------------------------------
> Date: Fri, 20 Dec 2013 09:01:32 -0500
> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
> filesystem
> From: dsuiter@rdx.com
> To: user@hadoop.apache.org
>
> I think most of your problem is coming from the options you are setting:
>
> "hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>
> You appear to be directing your namenode to run jobs in the *LOCAL* job
> runner and directing it to read from the *LOCAL* filesystem. Drop the
> *-jt* argument and it should run in distributed mode if your cluster is
> set up right. You don't need to do anything special to point Hadoop towards
> a NFS location, other than set up the NFS location properly and make sure
> if you are directing to it by name that it will resolve to the right
> address. Hadoop doesn't care where it is, as long as it can read from and
> write to it. The fact that you are telling it to read/write from/to a NFS
> location that happens to be mounted as a local filesystem object doesn't
> matter - you could direct it to the local /hduser/ path and set the -fs
> local option, and it would end up on the NFS mount, because that's where
> the NFS mount actually exists, or you could direct it to the absolute
> network location of the folder that you want, it shouldn't make a
> difference.
>
> *Devin Suiter*
> Jr. Data Solutions Software Engineer
> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
> Google Voice: 412-256-8556 | www.rdx.com
>
>
> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>
> Hello
>
> The picture below describes the deployment architecture I am trying to
> achieve.
> However, when I run the wordcount example code with the below
> configuration, by issuing the command from the master node, I notice only
> the master node spawning map tasks and completing the submitted job. Below
> is the command I used:
>
> *hadoop jar
> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount -fs local -jt local /hduser/mount_point/ /results*
>
> *Question: How can I leverage both the hadoop nodes for running MR,
> while serving my data from the common NFS mount point running my filesystem
> at the backend? Has any one tried such a setup before?*
> [image: Inline image 1]
>
> Thanks!
>
>
>
>
>
>
>
RE: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by java8964 <ja...@hotmail.com>.
When you said that the mappers seem to be accessing file sequentially, why do you think so?
NFS maybe changes something, but mappers shouldn't access file sequentially. NFS could make the file unsplittable, but you need to more test to verify it.
The class you want to check out is the org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
The above code is the key how the split list is generated. If it doesn't performance well for your underline storage system, you can always write your own InputFormat to utilize your own storage system.
Yong
From: atish.kathpal@gmail.com
Date: Wed, 8 Jan 2014 15:48:12 +0530
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
To: user@hadoop.apache.org
Figured out 1. The output of the reduce was going to the slave node, while I was looking for it in the master node. Which is perfectly fine. Need guidance for 2. though!
ThanksAtish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com> wrote:
Hi
By giving the complete URI, the MR jobs worked across both nodes. Thanks a lot for the advice.
Two issues though:1. On completion of the MR job, I see only the "_SUCCESS" file in the output directory, but no part-r file containing the actual results of the wordcount job. However I am seeing the correct output on running MR over HDFS. What is going wrong? Any place I can find logs for the MR job. I see no errors on the console.
Command used: hadoop jar /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
2. I am observing that the mappers seem to be accessing files sequentially, splitting the files across mappers, and then reading data in parallelel, then moving on to the next file. What I want instead is that, files themselves should be accessed in parallel, that is, if there are 10 files to be MRed, then MR should ask for each of these files in parallel in one go, and then work on the splits of these files in parallel.
Why do I need this? Some of the data coming from the NFS mount point is coming from offline media (which takes ~5-10 seconds of time before first bytes are received). So I would like all required files to be asked at the onset itself from the NFS mount point. This way several offline media will be spun up parallely and as the data from these media gets available MR can process them.
Would be glad to get inputs on these points!
ThanksAtish
Tip for those who are trying similar stuff::In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError: Java heap space", but I was able to rectify this with help from: http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com> wrote:
Thanks Devin, Yong, and Chris for your replies and suggestions. I will test the suggestions made by Yong and Devin and get back to you guys.
As on the bottlenecking issue, I agree, but I am trying to run few MR jobs on a traditional NAS server. I can live with a few bottlenecks, so long as I don't have to move the data to a dedicated HDFS cluster.
On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com> wrote:
Yong raises an important issue: You
have thrown out the I/O advantages of HDFS and also thrown out the
advantages of data locality. It would be interesting to know why
you are taking this approach.
Chris
On 12/20/2013 9:28 AM, java8964 wrote:
I believe the "-fs local" should be removed too.
The reason is that even you have a dedicated JobTracker after
removing "-jt local", but with "-fs local", I believe that all
the mappers will be run sequentially.
"-fs local" will force the mapreducer run in "local" mode,
which is really a test mode.
What you can do is to remove both "-fs local -jt local",
but give the FULL URI of the input and output path, to tell
Hadoop that they are local filesystem instead of HDFS.
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount file:///hduser/mount_point
file:///results"
Keep in mind followings:
1) The NFS mount need to be available in all your Task
Nodes, and mounted in the same way.
2) Even you can do that, but your sharing storage will be
your bottleneck. NFS won't work well for scalability.
Yong
Date: Fri, 20 Dec 2013 09:01:32 -0500
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS
mounted filesystem
From: dsuiter@rdx.com
To: user@hadoop.apache.org
I think most of your problem is coming from
the options you are setting:
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results"
You appear to be directing your namenode to run jobs
in the LOCAL job runner and directing it to read
from the LOCAL filesystem. Drop the -jt
argument and it should run in distributed mode if your
cluster is set up right. You don't need to do anything
special to point Hadoop towards a NFS location, other
than set up the NFS location properly and make sure if
you are directing to it by name that it will resolve to
the right address. Hadoop doesn't care where it is, as
long as it can read from and write to it. The fact that
you are telling it to read/write from/to a NFS location
that happens to be mounted as a local filesystem object
doesn't matter - you could direct it to the local
/hduser/ path and set the -fs local option, and it would
end up on the NFS mount, because that's where the NFS
mount actually exists, or you could direct it to the
absolute network location of the folder that you want,
it shouldn't make a difference.
Devin Suiter
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor |
Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com
On Fri, Dec 20, 2013 at 5:27
AM, Atish Kathpal <at...@gmail.com>
wrote:
Hello
The picture below describes the deployment
architecture I am trying to achieve.
However, when I run the wordcount example code
with the below configuration, by issuing the
command from the master node, I notice only the
master node spawning map tasks and completing the
submitted job. Below is the command I used:
hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results
Question: How can I leverage both the hadoop
nodes for running MR, while serving my data from
the common NFS mount point running my filesystem
at the backend? Has any one tried such a setup
before?
Thanks!
RE: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by java8964 <ja...@hotmail.com>.
When you said that the mappers seem to be accessing file sequentially, why do you think so?
NFS maybe changes something, but mappers shouldn't access file sequentially. NFS could make the file unsplittable, but you need to more test to verify it.
The class you want to check out is the org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
The above code is the key how the split list is generated. If it doesn't performance well for your underline storage system, you can always write your own InputFormat to utilize your own storage system.
Yong
From: atish.kathpal@gmail.com
Date: Wed, 8 Jan 2014 15:48:12 +0530
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
To: user@hadoop.apache.org
Figured out 1. The output of the reduce was going to the slave node, while I was looking for it in the master node. Which is perfectly fine. Need guidance for 2. though!
ThanksAtish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com> wrote:
Hi
By giving the complete URI, the MR jobs worked across both nodes. Thanks a lot for the advice.
Two issues though:1. On completion of the MR job, I see only the "_SUCCESS" file in the output directory, but no part-r file containing the actual results of the wordcount job. However I am seeing the correct output on running MR over HDFS. What is going wrong? Any place I can find logs for the MR job. I see no errors on the console.
Command used: hadoop jar /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
2. I am observing that the mappers seem to be accessing files sequentially, splitting the files across mappers, and then reading data in parallelel, then moving on to the next file. What I want instead is that, files themselves should be accessed in parallel, that is, if there are 10 files to be MRed, then MR should ask for each of these files in parallel in one go, and then work on the splits of these files in parallel.
Why do I need this? Some of the data coming from the NFS mount point is coming from offline media (which takes ~5-10 seconds of time before first bytes are received). So I would like all required files to be asked at the onset itself from the NFS mount point. This way several offline media will be spun up parallely and as the data from these media gets available MR can process them.
Would be glad to get inputs on these points!
ThanksAtish
Tip for those who are trying similar stuff::In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError: Java heap space", but I was able to rectify this with help from: http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com> wrote:
Thanks Devin, Yong, and Chris for your replies and suggestions. I will test the suggestions made by Yong and Devin and get back to you guys.
As on the bottlenecking issue, I agree, but I am trying to run few MR jobs on a traditional NAS server. I can live with a few bottlenecks, so long as I don't have to move the data to a dedicated HDFS cluster.
On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com> wrote:
Yong raises an important issue: You
have thrown out the I/O advantages of HDFS and also thrown out the
advantages of data locality. It would be interesting to know why
you are taking this approach.
Chris
On 12/20/2013 9:28 AM, java8964 wrote:
I believe the "-fs local" should be removed too.
The reason is that even you have a dedicated JobTracker after
removing "-jt local", but with "-fs local", I believe that all
the mappers will be run sequentially.
"-fs local" will force the mapreducer run in "local" mode,
which is really a test mode.
What you can do is to remove both "-fs local -jt local",
but give the FULL URI of the input and output path, to tell
Hadoop that they are local filesystem instead of HDFS.
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount file:///hduser/mount_point
file:///results"
Keep in mind followings:
1) The NFS mount need to be available in all your Task
Nodes, and mounted in the same way.
2) Even you can do that, but your sharing storage will be
your bottleneck. NFS won't work well for scalability.
Yong
Date: Fri, 20 Dec 2013 09:01:32 -0500
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS
mounted filesystem
From: dsuiter@rdx.com
To: user@hadoop.apache.org
I think most of your problem is coming from
the options you are setting:
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results"
You appear to be directing your namenode to run jobs
in the LOCAL job runner and directing it to read
from the LOCAL filesystem. Drop the -jt
argument and it should run in distributed mode if your
cluster is set up right. You don't need to do anything
special to point Hadoop towards a NFS location, other
than set up the NFS location properly and make sure if
you are directing to it by name that it will resolve to
the right address. Hadoop doesn't care where it is, as
long as it can read from and write to it. The fact that
you are telling it to read/write from/to a NFS location
that happens to be mounted as a local filesystem object
doesn't matter - you could direct it to the local
/hduser/ path and set the -fs local option, and it would
end up on the NFS mount, because that's where the NFS
mount actually exists, or you could direct it to the
absolute network location of the folder that you want,
it shouldn't make a difference.
Devin Suiter
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor |
Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com
On Fri, Dec 20, 2013 at 5:27
AM, Atish Kathpal <at...@gmail.com>
wrote:
Hello
The picture below describes the deployment
architecture I am trying to achieve.
However, when I run the wordcount example code
with the below configuration, by issuing the
command from the master node, I notice only the
master node spawning map tasks and completing the
submitted job. Below is the command I used:
hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results
Question: How can I leverage both the hadoop
nodes for running MR, while serving my data from
the common NFS mount point running my filesystem
at the backend? Has any one tried such a setup
before?
Thanks!
RE: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by java8964 <ja...@hotmail.com>.
When you said that the mappers seem to be accessing file sequentially, why do you think so?
NFS maybe changes something, but mappers shouldn't access file sequentially. NFS could make the file unsplittable, but you need to more test to verify it.
The class you want to check out is the org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
The above code is the key how the split list is generated. If it doesn't performance well for your underline storage system, you can always write your own InputFormat to utilize your own storage system.
Yong
From: atish.kathpal@gmail.com
Date: Wed, 8 Jan 2014 15:48:12 +0530
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
To: user@hadoop.apache.org
Figured out 1. The output of the reduce was going to the slave node, while I was looking for it in the master node. Which is perfectly fine. Need guidance for 2. though!
ThanksAtish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com> wrote:
Hi
By giving the complete URI, the MR jobs worked across both nodes. Thanks a lot for the advice.
Two issues though:1. On completion of the MR job, I see only the "_SUCCESS" file in the output directory, but no part-r file containing the actual results of the wordcount job. However I am seeing the correct output on running MR over HDFS. What is going wrong? Any place I can find logs for the MR job. I see no errors on the console.
Command used: hadoop jar /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
2. I am observing that the mappers seem to be accessing files sequentially, splitting the files across mappers, and then reading data in parallelel, then moving on to the next file. What I want instead is that, files themselves should be accessed in parallel, that is, if there are 10 files to be MRed, then MR should ask for each of these files in parallel in one go, and then work on the splits of these files in parallel.
Why do I need this? Some of the data coming from the NFS mount point is coming from offline media (which takes ~5-10 seconds of time before first bytes are received). So I would like all required files to be asked at the onset itself from the NFS mount point. This way several offline media will be spun up parallely and as the data from these media gets available MR can process them.
Would be glad to get inputs on these points!
ThanksAtish
Tip for those who are trying similar stuff::In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError: Java heap space", but I was able to rectify this with help from: http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com> wrote:
Thanks Devin, Yong, and Chris for your replies and suggestions. I will test the suggestions made by Yong and Devin and get back to you guys.
As on the bottlenecking issue, I agree, but I am trying to run few MR jobs on a traditional NAS server. I can live with a few bottlenecks, so long as I don't have to move the data to a dedicated HDFS cluster.
On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com> wrote:
Yong raises an important issue: You
have thrown out the I/O advantages of HDFS and also thrown out the
advantages of data locality. It would be interesting to know why
you are taking this approach.
Chris
On 12/20/2013 9:28 AM, java8964 wrote:
I believe the "-fs local" should be removed too.
The reason is that even you have a dedicated JobTracker after
removing "-jt local", but with "-fs local", I believe that all
the mappers will be run sequentially.
"-fs local" will force the mapreducer run in "local" mode,
which is really a test mode.
What you can do is to remove both "-fs local -jt local",
but give the FULL URI of the input and output path, to tell
Hadoop that they are local filesystem instead of HDFS.
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount file:///hduser/mount_point
file:///results"
Keep in mind followings:
1) The NFS mount need to be available in all your Task
Nodes, and mounted in the same way.
2) Even you can do that, but your sharing storage will be
your bottleneck. NFS won't work well for scalability.
Yong
Date: Fri, 20 Dec 2013 09:01:32 -0500
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS
mounted filesystem
From: dsuiter@rdx.com
To: user@hadoop.apache.org
I think most of your problem is coming from
the options you are setting:
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results"
You appear to be directing your namenode to run jobs
in the LOCAL job runner and directing it to read
from the LOCAL filesystem. Drop the -jt
argument and it should run in distributed mode if your
cluster is set up right. You don't need to do anything
special to point Hadoop towards a NFS location, other
than set up the NFS location properly and make sure if
you are directing to it by name that it will resolve to
the right address. Hadoop doesn't care where it is, as
long as it can read from and write to it. The fact that
you are telling it to read/write from/to a NFS location
that happens to be mounted as a local filesystem object
doesn't matter - you could direct it to the local
/hduser/ path and set the -fs local option, and it would
end up on the NFS mount, because that's where the NFS
mount actually exists, or you could direct it to the
absolute network location of the folder that you want,
it shouldn't make a difference.
Devin Suiter
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor |
Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com
On Fri, Dec 20, 2013 at 5:27
AM, Atish Kathpal <at...@gmail.com>
wrote:
Hello
The picture below describes the deployment
architecture I am trying to achieve.
However, when I run the wordcount example code
with the below configuration, by issuing the
command from the master node, I notice only the
master node spawning map tasks and completing the
submitted job. Below is the command I used:
hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results
Question: How can I leverage both the hadoop
nodes for running MR, while serving my data from
the common NFS mount point running my filesystem
at the backend? Has any one tried such a setup
before?
Thanks!
RE: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by java8964 <ja...@hotmail.com>.
When you said that the mappers seem to be accessing file sequentially, why do you think so?
NFS maybe changes something, but mappers shouldn't access file sequentially. NFS could make the file unsplittable, but you need to more test to verify it.
The class you want to check out is the org.apache.hadoop.mapred.FileInputFormat, especially method getSplits().
The above code is the key how the split list is generated. If it doesn't performance well for your underline storage system, you can always write your own InputFormat to utilize your own storage system.
Yong
From: atish.kathpal@gmail.com
Date: Wed, 8 Jan 2014 15:48:12 +0530
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
To: user@hadoop.apache.org
Figured out 1. The output of the reduce was going to the slave node, while I was looking for it in the master node. Which is perfectly fine. Need guidance for 2. though!
ThanksAtish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com> wrote:
Hi
By giving the complete URI, the MR jobs worked across both nodes. Thanks a lot for the advice.
Two issues though:1. On completion of the MR job, I see only the "_SUCCESS" file in the output directory, but no part-r file containing the actual results of the wordcount job. However I am seeing the correct output on running MR over HDFS. What is going wrong? Any place I can find logs for the MR job. I see no errors on the console.
Command used: hadoop jar /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
2. I am observing that the mappers seem to be accessing files sequentially, splitting the files across mappers, and then reading data in parallelel, then moving on to the next file. What I want instead is that, files themselves should be accessed in parallel, that is, if there are 10 files to be MRed, then MR should ask for each of these files in parallel in one go, and then work on the splits of these files in parallel.
Why do I need this? Some of the data coming from the NFS mount point is coming from offline media (which takes ~5-10 seconds of time before first bytes are received). So I would like all required files to be asked at the onset itself from the NFS mount point. This way several offline media will be spun up parallely and as the data from these media gets available MR can process them.
Would be glad to get inputs on these points!
ThanksAtish
Tip for those who are trying similar stuff::In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError: Java heap space", but I was able to rectify this with help from: http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com> wrote:
Thanks Devin, Yong, and Chris for your replies and suggestions. I will test the suggestions made by Yong and Devin and get back to you guys.
As on the bottlenecking issue, I agree, but I am trying to run few MR jobs on a traditional NAS server. I can live with a few bottlenecks, so long as I don't have to move the data to a dedicated HDFS cluster.
On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com> wrote:
Yong raises an important issue: You
have thrown out the I/O advantages of HDFS and also thrown out the
advantages of data locality. It would be interesting to know why
you are taking this approach.
Chris
On 12/20/2013 9:28 AM, java8964 wrote:
I believe the "-fs local" should be removed too.
The reason is that even you have a dedicated JobTracker after
removing "-jt local", but with "-fs local", I believe that all
the mappers will be run sequentially.
"-fs local" will force the mapreducer run in "local" mode,
which is really a test mode.
What you can do is to remove both "-fs local -jt local",
but give the FULL URI of the input and output path, to tell
Hadoop that they are local filesystem instead of HDFS.
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount file:///hduser/mount_point
file:///results"
Keep in mind followings:
1) The NFS mount need to be available in all your Task
Nodes, and mounted in the same way.
2) Even you can do that, but your sharing storage will be
your bottleneck. NFS won't work well for scalability.
Yong
Date: Fri, 20 Dec 2013 09:01:32 -0500
Subject: Re: Running Hadoop v2 clustered mode MR on an NFS
mounted filesystem
From: dsuiter@rdx.com
To: user@hadoop.apache.org
I think most of your problem is coming from
the options you are setting:
"hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results"
You appear to be directing your namenode to run jobs
in the LOCAL job runner and directing it to read
from the LOCAL filesystem. Drop the -jt
argument and it should run in distributed mode if your
cluster is set up right. You don't need to do anything
special to point Hadoop towards a NFS location, other
than set up the NFS location properly and make sure if
you are directing to it by name that it will resolve to
the right address. Hadoop doesn't care where it is, as
long as it can read from and write to it. The fact that
you are telling it to read/write from/to a NFS location
that happens to be mounted as a local filesystem object
doesn't matter - you could direct it to the local
/hduser/ path and set the -fs local option, and it would
end up on the NFS mount, because that's where the NFS
mount actually exists, or you could direct it to the
absolute network location of the folder that you want,
it shouldn't make a difference.
Devin Suiter
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor |
Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com
On Fri, Dec 20, 2013 at 5:27
AM, Atish Kathpal <at...@gmail.com>
wrote:
Hello
The picture below describes the deployment
architecture I am trying to achieve.
However, when I run the wordcount example code
with the below configuration, by issuing the
command from the master node, I notice only the
master node spawning map tasks and completing the
submitted job. Below is the command I used:
hadoop jar
/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
wordcount -fs local -jt local
/hduser/mount_point/ /results
Question: How can I leverage both the hadoop
nodes for running MR, while serving my data from
the common NFS mount point running my filesystem
at the backend? Has any one tried such a setup
before?
Thanks!
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Figured out 1. The output of the reduce was going to the slave node, while
I was looking for it in the master node. Which is perfectly fine.
Need guidance for 2. though!
Thanks
Atish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Figured out 1. The output of the reduce was going to the slave node, while
I was looking for it in the master node. Which is perfectly fine.
Need guidance for 2. though!
Thanks
Atish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Figured out 1. The output of the reduce was going to the slave node, while
I was looking for it in the master node. Which is perfectly fine.
Need guidance for 2. though!
Thanks
Atish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>
>
Re: Running Hadoop v2 clustered mode MR on an NFS mounted filesystem
Posted by Atish Kathpal <at...@gmail.com>.
Figured out 1. The output of the reduce was going to the slave node, while
I was looking for it in the master node. Which is perfectly fine.
Need guidance for 2. though!
Thanks
Atish
On Wed, Jan 8, 2014 at 3:30 PM, Atish Kathpal <at...@gmail.com>wrote:
> Hi
>
> By giving the complete URI, the MR jobs worked across both nodes. Thanks a
> lot for the advice.
>
> *Two issues though*:
> 1. On completion of the MR job, I see only the "_SUCCESS" file in the
> output directory, but no part-r file containing the actual results of the
> wordcount job. However I am seeing the correct output on running MR over
> HDFS. What is going wrong? Any place I can find logs for the MR job. I see
> no errors on the console.
> Command used:
> hadoop jar
> /home/hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> wordcount file:///home/hduser/testmount/ file:///home/hduser/testresults/
>
>
> 2. I am observing that the mappers seem to be accessing files
> sequentially, splitting the files across mappers, and then reading data in
> parallelel, then moving on to the next file. What I want instead is that,
> files themselves should be accessed in parallel, that is, if there are 10
> files to be MRed, then MR should ask for each of these files in parallel in
> one go, and then work on the splits of these files in parallel.
> *Why do I need this?* Some of the data coming from the NFS mount point is
> coming from offline media (which takes ~5-10 seconds of time before first
> bytes are received). So I would like all required files to be asked at the
> onset itself from the NFS mount point. This way several offline media will
> be spun up parallely and as the data from these media gets available MR can
> process them.
>
> Would be glad to get inputs on these points!
>
> Thanks
> Atish
>
> Tip for those who are trying similar stuff::
> In my case. after a while the jobs would fail, complaining of "java.lang.OutOfMemoryError:
> Java heap space<http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space>",
> but I was able to rectify this with help from:
> http://stackoverflow.com/questions/13674190/cdh-4-1-error-running-child-java-lang-outofmemoryerror-java-heap-space
>
>
>
>
>
> On Sun, Dec 22, 2013 at 2:47 PM, Atish Kathpal <at...@gmail.com>wrote:
>
>> Thanks Devin, Yong, and Chris for your replies and suggestions. I will
>> test the suggestions made by Yong and Devin and get back to you guys.
>>
>> As on the bottlenecking issue, I agree, but I am trying to run few MR
>> jobs on a traditional NAS server. I can live with a few bottlenecks, so
>> long as I don't have to move the data to a dedicated HDFS cluster.
>>
>>
>> On Sat, Dec 21, 2013 at 8:06 AM, Chris Mawata <ch...@gmail.com>wrote:
>>
>>> Yong raises an important issue: You have thrown out the I/O
>>> advantages of HDFS and also thrown out the advantages of data locality. It
>>> would be interesting to know why you are taking this approach.
>>> Chris
>>>
>>>
>>> On 12/20/2013 9:28 AM, java8964 wrote:
>>>
>>> I believe the "-fs local" should be removed too. The reason is that even
>>> you have a dedicated JobTracker after removing "-jt local", but with "-fs
>>> local", I believe that all the mappers will be run sequentially.
>>>
>>> "-fs local" will force the mapreducer run in "local" mode, which is
>>> really a test mode.
>>>
>>> What you can do is to remove both "-fs local -jt local", but give the
>>> FULL URI of the input and output path, to tell Hadoop that they are local
>>> filesystem instead of HDFS.
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount file:///hduser/mount_point file:///results"
>>>
>>> Keep in mind followings:
>>>
>>> 1) The NFS mount need to be available in all your Task Nodes, and
>>> mounted in the same way.
>>> 2) Even you can do that, but your sharing storage will be your
>>> bottleneck. NFS won't work well for scalability.
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Fri, 20 Dec 2013 09:01:32 -0500
>>> Subject: Re: Running Hadoop v2 clustered mode MR on an NFS mounted
>>> filesystem
>>> From: dsuiter@rdx.com
>>> To: user@hadoop.apache.org
>>>
>>> I think most of your problem is coming from the options you are setting:
>>>
>>> "hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount *-fs local -jt local* /hduser/mount_point/ /results"
>>>
>>> You appear to be directing your namenode to run jobs in the *LOCAL* job
>>> runner and directing it to read from the *LOCAL* filesystem. Drop the
>>> *-jt* argument and it should run in distributed mode if your cluster is
>>> set up right. You don't need to do anything special to point Hadoop towards
>>> a NFS location, other than set up the NFS location properly and make sure
>>> if you are directing to it by name that it will resolve to the right
>>> address. Hadoop doesn't care where it is, as long as it can read from and
>>> write to it. The fact that you are telling it to read/write from/to a NFS
>>> location that happens to be mounted as a local filesystem object doesn't
>>> matter - you could direct it to the local /hduser/ path and set the -fs
>>> local option, and it would end up on the NFS mount, because that's where
>>> the NFS mount actually exists, or you could direct it to the absolute
>>> network location of the folder that you want, it shouldn't make a
>>> difference.
>>>
>>> *Devin Suiter*
>>> Jr. Data Solutions Software Engineer
>>> 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
>>> Google Voice: 412-256-8556 | www.rdx.com
>>>
>>>
>>> On Fri, Dec 20, 2013 at 5:27 AM, Atish Kathpal <at...@gmail.com>wrote:
>>>
>>> Hello
>>>
>>> The picture below describes the deployment architecture I am trying to
>>> achieve.
>>> However, when I run the wordcount example code with the below
>>> configuration, by issuing the command from the master node, I notice only
>>> the master node spawning map tasks and completing the submitted job. Below
>>> is the command I used:
>>>
>>> *hadoop jar
>>> /hduser/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
>>> wordcount -fs local -jt local /hduser/mount_point/ /results*
>>>
>>> *Question: How can I leverage both the hadoop nodes for running MR,
>>> while serving my data from the common NFS mount point running my filesystem
>>> at the backend? Has any one tried such a setup before?*
>>> [image: Inline image 1]
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>
>