You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Agarwal, Nikhil" <Ni...@netapp.com> on 2013/02/17 07:22:47 UTC
Can I perfrom a MR on my local filesystem
Hi,
Recently I followed a blog to run Hadoop on a single node cluster.
I wanted to ask that in a single node set-up of Hadoop is it necessary to have the data copied into Hadoop's HDFS before running a MR on it. Can I run MR on my local file system too without copying the data to HDFS?
In the Hadoop source code I saw there are implementations of other file systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3 data store ? How does JobTracker or Tasktracker run in S3 ?
I would be very thankful to get a reply to this.
Thanks & Regards,
Nikhil
Re: Can I perfrom a MR on my local filesystem
Posted by Niels Basjes <Ni...@basjes.nl>.
Have a look at this
http://stackoverflow.com/questions/3546025/is-it-possible-to-run-hadoop-in-pseudo-distributed-operation-without-hdfs
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 17 feb. 2013 07:51 schreef "Agarwal, Nikhil" <Ni...@netapp.com>
het volgende:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>
Re: Can I perfrom a MR on my local filesystem
Posted by Nitin Pawar <ni...@gmail.com>.
Nikhil, may be this blog will help you understand it how s3 is used
http://www.technology-mania.com/2012/05/s3-instead-of-hdfs-with-hadoop_05.html
http://www.technology-mania.com/2011/05/s3-as-input-or-output-for-hadoop-mr.html
note s3 is storage house and it does not have any compute capacity to run
map reduce jobs
your map reduce jobs are always running on ec2 nodes (are you looking at
how the data is accessed by mapreduce jobs while s3 as filesystem? )
On Sun, Feb 17, 2013 at 11:52 AM, Agarwal, Nikhil <Nikhil.Agarwal@netapp.com
> wrote:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>
--
Nitin Pawar
Re: Can I perfrom a MR on my local filesystem
Posted by Nitin Pawar <ni...@gmail.com>.
Nikhil, may be this blog will help you understand it how s3 is used
http://www.technology-mania.com/2012/05/s3-instead-of-hdfs-with-hadoop_05.html
http://www.technology-mania.com/2011/05/s3-as-input-or-output-for-hadoop-mr.html
note s3 is storage house and it does not have any compute capacity to run
map reduce jobs
your map reduce jobs are always running on ec2 nodes (are you looking at
how the data is accessed by mapreduce jobs while s3 as filesystem? )
On Sun, Feb 17, 2013 at 11:52 AM, Agarwal, Nikhil <Nikhil.Agarwal@netapp.com
> wrote:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>
--
Nitin Pawar
Re: Can I perfrom a MR on my local filesystem
Posted by Nitin Pawar <ni...@gmail.com>.
Nikhil, may be this blog will help you understand it how s3 is used
http://www.technology-mania.com/2012/05/s3-instead-of-hdfs-with-hadoop_05.html
http://www.technology-mania.com/2011/05/s3-as-input-or-output-for-hadoop-mr.html
note s3 is storage house and it does not have any compute capacity to run
map reduce jobs
your map reduce jobs are always running on ec2 nodes (are you looking at
how the data is accessed by mapreduce jobs while s3 as filesystem? )
On Sun, Feb 17, 2013 at 11:52 AM, Agarwal, Nikhil <Nikhil.Agarwal@netapp.com
> wrote:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>
--
Nitin Pawar
Re: Can I perfrom a MR on my local filesystem
Posted by Nitin Pawar <ni...@gmail.com>.
Nikhil, may be this blog will help you understand it how s3 is used
http://www.technology-mania.com/2012/05/s3-instead-of-hdfs-with-hadoop_05.html
http://www.technology-mania.com/2011/05/s3-as-input-or-output-for-hadoop-mr.html
note s3 is storage house and it does not have any compute capacity to run
map reduce jobs
your map reduce jobs are always running on ec2 nodes (are you looking at
how the data is accessed by mapreduce jobs while s3 as filesystem? )
On Sun, Feb 17, 2013 at 11:52 AM, Agarwal, Nikhil <Nikhil.Agarwal@netapp.com
> wrote:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>
--
Nitin Pawar
Re: Can I perfrom a MR on my local filesystem
Posted by Niels Basjes <Ni...@basjes.nl>.
Have a look at this
http://stackoverflow.com/questions/3546025/is-it-possible-to-run-hadoop-in-pseudo-distributed-operation-without-hdfs
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 17 feb. 2013 07:51 schreef "Agarwal, Nikhil" <Ni...@netapp.com>
het volgende:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>
Re: Can I perfrom a MR on my local filesystem
Posted by Niels Basjes <Ni...@basjes.nl>.
Have a look at this
http://stackoverflow.com/questions/3546025/is-it-possible-to-run-hadoop-in-pseudo-distributed-operation-without-hdfs
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 17 feb. 2013 07:51 schreef "Agarwal, Nikhil" <Ni...@netapp.com>
het volgende:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>
Re: Can I perfrom a MR on my local filesystem
Posted by Niels Basjes <Ni...@basjes.nl>.
Have a look at this
http://stackoverflow.com/questions/3546025/is-it-possible-to-run-hadoop-in-pseudo-distributed-operation-without-hdfs
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 17 feb. 2013 07:51 schreef "Agarwal, Nikhil" <Ni...@netapp.com>
het volgende:
> Hi,
>
> ****
>
> Recently I followed a blog to run Hadoop on a single node cluster.****
>
> I wanted to ask that in a single node set-up of Hadoop is it necessary to
> have the data copied into Hadoop’s HDFS before running a MR on it. Can I
> run MR on my local file system too without copying the data to HDFS? ****
>
> In the Hadoop source code I saw there are implementations of other file
> systems too like S3, KFS, FTP, etc. so how does exactly a MR happen on S3
> data store ? How does JobTracker or Tasktracker run in S3 ? ****
>
> ** **
>
> I would be very thankful to get a reply to this.****
>
> ** **
>
> Thanks & Regards,****
>
> Nikhil****
>
> ** **
>