You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Shrivastava, Himnshu (GE Global Research, Non-GE)" <Hi...@ge.com> on 2014/06/20 08:27:17 UTC
Map reduce query
How can I give Input to a mapper from the command line? -D option can be used but what are the corresponding changes required in the mapper and the driver program ?
Regards,
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
U can try it using Distributed cache
In Driver
FileStatus[] list = fs.globStatus(extrafile);
for (FileStatus status : list) {
DistributedCache.addCacheFile(status.getPath().toUri(), conf);
}
In Map
URI[] cacheFiles = DistributedCache.getCacheFiles(conf);
Path getPath = new Path(cacheFiles[0].getPath());
BufferedReader bf = new BufferedReader(new
InputStreamReader(fs.open(getPath)));
On Fri, Jun 20, 2014 at 1:12 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Hi
>
> You can directly use this right?
>
> FileInputFormat.setInputPaths(job,new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> Or you need extra input file to feed into mapper?
>
>
> On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global
> Research, Non-GE) <Hi...@ge.com> wrote:
>
>> How can I give Input to a mapper from the command line? –D option can
>> be used but what are the corresponding changes required in the mapper and
>> the driver program ?
>>
>>
>>
>> Regards,
>>
>
>
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
U can try it using Distributed cache
In Driver
FileStatus[] list = fs.globStatus(extrafile);
for (FileStatus status : list) {
DistributedCache.addCacheFile(status.getPath().toUri(), conf);
}
In Map
URI[] cacheFiles = DistributedCache.getCacheFiles(conf);
Path getPath = new Path(cacheFiles[0].getPath());
BufferedReader bf = new BufferedReader(new
InputStreamReader(fs.open(getPath)));
On Fri, Jun 20, 2014 at 1:12 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Hi
>
> You can directly use this right?
>
> FileInputFormat.setInputPaths(job,new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> Or you need extra input file to feed into mapper?
>
>
> On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global
> Research, Non-GE) <Hi...@ge.com> wrote:
>
>> How can I give Input to a mapper from the command line? –D option can
>> be used but what are the corresponding changes required in the mapper and
>> the driver program ?
>>
>>
>>
>> Regards,
>>
>
>
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
U can try it using Distributed cache
In Driver
FileStatus[] list = fs.globStatus(extrafile);
for (FileStatus status : list) {
DistributedCache.addCacheFile(status.getPath().toUri(), conf);
}
In Map
URI[] cacheFiles = DistributedCache.getCacheFiles(conf);
Path getPath = new Path(cacheFiles[0].getPath());
BufferedReader bf = new BufferedReader(new
InputStreamReader(fs.open(getPath)));
On Fri, Jun 20, 2014 at 1:12 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Hi
>
> You can directly use this right?
>
> FileInputFormat.setInputPaths(job,new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> Or you need extra input file to feed into mapper?
>
>
> On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global
> Research, Non-GE) <Hi...@ge.com> wrote:
>
>> How can I give Input to a mapper from the command line? –D option can
>> be used but what are the corresponding changes required in the mapper and
>> the driver program ?
>>
>>
>>
>> Regards,
>>
>
>
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
U can try it using Distributed cache
In Driver
FileStatus[] list = fs.globStatus(extrafile);
for (FileStatus status : list) {
DistributedCache.addCacheFile(status.getPath().toUri(), conf);
}
In Map
URI[] cacheFiles = DistributedCache.getCacheFiles(conf);
Path getPath = new Path(cacheFiles[0].getPath());
BufferedReader bf = new BufferedReader(new
InputStreamReader(fs.open(getPath)));
On Fri, Jun 20, 2014 at 1:12 PM, unmesha sreeveni <un...@gmail.com>
wrote:
> Hi
>
> You can directly use this right?
>
> FileInputFormat.setInputPaths(job,new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
> Or you need extra input file to feed into mapper?
>
>
> On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global
> Research, Non-GE) <Hi...@ge.com> wrote:
>
>> How can I give Input to a mapper from the command line? –D option can
>> be used but what are the corresponding changes required in the mapper and
>> the driver program ?
>>
>>
>>
>> Regards,
>>
>
>
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
Hi
You can directly use this right?
FileInputFormat.setInputPaths(job,new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
Or you need extra input file to feed into mapper?
On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> How can I give Input to a mapper from the command line? –D option can be
> used but what are the corresponding changes required in the mapper and the
> driver program ?
>
>
>
> Regards,
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
Hi
You can directly use this right?
FileInputFormat.setInputPaths(job,new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
Or you need extra input file to feed into mapper?
On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> How can I give Input to a mapper from the command line? –D option can be
> used but what are the corresponding changes required in the mapper and the
> driver program ?
>
>
>
> Regards,
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
Hi
You can directly use this right?
FileInputFormat.setInputPaths(job,new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
Or you need extra input file to feed into mapper?
On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> How can I give Input to a mapper from the command line? –D option can be
> used but what are the corresponding changes required in the mapper and the
> driver program ?
>
>
>
> Regards,
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
Re: Map reduce query
Posted by unmesha sreeveni <un...@gmail.com>.
Hi
You can directly use this right?
FileInputFormat.setInputPaths(job,new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
Or you need extra input file to feed into mapper?
On Fri, Jun 20, 2014 at 11:57 AM, Shrivastava, Himnshu (GE Global Research,
Non-GE) <Hi...@ge.com> wrote:
> How can I give Input to a mapper from the command line? –D option can be
> used but what are the corresponding changes required in the mapper and the
> driver program ?
>
>
>
> Regards,
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/