Passing a Global Variable into a Mapper

[Arko Provo Mukherjee <ar...@gmail.com>]

Hi,

Is there a way to pass some data from the driver class to the Mapper
class without going through the HDFS?

Does the API provide us with some functionality to pass some variables?

Thanks a lot in advance!
Warm regards
Arko

Re: Passing a Global Variable into a Mapper

[John Armstrong <jo...@ccri.com>]

On Thu, 15 Sep 2011 12:43:57 -0500, Arko Provo Mukherjee
<ar...@gmail.com> wrote:
> Is there a way to pass some data from the driver class to the Mapper
> class without going through the HDFS?

I generally use the Configuration object embedded in the Job for that.  My
Tool implements Configurable so I create by job with

    Job job = new Job(getConf(), "JobName");

Then I set variables that my job's tasks will need:

    job.getconfiguration().setString("property.name","property.value");

On the inside, usually in my Mapper.setup() or Reducer.setup() I extract
them:

    Configuration conf = context.getConfiguration();
    property = conf.get("property.name");

where property is a String field in my mapper that will hold the
property's value.  Of course, change things mutatis mutandis for storing
other kinds of values than Strings in the Configuration.

hth

Re: Passing a Global Variable into a Mapper

[Wellington Chevreuil <we...@gmail.com>]

Hi Arko,

you can do that using the org.apache.hadoop.filecache.DistributedCache class.

You just need to put the file in the distributed cache, in your driver class:
        ...
        DistributedCache.addCacheFile(new
Path(YOUR_FILE_PATH).toUri(), job.getConfiguration());
        ...

Then get it from the Distributed Cache, on your map class:
        ...
        Path[] cacheFiles =
DistributedCache.getLocalCacheFiles(context.getConfiguration());
        ...

Hope this helps you.

Regards,
Wellington.

2011/9/15 Arko Provo Mukherjee <ar...@gmail.com>:
> Hi,
>
> Is there a way to pass some data from the driver class to the Mapper
> class without going through the HDFS?
>
> Does the API provide us with some functionality to pass some variables?
>
> Thanks a lot in advance!
> Warm regards
> Arko
>

Re: Passing a Global Variable into a Mapper

[Arko Provo Mukherjee <ar...@gmail.com>]

Thanks All!
Warm Regards
Arko

On Thu, Sep 15, 2011 at 12:53 PM, Swathi V <sw...@zinniasystems.com> wrote:
> Hi,
> I guess you can make use of job.getconfiguration().set() in driver to set
> the variable and in mapper you can make use of setup method like this:
> Configuration conf = context.getconfiguration();
> somevar = conf.get();
>
> On Thu, Sep 15, 2011 at 11:13 PM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> Is there a way to pass some data from the driver class to the Mapper
>> class without going through the HDFS?
>>
>> Does the API provide us with some functionality to pass some variables?
>>
>> Thanks a lot in advance!
>> Warm regards
>> Arko
>
>
>
> --
> Regards,
> Swathi.V.
>
>

Re: Passing a Global Variable into a Mapper

[Swathi V <sw...@zinniasystems.com>]

Hi,
I guess you can make use of job.getconfiguration().set() in driver to set
the variable and in mapper you can make use of setup method like this:
Configuration conf = context.getconfiguration();
somevar = conf.get();

On Thu, Sep 15, 2011 at 11:13 PM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:

> Hi,
>
> Is there a way to pass some data from the driver class to the Mapper
> class without going through the HDFS?
>
> Does the API provide us with some functionality to pass some variables?
>
> Thanks a lot in advance!
> Warm regards
> Arko
>



-- 
Regards,
Swathi.V.