You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Francis.Hu" <fr...@reachjunction.com> on 2013/08/15 06:13:09 UTC
答复: Passing an object in mapper
hi,
what I did when data need to be saved into DB :
//configure DB connection info.
DBConfiguration.configureDB(config,
HadoopJobConfigParser.getProperty(DB_DRIVER),
HadoopJobConfigParser.getProperty(DB_URL),
HadoopJobConfigParser.getProperty(DB_USER),
HadoopJobConfigParser.getProperty(DB_PWD));
//write a OutputKeyClass like below:
public class DisplayRequestRecord implements DBWritable,
WritableComparable<DisplayRequestRecord>{
//TODO implement write() and readFiels() methods
}
//set the job's output
job.setOutputKeyClass(DisplayRequestRecord.class);
job.setOutputValueClass(DisplayRequestRecord.class);
DBOutputFormat.setOutput(job, tableName(),Fields());
//define reducer class like below:
public class DBDisplayRequestReducerNew extends Reducer<Text,
DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{
public void reduce(Text key, Iterable<DoubleWritable> values,
Context context) throws IOException, InterruptedException {
DisplayRequestRecord dr = new DisplayRequestRecord();
context.write(dr,dr);
}
}
Hope it helps!
Thanks,
Francis.Hu
发件人: Sivaram RL [mailto:sivaramrls@gmail.com]
发送时间: Thursday, August 15, 2013 11:26
收件人: user@hadoop.apache.org
主题: Re: Passing an object in mapper
Hi,
In the Configuration object in your driver class you can set the properties
as key value pair. This configuration object will be set in the Job Object.
The same properties can be accessed in the mapper/reducer using the Context
Object -> getConfiguration() -> get(propertyName).
Hope this helps.
Regards,
Sivaram R L
On Thu, Aug 15, 2013 at 8:39 AM, jamal sasha <ja...@gmail.com> wrote:
Hi,
I am initializing an object in driver code.
For sake of argument let say I want to save data to some database..
say:
Connection con = new Connection(host, db);
Now, in reducer I want to do something like
con.write(key,value)
So, how do i pass this object from driver to mapper / reducer/?
Any clue? suggestions?
Thanks
Re: 答复: Passing an object in mapper
Posted by jamal sasha <ja...@gmail.com>.
Thanks guys :)
On Wed, Aug 14, 2013 at 9:13 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> hi, ****
>
> ** **
>
> what I did when data need to be saved into DB :****
>
> ** **
>
> //configure DB connection info.****
>
> DBConfiguration.*configureDB*(config,****
>
> HadoopJobConfigParser.*getProperty*(*DB_DRIVER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_URL*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_USER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_PWD*));****
>
> ** **
>
> //write a OutputKeyClass like below:****
>
> public class DisplayRequestRecord implements DBWritable,
> WritableComparable<DisplayRequestRecord>{****
>
> //TODO implement write() and readFiels() methods****
>
> }****
>
> ** **
>
> //set the job's output****
>
> job.setOutputKeyClass(DisplayRequestRecord.class);****
>
> job.setOutputValueClass(DisplayRequestRecord.class);****
>
> *DBOutputFormat.setOutput(job, tableName(),Fields());*
>
> * *
>
> //define reducer class like below:****
>
> public class DBDisplayRequestReducerNew extends Reducer<Text,
> DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{****
>
> *public* *void* reduce(Text key, Iterable<DoubleWritable> values,****
>
> Context context) *throws* IOException, InterruptedException {*
> ***
>
> DisplayRequestRecord dr = new DisplayRequestRecord();****
>
> context.write(dr,dr);****
>
> }****
>
> }****
>
> ** **
>
> ** **
>
> Hope it helps!****
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *发件人:* Sivaram RL [mailto:sivaramrls@gmail.com]
> *发送时间:* Thursday, August 15, 2013 11:26
> *收件人:* user@hadoop.apache.org
> *主题:* Re: Passing an object in mapper****
>
> ** **
>
> Hi,****
>
> ** **
>
> In the Configuration object in your driver class you can set the
> properties as key value pair. This configuration object will be set in the
> Job Object. The same properties can be accessed in the mapper/reducer using
> the Context Object -> getConfiguration() -> get(propertyName).****
>
> ** **
>
> Hope this helps.****
>
> ** **
>
> Regards,****
>
> Sivaram R L****
>
> ** **
>
> On Thu, Aug 15, 2013 at 8:39 AM, jamal sasha <ja...@gmail.com>
> wrote:****
>
> Hi,****
>
> I am initializing an object in driver code.****
>
> For sake of argument let say I want to save data to some database..****
>
> say:****
>
> Connection con = new Connection(host, db);****
>
> Now, in reducer I want to do something like****
>
> con.write(key,value) ****
>
> ** **
>
> So, how do i pass this object from driver to mapper / reducer/?
> Any clue? suggestions?****
>
> Thanks****
>
> ** **
>
Re: 答复: Passing an object in mapper
Posted by jamal sasha <ja...@gmail.com>.
Thanks guys :)
On Wed, Aug 14, 2013 at 9:13 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> hi, ****
>
> ** **
>
> what I did when data need to be saved into DB :****
>
> ** **
>
> //configure DB connection info.****
>
> DBConfiguration.*configureDB*(config,****
>
> HadoopJobConfigParser.*getProperty*(*DB_DRIVER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_URL*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_USER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_PWD*));****
>
> ** **
>
> //write a OutputKeyClass like below:****
>
> public class DisplayRequestRecord implements DBWritable,
> WritableComparable<DisplayRequestRecord>{****
>
> //TODO implement write() and readFiels() methods****
>
> }****
>
> ** **
>
> //set the job's output****
>
> job.setOutputKeyClass(DisplayRequestRecord.class);****
>
> job.setOutputValueClass(DisplayRequestRecord.class);****
>
> *DBOutputFormat.setOutput(job, tableName(),Fields());*
>
> * *
>
> //define reducer class like below:****
>
> public class DBDisplayRequestReducerNew extends Reducer<Text,
> DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{****
>
> *public* *void* reduce(Text key, Iterable<DoubleWritable> values,****
>
> Context context) *throws* IOException, InterruptedException {*
> ***
>
> DisplayRequestRecord dr = new DisplayRequestRecord();****
>
> context.write(dr,dr);****
>
> }****
>
> }****
>
> ** **
>
> ** **
>
> Hope it helps!****
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *发件人:* Sivaram RL [mailto:sivaramrls@gmail.com]
> *发送时间:* Thursday, August 15, 2013 11:26
> *收件人:* user@hadoop.apache.org
> *主题:* Re: Passing an object in mapper****
>
> ** **
>
> Hi,****
>
> ** **
>
> In the Configuration object in your driver class you can set the
> properties as key value pair. This configuration object will be set in the
> Job Object. The same properties can be accessed in the mapper/reducer using
> the Context Object -> getConfiguration() -> get(propertyName).****
>
> ** **
>
> Hope this helps.****
>
> ** **
>
> Regards,****
>
> Sivaram R L****
>
> ** **
>
> On Thu, Aug 15, 2013 at 8:39 AM, jamal sasha <ja...@gmail.com>
> wrote:****
>
> Hi,****
>
> I am initializing an object in driver code.****
>
> For sake of argument let say I want to save data to some database..****
>
> say:****
>
> Connection con = new Connection(host, db);****
>
> Now, in reducer I want to do something like****
>
> con.write(key,value) ****
>
> ** **
>
> So, how do i pass this object from driver to mapper / reducer/?
> Any clue? suggestions?****
>
> Thanks****
>
> ** **
>
Re: 答复: Passing an object in mapper
Posted by jamal sasha <ja...@gmail.com>.
Thanks guys :)
On Wed, Aug 14, 2013 at 9:13 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> hi, ****
>
> ** **
>
> what I did when data need to be saved into DB :****
>
> ** **
>
> //configure DB connection info.****
>
> DBConfiguration.*configureDB*(config,****
>
> HadoopJobConfigParser.*getProperty*(*DB_DRIVER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_URL*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_USER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_PWD*));****
>
> ** **
>
> //write a OutputKeyClass like below:****
>
> public class DisplayRequestRecord implements DBWritable,
> WritableComparable<DisplayRequestRecord>{****
>
> //TODO implement write() and readFiels() methods****
>
> }****
>
> ** **
>
> //set the job's output****
>
> job.setOutputKeyClass(DisplayRequestRecord.class);****
>
> job.setOutputValueClass(DisplayRequestRecord.class);****
>
> *DBOutputFormat.setOutput(job, tableName(),Fields());*
>
> * *
>
> //define reducer class like below:****
>
> public class DBDisplayRequestReducerNew extends Reducer<Text,
> DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{****
>
> *public* *void* reduce(Text key, Iterable<DoubleWritable> values,****
>
> Context context) *throws* IOException, InterruptedException {*
> ***
>
> DisplayRequestRecord dr = new DisplayRequestRecord();****
>
> context.write(dr,dr);****
>
> }****
>
> }****
>
> ** **
>
> ** **
>
> Hope it helps!****
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *发件人:* Sivaram RL [mailto:sivaramrls@gmail.com]
> *发送时间:* Thursday, August 15, 2013 11:26
> *收件人:* user@hadoop.apache.org
> *主题:* Re: Passing an object in mapper****
>
> ** **
>
> Hi,****
>
> ** **
>
> In the Configuration object in your driver class you can set the
> properties as key value pair. This configuration object will be set in the
> Job Object. The same properties can be accessed in the mapper/reducer using
> the Context Object -> getConfiguration() -> get(propertyName).****
>
> ** **
>
> Hope this helps.****
>
> ** **
>
> Regards,****
>
> Sivaram R L****
>
> ** **
>
> On Thu, Aug 15, 2013 at 8:39 AM, jamal sasha <ja...@gmail.com>
> wrote:****
>
> Hi,****
>
> I am initializing an object in driver code.****
>
> For sake of argument let say I want to save data to some database..****
>
> say:****
>
> Connection con = new Connection(host, db);****
>
> Now, in reducer I want to do something like****
>
> con.write(key,value) ****
>
> ** **
>
> So, how do i pass this object from driver to mapper / reducer/?
> Any clue? suggestions?****
>
> Thanks****
>
> ** **
>
Re: 答复: Passing an object in mapper
Posted by jamal sasha <ja...@gmail.com>.
Thanks guys :)
On Wed, Aug 14, 2013 at 9:13 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> hi, ****
>
> ** **
>
> what I did when data need to be saved into DB :****
>
> ** **
>
> //configure DB connection info.****
>
> DBConfiguration.*configureDB*(config,****
>
> HadoopJobConfigParser.*getProperty*(*DB_DRIVER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_URL*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_USER*),****
>
> HadoopJobConfigParser.*getProperty*(*DB_PWD*));****
>
> ** **
>
> //write a OutputKeyClass like below:****
>
> public class DisplayRequestRecord implements DBWritable,
> WritableComparable<DisplayRequestRecord>{****
>
> //TODO implement write() and readFiels() methods****
>
> }****
>
> ** **
>
> //set the job's output****
>
> job.setOutputKeyClass(DisplayRequestRecord.class);****
>
> job.setOutputValueClass(DisplayRequestRecord.class);****
>
> *DBOutputFormat.setOutput(job, tableName(),Fields());*
>
> * *
>
> //define reducer class like below:****
>
> public class DBDisplayRequestReducerNew extends Reducer<Text,
> DoubleWritable, DisplayRequestRecord, DisplayRequestRecord>{****
>
> *public* *void* reduce(Text key, Iterable<DoubleWritable> values,****
>
> Context context) *throws* IOException, InterruptedException {*
> ***
>
> DisplayRequestRecord dr = new DisplayRequestRecord();****
>
> context.write(dr,dr);****
>
> }****
>
> }****
>
> ** **
>
> ** **
>
> Hope it helps!****
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *发件人:* Sivaram RL [mailto:sivaramrls@gmail.com]
> *发送时间:* Thursday, August 15, 2013 11:26
> *收件人:* user@hadoop.apache.org
> *主题:* Re: Passing an object in mapper****
>
> ** **
>
> Hi,****
>
> ** **
>
> In the Configuration object in your driver class you can set the
> properties as key value pair. This configuration object will be set in the
> Job Object. The same properties can be accessed in the mapper/reducer using
> the Context Object -> getConfiguration() -> get(propertyName).****
>
> ** **
>
> Hope this helps.****
>
> ** **
>
> Regards,****
>
> Sivaram R L****
>
> ** **
>
> On Thu, Aug 15, 2013 at 8:39 AM, jamal sasha <ja...@gmail.com>
> wrote:****
>
> Hi,****
>
> I am initializing an object in driver code.****
>
> For sake of argument let say I want to save data to some database..****
>
> say:****
>
> Connection con = new Connection(host, db);****
>
> Now, in reducer I want to do something like****
>
> con.write(key,value) ****
>
> ** **
>
> So, how do i pass this object from driver to mapper / reducer/?
> Any clue? suggestions?****
>
> Thanks****
>
> ** **
>