You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Oliver B. Fischer" <o....@swe-blog.net> on 2009/10/21 14:49:09 UTC
Using Configuration instead of JobConf
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
according to the API-Dokumentation of 0.20.1 JobConf is deprecated and
we should use Configuration instead. However all examples on the webpage
still referece JobConf.
Is there a good example for replacing JobConf by Configuration?
Regards, Oliver
- --
Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
Tel. +49 30 44793251, Mobil: +49 178 7903538
Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJK3wNFAAoJELeiwuwqd1DGf4YH/3P83TQh+7L0GiKT4rdIe1HC
LdZ1weVcqnBKfFZYCsL64PfwPGhkl0hoTU/sU29qGGrnAN8QKbF/8m2P32RlysCV
ONT2nUlugHfEb8iQzF8Xx3uuCzU3AXYHK0eqme+thzfscxZKiGKzLTYb26qFnufG
AtTIEKMkv5SRLIn5YRFM/49dcSp2WIRCIEcc+0eofsxFvI9X2a8Sg8AMG5JzNqYT
SHMGSr3boBc4gFCNTCsRmJQIusPVWyJZICVss0q3491xdMVvGfAPTsbb6BLINFkL
j5BrbC9OaNnmX3X3kR2/orABwMj9y5Xd1YjXepeWG3QD3WftPVyoCIOE1v8m2s0=
=OFkK
-----END PGP SIGNATURE-----
Re: Using Configuration instead of JobConf
Posted by Ryan Smith <ry...@gmail.com>.
Also, FileInputFormat is deprecated in hadoop 0.20.1.
On Wed, Oct 21, 2009 at 10:15 AM, Mark Vigeant <mark.vigeant@riskmetrics.com
> wrote:
> Hi Oliver,
>
> I ran into the same problem a few weeks ago. What you want to do now is use
> the classes Job and Configuration. Here's basically what you need to do:
>
>
> Configuration conf = new Configuration(); //creates the configuration
> object
> conf.set(TableOutputFormat.OUTPUT_TABLE, name); //set the output format:
> I'm using this for HBase so it's an HTable.
> Job job = new Job(conf, NAME);
> job.setJarByClass(TableUploader.class);
> job.setInputFormatClass(FileInputFormat.class);
> job.setMapperClass(MapExample.class);
> job.setNumReduceTasks(0); //alternately, you could set a Reducer class.
> Check the API
> FileInputFormat.setInputPaths(job, new Path(input));
> job.setOutputFormatClass(TableOutputFormat.class);
> job.waitForCompletion(true); //submits the job, waits for it to be
> completed.
>
> -----Original Message-----
> From: Oliver B. Fischer [mailto:o.b.fischer@swe-blog.net]
> Sent: Wednesday, October 21, 2009 8:49 AM
> To: common-user@hadoop.apache.org
> Subject: Using Configuration instead of JobConf
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi,
>
> according to the API-Dokumentation of 0.20.1 JobConf is deprecated and
> we should use Configuration instead. However all examples on the webpage
> still referece JobConf.
>
> Is there a good example for replacing JobConf by Configuration?
>
> Regards, Oliver
>
> - --
> Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
> Tel. +49 30 44793251, Mobil: +49 178 7903538
> Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJK3wNFAAoJELeiwuwqd1DGf4YH/3P83TQh+7L0GiKT4rdIe1HC
> LdZ1weVcqnBKfFZYCsL64PfwPGhkl0hoTU/sU29qGGrnAN8QKbF/8m2P32RlysCV
> ONT2nUlugHfEb8iQzF8Xx3uuCzU3AXYHK0eqme+thzfscxZKiGKzLTYb26qFnufG
> AtTIEKMkv5SRLIn5YRFM/49dcSp2WIRCIEcc+0eofsxFvI9X2a8Sg8AMG5JzNqYT
> SHMGSr3boBc4gFCNTCsRmJQIusPVWyJZICVss0q3491xdMVvGfAPTsbb6BLINFkL
> j5BrbC9OaNnmX3X3kR2/orABwMj9y5Xd1YjXepeWG3QD3WftPVyoCIOE1v8m2s0=
> =OFkK
> -----END PGP SIGNATURE-----
>
Re: Using Configuration instead of JobConf
Posted by Ryan Smith <ry...@gmail.com>.
It would be great if someone could update the src code here:
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Source+Code
On Wed, Oct 21, 2009 at 10:15 AM, Mark Vigeant <mark.vigeant@riskmetrics.com
> wrote:
> Hi Oliver,
>
> I ran into the same problem a few weeks ago. What you want to do now is use
> the classes Job and Configuration. Here's basically what you need to do:
>
>
> Configuration conf = new Configuration(); //creates the configuration
> object
> conf.set(TableOutputFormat.OUTPUT_TABLE, name); //set the output format:
> I'm using this for HBase so it's an HTable.
> Job job = new Job(conf, NAME);
> job.setJarByClass(TableUploader.class);
> job.setInputFormatClass(FileInputFormat.class);
> job.setMapperClass(MapExample.class);
> job.setNumReduceTasks(0); //alternately, you could set a Reducer class.
> Check the API
> FileInputFormat.setInputPaths(job, new Path(input));
> job.setOutputFormatClass(TableOutputFormat.class);
> job.waitForCompletion(true); //submits the job, waits for it to be
> completed.
>
> -----Original Message-----
> From: Oliver B. Fischer [mailto:o.b.fischer@swe-blog.net]
> Sent: Wednesday, October 21, 2009 8:49 AM
> To: common-user@hadoop.apache.org
> Subject: Using Configuration instead of JobConf
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi,
>
> according to the API-Dokumentation of 0.20.1 JobConf is deprecated and
> we should use Configuration instead. However all examples on the webpage
> still referece JobConf.
>
> Is there a good example for replacing JobConf by Configuration?
>
> Regards, Oliver
>
> - --
> Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
> Tel. +49 30 44793251, Mobil: +49 178 7903538
> Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJK3wNFAAoJELeiwuwqd1DGf4YH/3P83TQh+7L0GiKT4rdIe1HC
> LdZ1weVcqnBKfFZYCsL64PfwPGhkl0hoTU/sU29qGGrnAN8QKbF/8m2P32RlysCV
> ONT2nUlugHfEb8iQzF8Xx3uuCzU3AXYHK0eqme+thzfscxZKiGKzLTYb26qFnufG
> AtTIEKMkv5SRLIn5YRFM/49dcSp2WIRCIEcc+0eofsxFvI9X2a8Sg8AMG5JzNqYT
> SHMGSr3boBc4gFCNTCsRmJQIusPVWyJZICVss0q3491xdMVvGfAPTsbb6BLINFkL
> j5BrbC9OaNnmX3X3kR2/orABwMj9y5Xd1YjXepeWG3QD3WftPVyoCIOE1v8m2s0=
> =OFkK
> -----END PGP SIGNATURE-----
>
RE: Using Configuration instead of JobConf
Posted by Mark Vigeant <ma...@riskmetrics.com>.
Hi Oliver,
I ran into the same problem a few weeks ago. What you want to do now is use the classes Job and Configuration. Here's basically what you need to do:
Configuration conf = new Configuration(); //creates the configuration object
conf.set(TableOutputFormat.OUTPUT_TABLE, name); //set the output format: I'm using this for HBase so it's an HTable.
Job job = new Job(conf, NAME);
job.setJarByClass(TableUploader.class);
job.setInputFormatClass(FileInputFormat.class);
job.setMapperClass(MapExample.class);
job.setNumReduceTasks(0); //alternately, you could set a Reducer class. Check the API
FileInputFormat.setInputPaths(job, new Path(input));
job.setOutputFormatClass(TableOutputFormat.class);
job.waitForCompletion(true); //submits the job, waits for it to be completed.
-----Original Message-----
From: Oliver B. Fischer [mailto:o.b.fischer@swe-blog.net]
Sent: Wednesday, October 21, 2009 8:49 AM
To: common-user@hadoop.apache.org
Subject: Using Configuration instead of JobConf
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
according to the API-Dokumentation of 0.20.1 JobConf is deprecated and
we should use Configuration instead. However all examples on the webpage
still referece JobConf.
Is there a good example for replacing JobConf by Configuration?
Regards, Oliver
- --
Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
Tel. +49 30 44793251, Mobil: +49 178 7903538
Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJK3wNFAAoJELeiwuwqd1DGf4YH/3P83TQh+7L0GiKT4rdIe1HC
LdZ1weVcqnBKfFZYCsL64PfwPGhkl0hoTU/sU29qGGrnAN8QKbF/8m2P32RlysCV
ONT2nUlugHfEb8iQzF8Xx3uuCzU3AXYHK0eqme+thzfscxZKiGKzLTYb26qFnufG
AtTIEKMkv5SRLIn5YRFM/49dcSp2WIRCIEcc+0eofsxFvI9X2a8Sg8AMG5JzNqYT
SHMGSr3boBc4gFCNTCsRmJQIusPVWyJZICVss0q3491xdMVvGfAPTsbb6BLINFkL
j5BrbC9OaNnmX3X3kR2/orABwMj9y5Xd1YjXepeWG3QD3WftPVyoCIOE1v8m2s0=
=OFkK
-----END PGP SIGNATURE-----
Re: Using Configuration instead of JobConf
Posted by "Oliver B. Fischer" <o....@swe-blog.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello Tim,
thank you. I found the examples and even the WordCount has been updated.
Only the documentation is out of date.
Bye,
Oliver
tim robertson schrieb:
> The org.apache.hadoop.examples.SecondarySort in 0.20.1 is an example
> using the org.apache.hadoop.conf.Configuration.
- --
Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
Tel. +49 30 44793251, Mobil: +49 178 7903538
Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJK6RlqAAoJELeiwuwqd1DGBoMIALplC9+t7Pr3hSik4J2Kitpo
BKGxtJvDS+FiNTGjLFC5DSQXyaFc2eTwSRRKPAXKfiOkAyluYXvuy+t6N5IeE+jH
iHIcY2rYIFwcGf2+yc2gmrySVVJwDctQVSJfNAVwk5L3jisCsLCciKEo+2zd6n7v
LpcccaHFRl3SZcTVhzaWdCfE//5GnEEryUDWs9qdCJjIzlvWmFPDdeNvTCi9/22Q
m/80M2ncQXMV3biA+dNARgPTcMajNMVw8Zjb9q2+oZ36nyvTi9DA48wniQKxzHcv
IFrBVHMtDewT39dNrnraU2rlA+/vP1UQ4pOsyU8mA/504v2IA+VF11ymMSGcQ64=
=Fo7q
-----END PGP SIGNATURE-----
Re: Using Configuration instead of JobConf
Posted by tim robertson <ti...@gmail.com>.
The org.apache.hadoop.examples.SecondarySort in 0.20.1 is an example
using the org.apache.hadoop.conf.Configuration.
Cheers,
Tim
On Tue, Oct 27, 2009 at 2:42 PM, Oliver B. Fischer
<o....@swe-blog.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Oliver B. Fischer schrieb:
>> Hi,
>>
>> according to the API-Dokumentation of 0.20.1 JobConf is deprecated and
>> we should use Configuration instead. However all examples on the webpage
>> still referece JobConf.
>>
>> Is there a good example for replacing JobConf by Configuration?
>>
>> Regards, Oliver
>>
>
> Hello, I created HADOOP-6336 for this.
>
>
> - --
> Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
> Tel. +49 30 44793251, Mobil: +49 178 7903538
> Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJK5vjUAAoJELeiwuwqd1DG3m8H/RmFTwvLvHEftRDCfA4q1d5M
> +M4XST8Yw3grBLbrJDCorZXirzvG0PskgWfGs4o7Dvhq2zPcF4uubcuqi8ej/Wxf
> Lo9I+LQdkwuXNdwWMASMODpyPEBtwiQOAT/QUpHD9b7mpzzDEealRJz+GP96yLaH
> PaWgBj5F3d7KuI9AsdsXPBr3WMc+7CAukL/G8Wz4UN4k8Z8hc7/yJq5ISFUB+NrR
> z/NdSVJhSQfwIB44826bmeNgEAIKbZVGFZs4pzuicXG0Q6bs77GX83pcne4r1Mmv
> RWYIzKbMUUDqF7TT1h4BsBE54crf5TabJWwmFqHiH9NaR8fOBYcM8VBlT/XbdyM=
> =MT4v
> -----END PGP SIGNATURE-----
>
Re: Using Configuration instead of JobConf
Posted by "Oliver B. Fischer" <o....@swe-blog.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Oliver B. Fischer schrieb:
> Hi,
>
> according to the API-Dokumentation of 0.20.1 JobConf is deprecated and
> we should use Configuration instead. However all examples on the webpage
> still referece JobConf.
>
> Is there a good example for replacing JobConf by Configuration?
>
> Regards, Oliver
>
Hello, I created HADOOP-6336 for this.
- --
Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
Tel. +49 30 44793251, Mobil: +49 178 7903538
Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJK5vjUAAoJELeiwuwqd1DG3m8H/RmFTwvLvHEftRDCfA4q1d5M
+M4XST8Yw3grBLbrJDCorZXirzvG0PskgWfGs4o7Dvhq2zPcF4uubcuqi8ej/Wxf
Lo9I+LQdkwuXNdwWMASMODpyPEBtwiQOAT/QUpHD9b7mpzzDEealRJz+GP96yLaH
PaWgBj5F3d7KuI9AsdsXPBr3WMc+7CAukL/G8Wz4UN4k8Z8hc7/yJq5ISFUB+NrR
z/NdSVJhSQfwIB44826bmeNgEAIKbZVGFZs4pzuicXG0Q6bs77GX83pcne4r1Mmv
RWYIzKbMUUDqF7TT1h4BsBE54crf5TabJWwmFqHiH9NaR8fOBYcM8VBlT/XbdyM=
=MT4v
-----END PGP SIGNATURE-----