You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Tianyin Xu <ti...@cs.ucsd.edu> on 2014/11/14 21:22:02 UTC
Are these configuration parameters deprecated?
Hi,
I'm very confused by some of the MapReduce configuration parameters
which appear in the latest version of mapred-default.xml.
http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
in code but
/* mapreduce/util/ConfigUtil.java */
55 new DeprecationDelta("mapred.temp.dir",
56 MRConfig.TEMP_DIR),
My interpretation is that it's renamed into "mapred.temp.dir".
However, when I grep the new name, I still cannot find any code except
some testing ones in
./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>From the semantics, this should be a "must-to-have" parameter for MR jobs...
Also, many parameters are like this. So I'm really confused.
Am I missing something? Are these parameters deprecated?
Thanks a lot!
Tianyin
Re: Are these configuration parameters deprecated?
Posted by Tianyin Xu <ti...@cs.ucsd.edu>.
So, these parameters are indeed deprecated?
Then how do I find the replaced one? Let's just say
"mapreduce.task.tmp.dir," how which one should I set to have the same
behavior?
@Niels, yes, exactly, Hadoop in particular has a high churn of
configuration renaming which is pretty annoying. Not even the
deprecated parameters are not complained, but also any invalid
parameters are like this.
~t
On Fri, Nov 14, 2014 at 1:03 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> A while ago I found a similar problem;
> I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
> many other complain at startup about deprecated parameters.
> It turns out that these deprecated names are still in *-default.xml files
> and in various other places in the code base.
>
> Perhaps an issue indicating that the use of the deprecated parameters should
> be removed from the main code base is in order here.
>
> Niels Basjes
>
> On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
>>
>> Hi,
>>
>> I'm very confused by some of the MapReduce configuration parameters
>> which appear in the latest version of mapred-default.xml.
>>
>> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
>> in code but
>>
>> /* mapreduce/util/ConfigUtil.java */
>> 55 new DeprecationDelta("mapred.temp.dir",
>> 56 MRConfig.TEMP_DIR),
>>
>> My interpretation is that it's renamed into "mapred.temp.dir".
>> However, when I grep the new name, I still cannot find any code except
>> some testing ones in
>>
>> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>>
>> From the semantics, this should be a "must-to-have" parameter for MR
>> jobs...
>>
>> Also, many parameters are like this. So I'm really confused.
>>
>> Am I missing something? Are these parameters deprecated?
>>
>> Thanks a lot!
>> Tianyin
>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
--
Tianyin XU,
http://cseweb.ucsd.edu/~tixu/
Re: Are these configuration parameters deprecated?
Posted by Tianyin Xu <ti...@cs.ucsd.edu>.
So, these parameters are indeed deprecated?
Then how do I find the replaced one? Let's just say
"mapreduce.task.tmp.dir," how which one should I set to have the same
behavior?
@Niels, yes, exactly, Hadoop in particular has a high churn of
configuration renaming which is pretty annoying. Not even the
deprecated parameters are not complained, but also any invalid
parameters are like this.
~t
On Fri, Nov 14, 2014 at 1:03 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> A while ago I found a similar problem;
> I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
> many other complain at startup about deprecated parameters.
> It turns out that these deprecated names are still in *-default.xml files
> and in various other places in the code base.
>
> Perhaps an issue indicating that the use of the deprecated parameters should
> be removed from the main code base is in order here.
>
> Niels Basjes
>
> On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
>>
>> Hi,
>>
>> I'm very confused by some of the MapReduce configuration parameters
>> which appear in the latest version of mapred-default.xml.
>>
>> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
>> in code but
>>
>> /* mapreduce/util/ConfigUtil.java */
>> 55 new DeprecationDelta("mapred.temp.dir",
>> 56 MRConfig.TEMP_DIR),
>>
>> My interpretation is that it's renamed into "mapred.temp.dir".
>> However, when I grep the new name, I still cannot find any code except
>> some testing ones in
>>
>> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>>
>> From the semantics, this should be a "must-to-have" parameter for MR
>> jobs...
>>
>> Also, many parameters are like this. So I'm really confused.
>>
>> Am I missing something? Are these parameters deprecated?
>>
>> Thanks a lot!
>> Tianyin
>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
--
Tianyin XU,
http://cseweb.ucsd.edu/~tixu/
Re: Are these configuration parameters deprecated?
Posted by Tianyin Xu <ti...@cs.ucsd.edu>.
So, these parameters are indeed deprecated?
Then how do I find the replaced one? Let's just say
"mapreduce.task.tmp.dir," how which one should I set to have the same
behavior?
@Niels, yes, exactly, Hadoop in particular has a high churn of
configuration renaming which is pretty annoying. Not even the
deprecated parameters are not complained, but also any invalid
parameters are like this.
~t
On Fri, Nov 14, 2014 at 1:03 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> A while ago I found a similar problem;
> I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
> many other complain at startup about deprecated parameters.
> It turns out that these deprecated names are still in *-default.xml files
> and in various other places in the code base.
>
> Perhaps an issue indicating that the use of the deprecated parameters should
> be removed from the main code base is in order here.
>
> Niels Basjes
>
> On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
>>
>> Hi,
>>
>> I'm very confused by some of the MapReduce configuration parameters
>> which appear in the latest version of mapred-default.xml.
>>
>> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
>> in code but
>>
>> /* mapreduce/util/ConfigUtil.java */
>> 55 new DeprecationDelta("mapred.temp.dir",
>> 56 MRConfig.TEMP_DIR),
>>
>> My interpretation is that it's renamed into "mapred.temp.dir".
>> However, when I grep the new name, I still cannot find any code except
>> some testing ones in
>>
>> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>>
>> From the semantics, this should be a "must-to-have" parameter for MR
>> jobs...
>>
>> Also, many parameters are like this. So I'm really confused.
>>
>> Am I missing something? Are these parameters deprecated?
>>
>> Thanks a lot!
>> Tianyin
>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
--
Tianyin XU,
http://cseweb.ucsd.edu/~tixu/
Re: Are these configuration parameters deprecated?
Posted by Tianyin Xu <ti...@cs.ucsd.edu>.
So, these parameters are indeed deprecated?
Then how do I find the replaced one? Let's just say
"mapreduce.task.tmp.dir," how which one should I set to have the same
behavior?
@Niels, yes, exactly, Hadoop in particular has a high churn of
configuration renaming which is pretty annoying. Not even the
deprecated parameters are not complained, but also any invalid
parameters are like this.
~t
On Fri, Nov 14, 2014 at 1:03 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> A while ago I found a similar problem;
> I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
> many other complain at startup about deprecated parameters.
> It turns out that these deprecated names are still in *-default.xml files
> and in various other places in the code base.
>
> Perhaps an issue indicating that the use of the deprecated parameters should
> be removed from the main code base is in order here.
>
> Niels Basjes
>
> On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
>>
>> Hi,
>>
>> I'm very confused by some of the MapReduce configuration parameters
>> which appear in the latest version of mapred-default.xml.
>>
>> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
>> in code but
>>
>> /* mapreduce/util/ConfigUtil.java */
>> 55 new DeprecationDelta("mapred.temp.dir",
>> 56 MRConfig.TEMP_DIR),
>>
>> My interpretation is that it's renamed into "mapred.temp.dir".
>> However, when I grep the new name, I still cannot find any code except
>> some testing ones in
>>
>> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>>
>> From the semantics, this should be a "must-to-have" parameter for MR
>> jobs...
>>
>> Also, many parameters are like this. So I'm really confused.
>>
>> Am I missing something? Are these parameters deprecated?
>>
>> Thanks a lot!
>> Tianyin
>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
--
Tianyin XU,
http://cseweb.ucsd.edu/~tixu/
Re: Are these configuration parameters deprecated?
Posted by Niels Basjes <Ni...@basjes.nl>.
A while ago I found a similar problem;
I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
many other complain at startup about deprecated parameters.
It turns out that these deprecated names are still in *-default.xml files
and in various other places in the code base.
Perhaps an issue indicating that the use of the deprecated parameters
should be removed from the main code base is in order here.
Niels Basjes
On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
> Hi,
>
> I'm very confused by some of the MapReduce configuration parameters
> which appear in the latest version of mapred-default.xml.
>
> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
> in code but
>
> /* mapreduce/util/ConfigUtil.java */
> 55 new DeprecationDelta("mapred.temp.dir",
> 56 MRConfig.TEMP_DIR),
>
> My interpretation is that it's renamed into "mapred.temp.dir".
> However, when I grep the new name, I still cannot find any code except
> some testing ones in
>
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>
> From the semantics, this should be a "must-to-have" parameter for MR
> jobs...
>
> Also, many parameters are like this. So I'm really confused.
>
> Am I missing something? Are these parameters deprecated?
>
> Thanks a lot!
> Tianyin
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Are these configuration parameters deprecated?
Posted by Niels Basjes <Ni...@basjes.nl>.
A while ago I found a similar problem;
I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
many other complain at startup about deprecated parameters.
It turns out that these deprecated names are still in *-default.xml files
and in various other places in the code base.
Perhaps an issue indicating that the use of the deprecated parameters
should be removed from the main code base is in order here.
Niels Basjes
On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
> Hi,
>
> I'm very confused by some of the MapReduce configuration parameters
> which appear in the latest version of mapred-default.xml.
>
> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
> in code but
>
> /* mapreduce/util/ConfigUtil.java */
> 55 new DeprecationDelta("mapred.temp.dir",
> 56 MRConfig.TEMP_DIR),
>
> My interpretation is that it's renamed into "mapred.temp.dir".
> However, when I grep the new name, I still cannot find any code except
> some testing ones in
>
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>
> From the semantics, this should be a "must-to-have" parameter for MR
> jobs...
>
> Also, many parameters are like this. So I'm really confused.
>
> Am I missing something? Are these parameters deprecated?
>
> Thanks a lot!
> Tianyin
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Are these configuration parameters deprecated?
Posted by Niels Basjes <Ni...@basjes.nl>.
A while ago I found a similar problem;
I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
many other complain at startup about deprecated parameters.
It turns out that these deprecated names are still in *-default.xml files
and in various other places in the code base.
Perhaps an issue indicating that the use of the deprecated parameters
should be removed from the main code base is in order here.
Niels Basjes
On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
> Hi,
>
> I'm very confused by some of the MapReduce configuration parameters
> which appear in the latest version of mapred-default.xml.
>
> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
> in code but
>
> /* mapreduce/util/ConfigUtil.java */
> 55 new DeprecationDelta("mapred.temp.dir",
> 56 MRConfig.TEMP_DIR),
>
> My interpretation is that it's renamed into "mapred.temp.dir".
> However, when I grep the new name, I still cannot find any code except
> some testing ones in
>
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>
> From the semantics, this should be a "must-to-have" parameter for MR
> jobs...
>
> Also, many parameters are like this. So I'm really confused.
>
> Am I missing something? Are these parameters deprecated?
>
> Thanks a lot!
> Tianyin
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Are these configuration parameters deprecated?
Posted by Niels Basjes <Ni...@basjes.nl>.
A while ago I found a similar problem;
I was wondering why do a lot of tools like the hdfs, hbase shell, pig and
many other complain at startup about deprecated parameters.
It turns out that these deprecated names are still in *-default.xml files
and in various other places in the code base.
Perhaps an issue indicating that the use of the deprecated parameters
should be removed from the main code base is in order here.
Niels Basjes
On Fri, Nov 14, 2014 at 9:22 PM, Tianyin Xu <ti...@cs.ucsd.edu> wrote:
> Hi,
>
> I'm very confused by some of the MapReduce configuration parameters
> which appear in the latest version of mapred-default.xml.
>
> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> Take "mapreduce.task.tmp.dir" as an example, I fail to find its usage
> in code but
>
> /* mapreduce/util/ConfigUtil.java */
> 55 new DeprecationDelta("mapred.temp.dir",
> 56 MRConfig.TEMP_DIR),
>
> My interpretation is that it's renamed into "mapred.temp.dir".
> However, when I grep the new name, I still cannot find any code except
> some testing ones in
>
> ./hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
>
> From the semantics, this should be a "must-to-have" parameter for MR
> jobs...
>
> Also, many parameters are like this. So I'm really confused.
>
> Am I missing something? Are these parameters deprecated?
>
> Thanks a lot!
> Tianyin
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes