You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Allen Wittenauer <aw...@effectivemachines.com> on 2016/09/07 13:44:56 UTC
[VOTE] Merge HADOOP-13341
I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
Here's a quick summary:
* makes the rules around _OPTS consistent across all the projects
* makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
* with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
* removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
* makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
* by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
[RESULTS] Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
The vote passes with 3 +1 binding votes.
I'll be merging this later today.
Thanks everyone!
> On Sep 7, 2016, at 6:44 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Chris Douglas <ch...@gmail.com>.
+1 (also on JIRA) -C
On Wed, Sep 7, 2016 at 6:44 AM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
Absolutely, that is a great output, very clear and provides a very good user experience.
Thanks
Anu
On 9/9/16, 3:06 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>>
>> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
>
> Thanks!
>
> There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
>
>> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
>
>
> That's a great suggestion. Would it be better if the usage output was more like:
>
>---snip---
>Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
>
> OPTIONS is none or any of:
>
>--buildpaths attempt to add class files from build tree
>--config dir Hadoop config directory
>--daemon (start|status|stop) operate on a daemon
>--debug turn on shell script debug mode
>--help usage information
>--hostnames list[,of,host,names] hosts to use in worker mode
>--hosts filename list of hosts to use in worker mode
>--loglevel level set the log4j level for this command
>--workers turn on worker mode
>
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
>
> We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
Absolutely, that is a great output, very clear and provides a very good user experience.
Thanks
Anu
On 9/9/16, 3:06 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>>
>> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
>
> Thanks!
>
> There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
>
>> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
>
>
> That's a great suggestion. Would it be better if the usage output was more like:
>
>---snip---
>Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
>
> OPTIONS is none or any of:
>
>--buildpaths attempt to add class files from build tree
>--config dir Hadoop config directory
>--daemon (start|status|stop) operate on a daemon
>--debug turn on shell script debug mode
>--help usage information
>--hostnames list[,of,host,names] hosts to use in worker mode
>--hosts filename list of hosts to use in worker mode
>--loglevel level set the log4j level for this command
>--workers turn on worker mode
>
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
>
> We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
Absolutely, that is a great output, very clear and provides a very good user experience.
Thanks
Anu
On 9/9/16, 3:06 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>>
>> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
>
> Thanks!
>
> There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
>
>> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
>
>
> That's a great suggestion. Would it be better if the usage output was more like:
>
>---snip---
>Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
>
> OPTIONS is none or any of:
>
>--buildpaths attempt to add class files from build tree
>--config dir Hadoop config directory
>--daemon (start|status|stop) operate on a daemon
>--debug turn on shell script debug mode
>--help usage information
>--hostnames list[,of,host,names] hosts to use in worker mode
>--hosts filename list of hosts to use in worker mode
>--loglevel level set the log4j level for this command
>--workers turn on worker mode
>
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
>
> We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
Absolutely, that is a great output, very clear and provides a very good user experience.
Thanks
Anu
On 9/9/16, 3:06 PM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
>> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>>
>> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
>
> Thanks!
>
> There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
>
>> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
>
>
> That's a great suggestion. Would it be better if the usage output was more like:
>
>---snip---
>Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
>
> OPTIONS is none or any of:
>
>--buildpaths attempt to add class files from build tree
>--config dir Hadoop config directory
>--daemon (start|status|stop) operate on a daemon
>--debug turn on shell script debug mode
>--help usage information
>--hostnames list[,of,host,names] hosts to use in worker mode
>--hosts filename list of hosts to use in worker mode
>--loglevel level set the log4j level for this command
>--workers turn on worker mode
>
> SUBCOMMAND is one of:
>
>
>Clients:
> cacheadmin configure the HDFS cache
> classpath prints the class path needed to get the hadoop jar and the required libraries
> crypto configure HDFS encryption zones
> ...
>
>Daemons:
> balancer run a cluster balancing utility
> datanode run a DFS datanode
> namenode run the DFS name node
>...
>---snip---
>
> We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>
> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
Thanks!
There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
That's a great suggestion. Would it be better if the usage output was more like:
---snip---
Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
OPTIONS is none or any of:
--buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--daemon (start|status|stop) operate on a daemon
--debug turn on shell script debug mode
--help usage information
--hostnames list[,of,host,names] hosts to use in worker mode
--hosts filename list of hosts to use in worker mode
--loglevel level set the log4j level for this command
--workers turn on worker mode
SUBCOMMAND is one of:
Clients:
cacheadmin configure the HDFS cache
classpath prints the class path needed to get the hadoop jar and the required libraries
crypto configure HDFS encryption zones
...
Daemons:
balancer run a cluster balancing utility
datanode run a DFS datanode
namenode run the DFS name node
...
---snip---
We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>
> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
Thanks!
There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
That's a great suggestion. Would it be better if the usage output was more like:
---snip---
Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
OPTIONS is none or any of:
--buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--daemon (start|status|stop) operate on a daemon
--debug turn on shell script debug mode
--help usage information
--hostnames list[,of,host,names] hosts to use in worker mode
--hosts filename list of hosts to use in worker mode
--loglevel level set the log4j level for this command
--workers turn on worker mode
SUBCOMMAND is one of:
Clients:
cacheadmin configure the HDFS cache
classpath prints the class path needed to get the hadoop jar and the required libraries
crypto configure HDFS encryption zones
...
Daemons:
balancer run a cluster balancing utility
datanode run a DFS datanode
namenode run the DFS name node
...
---snip---
We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>
> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
Thanks!
There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
That's a great suggestion. Would it be better if the usage output was more like:
---snip---
Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
OPTIONS is none or any of:
--buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--daemon (start|status|stop) operate on a daemon
--debug turn on shell script debug mode
--help usage information
--hostnames list[,of,host,names] hosts to use in worker mode
--hosts filename list of hosts to use in worker mode
--loglevel level set the log4j level for this command
--workers turn on worker mode
SUBCOMMAND is one of:
Clients:
cacheadmin configure the HDFS cache
classpath prints the class path needed to get the hadoop jar and the required libraries
crypto configure HDFS encryption zones
...
Daemons:
balancer run a cluster balancing utility
datanode run a DFS datanode
namenode run the DFS name node
...
---snip---
We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 9, 2016, at 2:15 PM, Anu Engineer <ae...@hortonworks.com> wrote:
>
> +1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
Thanks!
There's still a handful of HDFS and MAPRED vars that begin with HADOOP, but those should be trivial to knock out after a pattern has been established.
> I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
That's a great suggestion. Would it be better if the usage output was more like:
---snip---
Usage: hdfs [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
OPTIONS is none or any of:
--buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--daemon (start|status|stop) operate on a daemon
--debug turn on shell script debug mode
--help usage information
--hostnames list[,of,host,names] hosts to use in worker mode
--hosts filename list of hosts to use in worker mode
--loglevel level set the log4j level for this command
--workers turn on worker mode
SUBCOMMAND is one of:
Clients:
cacheadmin configure the HDFS cache
classpath prints the class path needed to get the hadoop jar and the required libraries
crypto configure HDFS encryption zones
...
Daemons:
balancer run a cluster balancing utility
datanode run a DFS datanode
namenode run the DFS name node
...
---snip---
We do something similar in Apache Yetus and shouldn't be too hard to do in Apache Hadoop. We couldn't read SUPPORTDAEMONIZATION to place things, but as long as people put their new commands in the correct section in hadoop_usage, it should work.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
+1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
On 9/7/16, 6:44 AM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
>* makes the rules around _OPTS consistent across all the projects
>* makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
>* with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
>* removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
>* makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
>* by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
[RESULTS] Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
The vote passes with 3 +1 binding votes.
I'll be merging this later today.
Thanks everyone!
> On Sep 7, 2016, at 6:44 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 8 Sep 2016, at 16:08, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
>
> Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
>
>> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
>
> Sure. https://github.com/apache/hadoop/pull/126 has been opened.
>
> Thanks!
LGTM: +1. Added the vote on the JIRA too
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 8 Sep 2016, at 16:08, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
>
> Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
>
>> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
>
> Sure. https://github.com/apache/hadoop/pull/126 has been opened.
>
> Thanks!
LGTM: +1. Added the vote on the JIRA too
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 8 Sep 2016, at 16:08, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
>
> Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
>
>> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
>
> Sure. https://github.com/apache/hadoop/pull/126 has been opened.
>
> Thanks!
LGTM: +1. Added the vote on the JIRA too
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 8 Sep 2016, at 16:08, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
>
> Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
>
>> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
>
> Sure. https://github.com/apache/hadoop/pull/126 has been opened.
>
> Thanks!
LGTM: +1. Added the vote on the JIRA too
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Sure. https://github.com/apache/hadoop/pull/126 has been opened.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Sure. https://github.com/apache/hadoop/pull/126 has been opened.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Sure. https://github.com/apache/hadoop/pull/126 has been opened.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 8, 2016, at 2:50 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
Just a heads up that I'm using bash variable references. While not exactly rare, they are uncommon. [We use them in lots of places in the shell code already, so no new ground being broken.]
> Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Sure. https://github.com/apache/hadoop/pull/126 has been opened.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Sep 2016, at 14:44, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
I can see the branch at https://github.com/apache/hadoop/tree/HADOOP-13341
And the patch at https://github.com/apache/hadoop/blob/HADOOP-13341/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
But...the latter only does the _USER stuff, I don't see it ouching the _OPTS stuff.
Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Like you said, one thing that will change is that env var going from HADOOP_namenode_USER to HDFS_NAMENODE_USER . Like you say, incompatible with alpha-1
welll, we've a track record of breaking compatibility from alphas, so I'm OK with that,
-Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
+1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
On 9/7/16, 6:44 AM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
>* makes the rules around _OPTS consistent across all the projects
>* makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
>* with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
>* removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
>* makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
>* by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
+1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
On 9/7/16, 6:44 AM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
>* makes the rules around _OPTS consistent across all the projects
>* makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
>* with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
>* removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
>* makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
>* by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Sep 2016, at 14:44, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
I can see the branch at https://github.com/apache/hadoop/tree/HADOOP-13341
And the patch at https://github.com/apache/hadoop/blob/HADOOP-13341/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
But...the latter only does the _USER stuff, I don't see it ouching the _OPTS stuff.
Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Like you said, one thing that will change is that env var going from HADOOP_namenode_USER to HDFS_NAMENODE_USER . Like you say, incompatible with alpha-1
welll, we've a track record of breaking compatibility from alphas, so I'm OK with that,
-Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Sep 2016, at 14:44, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
I can see the branch at https://github.com/apache/hadoop/tree/HADOOP-13341
And the patch at https://github.com/apache/hadoop/blob/HADOOP-13341/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
But...the latter only does the _USER stuff, I don't see it ouching the _OPTS stuff.
Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Like you said, one thing that will change is that env var going from HADOOP_namenode_USER to HDFS_NAMENODE_USER . Like you say, incompatible with alpha-1
welll, we've a track record of breaking compatibility from alphas, so I'm OK with that,
-Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Sep 2016, at 14:44, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
I'm trying to do the review effort here even though I don't know detailed bash, as I expect I don't know any less than others, and what better way to learn than reviewing code written by people that do know bash?
I can see the branch at https://github.com/apache/hadoop/tree/HADOOP-13341
And the patch at https://github.com/apache/hadoop/blob/HADOOP-13341/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
But...the latter only does the _USER stuff, I don't see it ouching the _OPTS stuff.
Could you submit a PR of that HADOOP-13341 branch, so I can review it there.
Like you said, one thing that will change is that env var going from HADOOP_namenode_USER to HDFS_NAMENODE_USER . Like you say, incompatible with alpha-1
welll, we've a track record of breaking compatibility from alphas, so I'm OK with that,
-Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Anu Engineer <ae...@hortonworks.com>.
+1, Thanks for the effort. It brings in a world of consistency to the hadoop vars; and as usual reading your bash code was very educative.
I had a minor suggestion though. since we have classified the _OPTS to client and daemon opts, for new people it is hard to know which of these subcommands are daemon vs. a client command. Maybe we can add a special char in the help message to indicate which are daemons or just document it? Only way I know right now is to look the appropriate script and see if HADOOP_SUBCMD_SUPPORTDAEMONIZATION is set to true.
On 9/7/16, 6:44 AM, "Allen Wittenauer" <aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
>* makes the rules around _OPTS consistent across all the projects
>* makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
>* with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
>* removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
>* makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
>* by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge HADOOP-13341
Posted by Chris Douglas <ch...@gmail.com>.
+1 (also on JIRA) -C
On Wed, Sep 7, 2016 at 6:44 AM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
[RESULTS] Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
The vote passes with 3 +1 binding votes.
I'll be merging this later today.
Thanks everyone!
> On Sep 7, 2016, at 6:44 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
[RESULTS] Re: [VOTE] Merge HADOOP-13341
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
The vote passes with 3 +1 binding votes.
I'll be merging this later today.
Thanks everyone!
> On Sep 7, 2016, at 6:44 AM, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Chris Douglas <ch...@gmail.com>.
+1 (also on JIRA) -C
On Wed, Sep 7, 2016 at 6:44 AM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge HADOOP-13341
Posted by Chris Douglas <ch...@gmail.com>.
+1 (also on JIRA) -C
On Wed, Sep 7, 2016 at 6:44 AM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
>
> I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was developed exclusively by me. As usual with large shell script changes, it's been broken up into several smaller commits to make it easier to read. The core of the functionality is almost entirely in hadoop-functions.sh with the majority of the rest of the new additions either being documentation or test code. In addition, large swaths of code is removed from the hadoop, hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better documentation around this feature's existence. Note that this is an incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good chunk of the "directory name containing spaces blows up the bash code" problems that's been around since the beginning of the project
>
> Thanks!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org