You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Allen Wittenauer <aw...@apache.org> on 2016/05/09 20:26:22 UTC
[VOTE] Merge feature branch HADOOP-12930
Hey gang!
I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch was developed exclusively by me as per the discussion two months ago as a way to make what would be a rather large patch hopefully easier to review. The vast majority of the branch is code movement in the same file, additional license headers, maven assembly hooks for distribution, and variable renames. Not a whole lot of new code, but a big diff file none-the-less.
This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ commands to allow for subcommands to be added or modified at runtime. This allows for individual users or entire sites to tweak the execution environment to suit their local needs. For example, it has been a practice for some locations to change the distcp jar out for a custom one. Using this functionality, it is possible that the ‘hadoop distcp’ command could run the local version without overwriting the bundled jar and for existing documentation (read: results from Internet searches) to work as written without modification. This has the potential to be a huge win, especially for:
* advanced end users looking to supplement the Apache Hadoop experience
* operations teams that may be able to leverage existing documentation without having to remain local “exception” docs
* development groups wanting an easy way to trial experimental features
Additionally, this branch includes the following, related changes:
* Adds the first unit tests for the ‘hadoop’ command
* Adds the infrastructure for hdfs script testing and the first unit test for the ‘hdfs’ command
* Modifies the hadoop-tools components to be dynamic rather than hard coded
* Renames the shell profiles for hdfs, mapred, and yarn to be consistent with other bundled profiles, including the ones introduced in this branch
Documentation, including a ‘hello world’-style example, is in the UnixShellGuide markdown file. (Of course!)
I am at ApacheCon this week if anyone wants to discuss in-depth.
Thanks!
P.S.,
There are still two open sub-tasks. These are blocked by other issues so that we may add unit testing to the shell code in those respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
[RESULT] Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
Vote passes.
1 = +1 non-binding
4 = +1 binding
I’ll squash and commit here in a sec.
Thanks everyone!
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> Fix (with unit tests!) committed.
Thanks, Allen.
+1.
- checked out HADOOP-12930 and ran shelltest on my local
- ran 'mvn package -Pdist -Pnative -DskipTests'
- started pseudo-distributed cluster by start-dfs.sh and start-yarn.sh
- ran 'hadoop distcp' and 'mapred streaming' successfully
- created etc/hadoop/shellprofile.d/test.sh and added subcommands
described in UnixShellGuide.md
- tested overriding built-in subcommands
Masatake Iwasaki
On 5/16/16 00:30, Allen Wittenauer wrote:
>> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>>
>>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>>
>>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}\u201d
>> Easy fix. The $* should just be *. I\u2019ll open an issue and fix it here in a sec.
>>
> Fix (with unit tests!) committed.
>
> Thanks
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> Fix (with unit tests!) committed.
Thanks, Allen.
+1.
- checked out HADOOP-12930 and ran shelltest on my local
- ran 'mvn package -Pdist -Pnative -DskipTests'
- started pseudo-distributed cluster by start-dfs.sh and start-yarn.sh
- ran 'hadoop distcp' and 'mapred streaming' successfully
- created etc/hadoop/shellprofile.d/test.sh and added subcommands
described in UnixShellGuide.md
- tested overriding built-in subcommands
Masatake Iwasaki
On 5/16/16 00:30, Allen Wittenauer wrote:
>> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>>
>>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>>
>>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}\u201d
>> Easy fix. The $* should just be *. I\u2019ll open an issue and fix it here in a sec.
>>
> Fix (with unit tests!) committed.
>
> Thanks
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> Fix (with unit tests!) committed.
Thanks, Allen.
+1.
- checked out HADOOP-12930 and ran shelltest on my local
- ran 'mvn package -Pdist -Pnative -DskipTests'
- started pseudo-distributed cluster by start-dfs.sh and start-yarn.sh
- ran 'hadoop distcp' and 'mapred streaming' successfully
- created etc/hadoop/shellprofile.d/test.sh and added subcommands
described in UnixShellGuide.md
- tested overriding built-in subcommands
Masatake Iwasaki
On 5/16/16 00:30, Allen Wittenauer wrote:
>> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>>
>>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>>
>>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}\u201d
>> Easy fix. The $* should just be *. I\u2019ll open an issue and fix it here in a sec.
>>
> Fix (with unit tests!) committed.
>
> Thanks
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
> Fix (with unit tests!) committed.
Thanks, Allen.
+1.
- checked out HADOOP-12930 and ran shelltest on my local
- ran 'mvn package -Pdist -Pnative -DskipTests'
- started pseudo-distributed cluster by start-dfs.sh and start-yarn.sh
- ran 'hadoop distcp' and 'mapred streaming' successfully
- created etc/hadoop/shellprofile.d/test.sh and added subcommands
described in UnixShellGuide.md
- tested overriding built-in subcommands
Masatake Iwasaki
On 5/16/16 00:30, Allen Wittenauer wrote:
>> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>>
>>
>>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>>
>>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}\u201d
>> Easy fix. The $* should just be *. I\u2019ll open an issue and fix it here in a sec.
>>
> Fix (with unit tests!) committed.
>
> Thanks
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>
>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
>
> Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
>
Fix (with unit tests!) committed.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>
>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
>
> Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
>
Fix (with unit tests!) committed.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>
>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
>
> Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
>
Fix (with unit tests!) committed.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 15, 2016, at 7:27 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
>
>> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>>
>> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
>
> Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
>
Fix (with unit tests!) committed.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 9:04 PM, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> + hadoop_debug "Calling dynamically: hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}”
Easy fix. The $* should just be *. I’ll open an issue and fix it here in a sec.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Allen,
I had a problem when I was testing HADOOP-12930 branch. (built by mvn
package -Pdist)
Single argument is ok.::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo
foo
Error on multiple arguments. ::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo bar
bin/hadoop: line 200: foo bar: syntax error in expression (error
token is "bar")
Error: Could not find or load main class
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop distcp /foo /bar
bin/hadoop: line 200: /foo /bar: syntax error: operand expected
(error token is "/foo /bar")
Error: Could not find or load main class
The fix of HADOOP-13120 seems to be related.::
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
index 61fdc2e..0f6982b 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -197,6 +197,7 @@ shift
HADOOP_SUBCMD_ARGS=("$@")
if declare -f hadoop_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
+ hadoop_debug "Calling dynamically:
hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}"
"hadoop_subcommand_${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
else
hadoopcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
Thanks,
Masatake Iwasaki
On 5/15/16 02:33, Allen Wittenauer wrote:
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Allen,
I had a problem when I was testing HADOOP-12930 branch. (built by mvn
package -Pdist)
Single argument is ok.::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo
foo
Error on multiple arguments. ::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo bar
bin/hadoop: line 200: foo bar: syntax error in expression (error
token is "bar")
Error: Could not find or load main class
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop distcp /foo /bar
bin/hadoop: line 200: /foo /bar: syntax error: operand expected
(error token is "/foo /bar")
Error: Could not find or load main class
The fix of HADOOP-13120 seems to be related.::
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
index 61fdc2e..0f6982b 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -197,6 +197,7 @@ shift
HADOOP_SUBCMD_ARGS=("$@")
if declare -f hadoop_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
+ hadoop_debug "Calling dynamically:
hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}"
"hadoop_subcommand_${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
else
hadoopcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
Thanks,
Masatake Iwasaki
On 5/15/16 02:33, Allen Wittenauer wrote:
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
+1.
- Checked out HADOOP-12930
- Built by "mvn package -Pdist -DskipTests -Dtar" successfully
- Added custom subcommands to ~/.hadooprc and confirmed they worked
- Built documentation and it looks good
Thanks,
Akira
On 5/15/16 02:33, Allen Wittenauer wrote:
>
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>>
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>>
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>>
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>>
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>>
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>>
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>>
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Allen,
I had a problem when I was testing HADOOP-12930 branch. (built by mvn
package -Pdist)
Single argument is ok.::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo
foo
Error on multiple arguments. ::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo bar
bin/hadoop: line 200: foo bar: syntax error in expression (error
token is "bar")
Error: Could not find or load main class
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop distcp /foo /bar
bin/hadoop: line 200: /foo /bar: syntax error: operand expected
(error token is "/foo /bar")
Error: Could not find or load main class
The fix of HADOOP-13120 seems to be related.::
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
index 61fdc2e..0f6982b 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -197,6 +197,7 @@ shift
HADOOP_SUBCMD_ARGS=("$@")
if declare -f hadoop_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
+ hadoop_debug "Calling dynamically:
hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}"
"hadoop_subcommand_${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
else
hadoopcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
Thanks,
Masatake Iwasaki
On 5/15/16 02:33, Allen Wittenauer wrote:
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
+1.
- Checked out HADOOP-12930
- Built by "mvn package -Pdist -DskipTests -Dtar" successfully
- Added custom subcommands to ~/.hadooprc and confirmed they worked
- Built documentation and it looks good
Thanks,
Akira
On 5/15/16 02:33, Allen Wittenauer wrote:
>
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>>
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>>
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>>
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>>
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>>
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>>
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>>
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
+1.
- Checked out HADOOP-12930
- Built by "mvn package -Pdist -DskipTests -Dtar" successfully
- Added custom subcommands to ~/.hadooprc and confirmed they worked
- Built documentation and it looks good
Thanks,
Akira
On 5/15/16 02:33, Allen Wittenauer wrote:
>
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>>
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>>
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>>
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>>
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>>
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>>
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>>
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Hi Allen,
I had a problem when I was testing HADOOP-12930 branch. (built by mvn
package -Pdist)
Single argument is ok.::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo
foo
Error on multiple arguments. ::
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop hello foo bar
bin/hadoop: line 200: foo bar: syntax error in expression (error
token is "bar")
Error: Could not find or load main class
[iwasakims@centos7 HADOOP-12930]$ bin/hadoop distcp /foo /bar
bin/hadoop: line 200: /foo /bar: syntax error: operand expected
(error token is "/foo /bar")
Error: Could not find or load main class
The fix of HADOOP-13120 seems to be related.::
diff --git a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
index 61fdc2e..0f6982b 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop
@@ -197,6 +197,7 @@ shift
HADOOP_SUBCMD_ARGS=("$@")
if declare -f hadoop_subcommand_"${HADOOP_SUBCMD}" >/dev/null 2>&1; then
+ hadoop_debug "Calling dynamically:
hadoop_subcommand_${HADOOP_SUBCMD} ${HADOOP_SUBCMD_ARGS[$*]}"
"hadoop_subcommand_${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
else
hadoopcmd_case "${HADOOP_SUBCMD}" "${HADOOP_SUBCMD_ARGS[@]}"
Thanks,
Masatake Iwasaki
On 5/15/16 02:33, Allen Wittenauer wrote:
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
+1.
- Checked out HADOOP-12930
- Built by "mvn package -Pdist -DskipTests -Dtar" successfully
- Added custom subcommands to ~/.hadooprc and confirmed they worked
- Built documentation and it looks good
Thanks,
Akira
On 5/15/16 02:33, Allen Wittenauer wrote:
>
> This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project\u2026. it\u2019d be great if some others could take a look.
>
> Thanks!
>
>> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>> +1. I looked at the patches on the branch, wasn't too bad to review. As
>> Allen said, there's some code movement, assorted other nice doc and shell
>> fixups.
>>
>> Found one extra typo, which I added to HADOOP-13129.
>>
>> Best,
>> Andrew
>>
>> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> +1 (non-binding)
>>>
>>> reviewed everything, filed an additional subtask for a very trivial
>>> typo in the docs. should be fine to make a full issue after close and
>>> then fix.
>>>
>>> tried merging locally, tried running through new shell tests (both
>>> with and without bats installed), tried making an example custom
>>> command (valid and malformed). everything looks great.
>>>
>>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>>
>>>> Hey gang!
>>>>
>>>> I\u2019d like to call a vote to run for 7 days (ending May 16 at
>>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>>> was developed exclusively by me as per the discussion two months ago as a
>>> way to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>>
>>>> This branch modifies the \u2018hadoop\u2019, \u2018hdfs\u2019, \u2018mapred\u2019, and \u2018yarn\u2019
>>> commands to allow for subcommands to be added or modified at runtime. This
>>> allows for individual users or entire sites to tweak the execution
>>> environment to suit their local needs. For example, it has been a practice
>>> for some locations to change the distcp jar out for a custom one. Using
>>> this functionality, it is possible that the \u2018hadoop distcp\u2019 command could
>>> run the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win, especially
>>> for:
>>>>
>>>> * advanced end users looking to supplement the Apache
>>> Hadoop experience
>>>> * operations teams that may be able to leverage existing
>>> documentation without having to remain local \u201cexception\u201d docs
>>>> * development groups wanting an easy way to trial
>>> experimental features
>>>>
>>>> Additionally, this branch includes the following, related
>>> changes:
>>>>
>>>> * Adds the first unit tests for the \u2018hadoop\u2019 command
>>>> * Adds the infrastructure for hdfs script testing and
>>> the first unit test for the \u2018hdfs\u2019 command
>>>> * Modifies the hadoop-tools components to be dynamic
>>> rather than hard coded
>>>> * Renames the shell profiles for hdfs, mapred, and yarn
>>> to be consistent with other bundled profiles, including the ones introduced
>>> in this branch
>>>>
>>>> Documentation, including a \u2018hello world\u2019-style example, is in
>>> the UnixShellGuide markdown file. (Of course!)
>>>>
>>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>>
>>>> Thanks!
>>>>
>>>> P.S.,
>>>>
>>>> There are still two open sub-tasks. These are blocked by other
>>> issues so that we may add unit testing to the shell code in those
>>> respective areas. I\u2019ll covert to full issues after HADOOP-12930 is closed.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>> busbey
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project…. it’d be great if some others could take a look.
Thanks!
> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> +1. I looked at the patches on the branch, wasn't too bad to review. As
> Allen said, there's some code movement, assorted other nice doc and shell
> fixups.
>
> Found one extra typo, which I added to HADOOP-13129.
>
> Best,
> Andrew
>
> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> +1 (non-binding)
>>
>> reviewed everything, filed an additional subtask for a very trivial
>> typo in the docs. should be fine to make a full issue after close and
>> then fix.
>>
>> tried merging locally, tried running through new shell tests (both
>> with and without bats installed), tried making an example custom
>> command (valid and malformed). everything looks great.
>>
>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>
>>> Hey gang!
>>>
>>> I’d like to call a vote to run for 7 days (ending May 16 at
>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>> was developed exclusively by me as per the discussion two months ago as a
>> way to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>>
>>> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
>> commands to allow for subcommands to be added or modified at runtime. This
>> allows for individual users or entire sites to tweak the execution
>> environment to suit their local needs. For example, it has been a practice
>> for some locations to change the distcp jar out for a custom one. Using
>> this functionality, it is possible that the ‘hadoop distcp’ command could
>> run the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>>
>>> * advanced end users looking to supplement the Apache
>> Hadoop experience
>>> * operations teams that may be able to leverage existing
>> documentation without having to remain local “exception” docs
>>> * development groups wanting an easy way to trial
>> experimental features
>>>
>>> Additionally, this branch includes the following, related
>> changes:
>>>
>>> * Adds the first unit tests for the ‘hadoop’ command
>>> * Adds the infrastructure for hdfs script testing and
>> the first unit test for the ‘hdfs’ command
>>> * Modifies the hadoop-tools components to be dynamic
>> rather than hard coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn
>> to be consistent with other bundled profiles, including the ones introduced
>> in this branch
>>>
>>> Documentation, including a ‘hello world’-style example, is in
>> the UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other
>> issues so that we may add unit testing to the shell code in those
>> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>
>>
>>
>> --
>> busbey
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project…. it’d be great if some others could take a look.
Thanks!
> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> +1. I looked at the patches on the branch, wasn't too bad to review. As
> Allen said, there's some code movement, assorted other nice doc and shell
> fixups.
>
> Found one extra typo, which I added to HADOOP-13129.
>
> Best,
> Andrew
>
> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> +1 (non-binding)
>>
>> reviewed everything, filed an additional subtask for a very trivial
>> typo in the docs. should be fine to make a full issue after close and
>> then fix.
>>
>> tried merging locally, tried running through new shell tests (both
>> with and without bats installed), tried making an example custom
>> command (valid and malformed). everything looks great.
>>
>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>
>>> Hey gang!
>>>
>>> I’d like to call a vote to run for 7 days (ending May 16 at
>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>> was developed exclusively by me as per the discussion two months ago as a
>> way to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>>
>>> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
>> commands to allow for subcommands to be added or modified at runtime. This
>> allows for individual users or entire sites to tweak the execution
>> environment to suit their local needs. For example, it has been a practice
>> for some locations to change the distcp jar out for a custom one. Using
>> this functionality, it is possible that the ‘hadoop distcp’ command could
>> run the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>>
>>> * advanced end users looking to supplement the Apache
>> Hadoop experience
>>> * operations teams that may be able to leverage existing
>> documentation without having to remain local “exception” docs
>>> * development groups wanting an easy way to trial
>> experimental features
>>>
>>> Additionally, this branch includes the following, related
>> changes:
>>>
>>> * Adds the first unit tests for the ‘hadoop’ command
>>> * Adds the infrastructure for hdfs script testing and
>> the first unit test for the ‘hdfs’ command
>>> * Modifies the hadoop-tools components to be dynamic
>> rather than hard coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn
>> to be consistent with other bundled profiles, including the ones introduced
>> in this branch
>>>
>>> Documentation, including a ‘hello world’-style example, is in
>> the UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other
>> issues so that we may add unit testing to the shell code in those
>> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>
>>
>>
>> --
>> busbey
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project…. it’d be great if some others could take a look.
Thanks!
> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> +1. I looked at the patches on the branch, wasn't too bad to review. As
> Allen said, there's some code movement, assorted other nice doc and shell
> fixups.
>
> Found one extra typo, which I added to HADOOP-13129.
>
> Best,
> Andrew
>
> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> +1 (non-binding)
>>
>> reviewed everything, filed an additional subtask for a very trivial
>> typo in the docs. should be fine to make a full issue after close and
>> then fix.
>>
>> tried merging locally, tried running through new shell tests (both
>> with and without bats installed), tried making an example custom
>> command (valid and malformed). everything looks great.
>>
>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>
>>> Hey gang!
>>>
>>> I’d like to call a vote to run for 7 days (ending May 16 at
>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>> was developed exclusively by me as per the discussion two months ago as a
>> way to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>>
>>> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
>> commands to allow for subcommands to be added or modified at runtime. This
>> allows for individual users or entire sites to tweak the execution
>> environment to suit their local needs. For example, it has been a practice
>> for some locations to change the distcp jar out for a custom one. Using
>> this functionality, it is possible that the ‘hadoop distcp’ command could
>> run the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>>
>>> * advanced end users looking to supplement the Apache
>> Hadoop experience
>>> * operations teams that may be able to leverage existing
>> documentation without having to remain local “exception” docs
>>> * development groups wanting an easy way to trial
>> experimental features
>>>
>>> Additionally, this branch includes the following, related
>> changes:
>>>
>>> * Adds the first unit tests for the ‘hadoop’ command
>>> * Adds the infrastructure for hdfs script testing and
>> the first unit test for the ‘hdfs’ command
>>> * Modifies the hadoop-tools components to be dynamic
>> rather than hard coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn
>> to be consistent with other bundled profiles, including the ones introduced
>> in this branch
>>>
>>> Documentation, including a ‘hello world’-style example, is in
>> the UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other
>> issues so that we may add unit testing to the shell code in those
>> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>
>>
>>
>> --
>> busbey
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
This vote closes in 2 days and the only response has been from a non-committer and one of the 137 other committers on the project…. it’d be great if some others could take a look.
Thanks!
> On May 12, 2016, at 6:07 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> +1. I looked at the patches on the branch, wasn't too bad to review. As
> Allen said, there's some code movement, assorted other nice doc and shell
> fixups.
>
> Found one extra typo, which I added to HADOOP-13129.
>
> Best,
> Andrew
>
> On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> +1 (non-binding)
>>
>> reviewed everything, filed an additional subtask for a very trivial
>> typo in the docs. should be fine to make a full issue after close and
>> then fix.
>>
>> tried merging locally, tried running through new shell tests (both
>> with and without bats installed), tried making an example custom
>> command (valid and malformed). everything looks great.
>>
>> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>>>
>>> Hey gang!
>>>
>>> I’d like to call a vote to run for 7 days (ending May 16 at
>> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
>> was developed exclusively by me as per the discussion two months ago as a
>> way to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>>
>>> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
>> commands to allow for subcommands to be added or modified at runtime. This
>> allows for individual users or entire sites to tweak the execution
>> environment to suit their local needs. For example, it has been a practice
>> for some locations to change the distcp jar out for a custom one. Using
>> this functionality, it is possible that the ‘hadoop distcp’ command could
>> run the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>>
>>> * advanced end users looking to supplement the Apache
>> Hadoop experience
>>> * operations teams that may be able to leverage existing
>> documentation without having to remain local “exception” docs
>>> * development groups wanting an easy way to trial
>> experimental features
>>>
>>> Additionally, this branch includes the following, related
>> changes:
>>>
>>> * Adds the first unit tests for the ‘hadoop’ command
>>> * Adds the infrastructure for hdfs script testing and
>> the first unit test for the ‘hdfs’ command
>>> * Modifies the hadoop-tools components to be dynamic
>> rather than hard coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn
>> to be consistent with other bundled profiles, including the ones introduced
>> in this branch
>>>
>>> Documentation, including a ‘hello world’-style example, is in
>> the UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other
>> issues so that we may add unit testing to the shell code in those
>> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>
>>
>>
>> --
>> busbey
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Andrew Wang <an...@cloudera.com>.
+1. I looked at the patches on the branch, wasn't too bad to review. As
Allen said, there's some code movement, assorted other nice doc and shell
fixups.
Found one extra typo, which I added to HADOOP-13129.
Best,
Andrew
On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
> +1 (non-binding)
>
> reviewed everything, filed an additional subtask for a very trivial
> typo in the docs. should be fine to make a full issue after close and
> then fix.
>
> tried merging locally, tried running through new shell tests (both
> with and without bats installed), tried making an example custom
> command (valid and malformed). everything looks great.
>
> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
> >
> > Hey gang!
> >
> > I’d like to call a vote to run for 7 days (ending May 16 at
> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
> was developed exclusively by me as per the discussion two months ago as a
> way to make what would be a rather large patch hopefully easier to review.
> The vast majority of the branch is code movement in the same file,
> additional license headers, maven assembly hooks for distribution, and
> variable renames. Not a whole lot of new code, but a big diff file
> none-the-less.
> >
> > This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
> commands to allow for subcommands to be added or modified at runtime. This
> allows for individual users or entire sites to tweak the execution
> environment to suit their local needs. For example, it has been a practice
> for some locations to change the distcp jar out for a custom one. Using
> this functionality, it is possible that the ‘hadoop distcp’ command could
> run the local version without overwriting the bundled jar and for existing
> documentation (read: results from Internet searches) to work as written
> without modification. This has the potential to be a huge win, especially
> for:
> >
> > * advanced end users looking to supplement the Apache
> Hadoop experience
> > * operations teams that may be able to leverage existing
> documentation without having to remain local “exception” docs
> > * development groups wanting an easy way to trial
> experimental features
> >
> > Additionally, this branch includes the following, related
> changes:
> >
> > * Adds the first unit tests for the ‘hadoop’ command
> > * Adds the infrastructure for hdfs script testing and
> the first unit test for the ‘hdfs’ command
> > * Modifies the hadoop-tools components to be dynamic
> rather than hard coded
> > * Renames the shell profiles for hdfs, mapred, and yarn
> to be consistent with other bundled profiles, including the ones introduced
> in this branch
> >
> > Documentation, including a ‘hello world’-style example, is in
> the UnixShellGuide markdown file. (Of course!)
> >
> > I am at ApacheCon this week if anyone wants to discuss in-depth.
> >
> > Thanks!
> >
> > P.S.,
> >
> > There are still two open sub-tasks. These are blocked by other
> issues so that we may add unit testing to the shell code in those
> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
>
>
> --
> busbey
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Andrew Wang <an...@cloudera.com>.
+1. I looked at the patches on the branch, wasn't too bad to review. As
Allen said, there's some code movement, assorted other nice doc and shell
fixups.
Found one extra typo, which I added to HADOOP-13129.
Best,
Andrew
On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
> +1 (non-binding)
>
> reviewed everything, filed an additional subtask for a very trivial
> typo in the docs. should be fine to make a full issue after close and
> then fix.
>
> tried merging locally, tried running through new shell tests (both
> with and without bats installed), tried making an example custom
> command (valid and malformed). everything looks great.
>
> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
> >
> > Hey gang!
> >
> > I’d like to call a vote to run for 7 days (ending May 16 at
> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
> was developed exclusively by me as per the discussion two months ago as a
> way to make what would be a rather large patch hopefully easier to review.
> The vast majority of the branch is code movement in the same file,
> additional license headers, maven assembly hooks for distribution, and
> variable renames. Not a whole lot of new code, but a big diff file
> none-the-less.
> >
> > This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
> commands to allow for subcommands to be added or modified at runtime. This
> allows for individual users or entire sites to tweak the execution
> environment to suit their local needs. For example, it has been a practice
> for some locations to change the distcp jar out for a custom one. Using
> this functionality, it is possible that the ‘hadoop distcp’ command could
> run the local version without overwriting the bundled jar and for existing
> documentation (read: results from Internet searches) to work as written
> without modification. This has the potential to be a huge win, especially
> for:
> >
> > * advanced end users looking to supplement the Apache
> Hadoop experience
> > * operations teams that may be able to leverage existing
> documentation without having to remain local “exception” docs
> > * development groups wanting an easy way to trial
> experimental features
> >
> > Additionally, this branch includes the following, related
> changes:
> >
> > * Adds the first unit tests for the ‘hadoop’ command
> > * Adds the infrastructure for hdfs script testing and
> the first unit test for the ‘hdfs’ command
> > * Modifies the hadoop-tools components to be dynamic
> rather than hard coded
> > * Renames the shell profiles for hdfs, mapred, and yarn
> to be consistent with other bundled profiles, including the ones introduced
> in this branch
> >
> > Documentation, including a ‘hello world’-style example, is in
> the UnixShellGuide markdown file. (Of course!)
> >
> > I am at ApacheCon this week if anyone wants to discuss in-depth.
> >
> > Thanks!
> >
> > P.S.,
> >
> > There are still two open sub-tasks. These are blocked by other
> issues so that we may add unit testing to the shell code in those
> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
>
>
> --
> busbey
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Andrew Wang <an...@cloudera.com>.
+1. I looked at the patches on the branch, wasn't too bad to review. As
Allen said, there's some code movement, assorted other nice doc and shell
fixups.
Found one extra typo, which I added to HADOOP-13129.
Best,
Andrew
On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
> +1 (non-binding)
>
> reviewed everything, filed an additional subtask for a very trivial
> typo in the docs. should be fine to make a full issue after close and
> then fix.
>
> tried merging locally, tried running through new shell tests (both
> with and without bats installed), tried making an example custom
> command (valid and malformed). everything looks great.
>
> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
> >
> > Hey gang!
> >
> > I’d like to call a vote to run for 7 days (ending May 16 at
> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
> was developed exclusively by me as per the discussion two months ago as a
> way to make what would be a rather large patch hopefully easier to review.
> The vast majority of the branch is code movement in the same file,
> additional license headers, maven assembly hooks for distribution, and
> variable renames. Not a whole lot of new code, but a big diff file
> none-the-less.
> >
> > This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
> commands to allow for subcommands to be added or modified at runtime. This
> allows for individual users or entire sites to tweak the execution
> environment to suit their local needs. For example, it has been a practice
> for some locations to change the distcp jar out for a custom one. Using
> this functionality, it is possible that the ‘hadoop distcp’ command could
> run the local version without overwriting the bundled jar and for existing
> documentation (read: results from Internet searches) to work as written
> without modification. This has the potential to be a huge win, especially
> for:
> >
> > * advanced end users looking to supplement the Apache
> Hadoop experience
> > * operations teams that may be able to leverage existing
> documentation without having to remain local “exception” docs
> > * development groups wanting an easy way to trial
> experimental features
> >
> > Additionally, this branch includes the following, related
> changes:
> >
> > * Adds the first unit tests for the ‘hadoop’ command
> > * Adds the infrastructure for hdfs script testing and
> the first unit test for the ‘hdfs’ command
> > * Modifies the hadoop-tools components to be dynamic
> rather than hard coded
> > * Renames the shell profiles for hdfs, mapred, and yarn
> to be consistent with other bundled profiles, including the ones introduced
> in this branch
> >
> > Documentation, including a ‘hello world’-style example, is in
> the UnixShellGuide markdown file. (Of course!)
> >
> > I am at ApacheCon this week if anyone wants to discuss in-depth.
> >
> > Thanks!
> >
> > P.S.,
> >
> > There are still two open sub-tasks. These are blocked by other
> issues so that we may add unit testing to the shell code in those
> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
>
>
> --
> busbey
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Andrew Wang <an...@cloudera.com>.
+1. I looked at the patches on the branch, wasn't too bad to review. As
Allen said, there's some code movement, assorted other nice doc and shell
fixups.
Found one extra typo, which I added to HADOOP-13129.
Best,
Andrew
On Wed, May 11, 2016 at 1:14 AM, Sean Busbey <bu...@cloudera.com> wrote:
> +1 (non-binding)
>
> reviewed everything, filed an additional subtask for a very trivial
> typo in the docs. should be fine to make a full issue after close and
> then fix.
>
> tried merging locally, tried running through new shell tests (both
> with and without bats installed), tried making an example custom
> command (valid and malformed). everything looks great.
>
> On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
> >
> > Hey gang!
> >
> > I’d like to call a vote to run for 7 days (ending May 16 at
> 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch
> was developed exclusively by me as per the discussion two months ago as a
> way to make what would be a rather large patch hopefully easier to review.
> The vast majority of the branch is code movement in the same file,
> additional license headers, maven assembly hooks for distribution, and
> variable renames. Not a whole lot of new code, but a big diff file
> none-the-less.
> >
> > This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’
> commands to allow for subcommands to be added or modified at runtime. This
> allows for individual users or entire sites to tweak the execution
> environment to suit their local needs. For example, it has been a practice
> for some locations to change the distcp jar out for a custom one. Using
> this functionality, it is possible that the ‘hadoop distcp’ command could
> run the local version without overwriting the bundled jar and for existing
> documentation (read: results from Internet searches) to work as written
> without modification. This has the potential to be a huge win, especially
> for:
> >
> > * advanced end users looking to supplement the Apache
> Hadoop experience
> > * operations teams that may be able to leverage existing
> documentation without having to remain local “exception” docs
> > * development groups wanting an easy way to trial
> experimental features
> >
> > Additionally, this branch includes the following, related
> changes:
> >
> > * Adds the first unit tests for the ‘hadoop’ command
> > * Adds the infrastructure for hdfs script testing and
> the first unit test for the ‘hdfs’ command
> > * Modifies the hadoop-tools components to be dynamic
> rather than hard coded
> > * Renames the shell profiles for hdfs, mapred, and yarn
> to be consistent with other bundled profiles, including the ones introduced
> in this branch
> >
> > Documentation, including a ‘hello world’-style example, is in
> the UnixShellGuide markdown file. (Of course!)
> >
> > I am at ApacheCon this week if anyone wants to discuss in-depth.
> >
> > Thanks!
> >
> > P.S.,
> >
> > There are still two open sub-tasks. These are blocked by other
> issues so that we may add unit testing to the shell code in those
> respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
> >
>
>
>
> --
> busbey
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Sean Busbey <bu...@cloudera.com>.
+1 (non-binding)
reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.
tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.
On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
> Hey gang!
>
> I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch was developed exclusively by me as per the discussion two months ago as a way to make what would be a rather large patch hopefully easier to review. The vast majority of the branch is code movement in the same file, additional license headers, maven assembly hooks for distribution, and variable renames. Not a whole lot of new code, but a big diff file none-the-less.
>
> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ commands to allow for subcommands to be added or modified at runtime. This allows for individual users or entire sites to tweak the execution environment to suit their local needs. For example, it has been a practice for some locations to change the distcp jar out for a custom one. Using this functionality, it is possible that the ‘hadoop distcp’ command could run the local version without overwriting the bundled jar and for existing documentation (read: results from Internet searches) to work as written without modification. This has the potential to be a huge win, especially for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation without having to remain local “exception” docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the ‘hadoop’ command
> * Adds the infrastructure for hdfs script testing and the first unit test for the ‘hdfs’ command
> * Modifies the hadoop-tools components to be dynamic rather than hard coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be consistent with other bundled profiles, including the ones introduced in this branch
>
> Documentation, including a ‘hello world’-style example, is in the UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
> P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues so that we may add unit testing to the shell code in those respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
--
busbey
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
Understood about the tests.
--Chris Nauroth
On 5/15/16, 7:30 AM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com>
>>wrote:
>>
>> +1 (binding)
>>
>> -Tried a dry-run merge of HADOOP-12930 to trunk.
>> -Successfully built distro on Windows.
>> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
>> commands through Cygwin.
>> -Reviewed documentation.
>>
>> Allen, thank you for the contribution. Would you please attach a full
>> patch to HADOOP-12930 to check pre-commit results?
>
>
> Nope. The whole reason this was done as a branch with multiple patches
>was to prevent Jenkins from getting overwhelmed since it would trigger
>full unit tests on pretty much the entire code base….
>
>> While testing this, I discovered a bug in the distro build for Windows.
>> Could someone please code review my patch on HADOOP-13149?
>
> Done!
>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>>
>>>
>>> Hey gang!
>>>
>>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT)
>>>to
>>> merge the HADOOP-12930 feature branch into trunk. This branch was
>>> developed exclusively by me as per the discussion two months ago as a
>>>way
>>> to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>
>>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹
>>>commands
>>> to allow for subcommands to be added or modified at runtime. This
>>>allows
>>> for individual users or entire sites to tweak the execution environment
>>> to suit their local needs. For example, it has been a practice for
>>>some
>>> locations to change the distcp jar out for a custom one. Using this
>>> functionality, it is possible that the Œhadoop distcp¹ command could
>>>run
>>> the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win,
>>>especially
>>> for:
>>>
>>> * advanced end users looking to supplement the Apache Hadoop
>>>experience
>>> * operations teams that may be able to leverage existing
>>>documentation
>>> without having to remain local ³exception² docs
>>> * development groups wanting an easy way to trial experimental
>>>features
>>>
>>> Additionally, this branch includes the following, related changes:
>>>
>>> * Adds the first unit tests for the Œhadoop¹ command
>>> * Adds the infrastructure for hdfs script testing and the first unit
>>> test for the Œhdfs¹ command
>>> * Modifies the hadoop-tools components to be dynamic rather than hard
>>> coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>>> consistent with other bundled profiles, including the ones introduced
>>>in
>>> this branch
>>>
>>> Documentation, including a Œhello world¹-style example, is in the
>>> UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other issues
>>> so that we may add unit testing to the shell code in those respective
>>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
Understood about the tests.
--Chris Nauroth
On 5/15/16, 7:30 AM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com>
>>wrote:
>>
>> +1 (binding)
>>
>> -Tried a dry-run merge of HADOOP-12930 to trunk.
>> -Successfully built distro on Windows.
>> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
>> commands through Cygwin.
>> -Reviewed documentation.
>>
>> Allen, thank you for the contribution. Would you please attach a full
>> patch to HADOOP-12930 to check pre-commit results?
>
>
> Nope. The whole reason this was done as a branch with multiple patches
>was to prevent Jenkins from getting overwhelmed since it would trigger
>full unit tests on pretty much the entire code base….
>
>> While testing this, I discovered a bug in the distro build for Windows.
>> Could someone please code review my patch on HADOOP-13149?
>
> Done!
>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>>
>>>
>>> Hey gang!
>>>
>>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT)
>>>to
>>> merge the HADOOP-12930 feature branch into trunk. This branch was
>>> developed exclusively by me as per the discussion two months ago as a
>>>way
>>> to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>
>>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹
>>>commands
>>> to allow for subcommands to be added or modified at runtime. This
>>>allows
>>> for individual users or entire sites to tweak the execution environment
>>> to suit their local needs. For example, it has been a practice for
>>>some
>>> locations to change the distcp jar out for a custom one. Using this
>>> functionality, it is possible that the Œhadoop distcp¹ command could
>>>run
>>> the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win,
>>>especially
>>> for:
>>>
>>> * advanced end users looking to supplement the Apache Hadoop
>>>experience
>>> * operations teams that may be able to leverage existing
>>>documentation
>>> without having to remain local ³exception² docs
>>> * development groups wanting an easy way to trial experimental
>>>features
>>>
>>> Additionally, this branch includes the following, related changes:
>>>
>>> * Adds the first unit tests for the Œhadoop¹ command
>>> * Adds the infrastructure for hdfs script testing and the first unit
>>> test for the Œhdfs¹ command
>>> * Modifies the hadoop-tools components to be dynamic rather than hard
>>> coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>>> consistent with other bundled profiles, including the ones introduced
>>>in
>>> this branch
>>>
>>> Documentation, including a Œhello world¹-style example, is in the
>>> UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other issues
>>> so that we may add unit testing to the shell code in those respective
>>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
Understood about the tests.
--Chris Nauroth
On 5/15/16, 7:30 AM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com>
>>wrote:
>>
>> +1 (binding)
>>
>> -Tried a dry-run merge of HADOOP-12930 to trunk.
>> -Successfully built distro on Windows.
>> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
>> commands through Cygwin.
>> -Reviewed documentation.
>>
>> Allen, thank you for the contribution. Would you please attach a full
>> patch to HADOOP-12930 to check pre-commit results?
>
>
> Nope. The whole reason this was done as a branch with multiple patches
>was to prevent Jenkins from getting overwhelmed since it would trigger
>full unit tests on pretty much the entire code base….
>
>> While testing this, I discovered a bug in the distro build for Windows.
>> Could someone please code review my patch on HADOOP-13149?
>
> Done!
>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>>
>>>
>>> Hey gang!
>>>
>>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT)
>>>to
>>> merge the HADOOP-12930 feature branch into trunk. This branch was
>>> developed exclusively by me as per the discussion two months ago as a
>>>way
>>> to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>
>>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹
>>>commands
>>> to allow for subcommands to be added or modified at runtime. This
>>>allows
>>> for individual users or entire sites to tweak the execution environment
>>> to suit their local needs. For example, it has been a practice for
>>>some
>>> locations to change the distcp jar out for a custom one. Using this
>>> functionality, it is possible that the Œhadoop distcp¹ command could
>>>run
>>> the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win,
>>>especially
>>> for:
>>>
>>> * advanced end users looking to supplement the Apache Hadoop
>>>experience
>>> * operations teams that may be able to leverage existing
>>>documentation
>>> without having to remain local ³exception² docs
>>> * development groups wanting an easy way to trial experimental
>>>features
>>>
>>> Additionally, this branch includes the following, related changes:
>>>
>>> * Adds the first unit tests for the Œhadoop¹ command
>>> * Adds the infrastructure for hdfs script testing and the first unit
>>> test for the Œhdfs¹ command
>>> * Modifies the hadoop-tools components to be dynamic rather than hard
>>> coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>>> consistent with other bundled profiles, including the ones introduced
>>>in
>>> this branch
>>>
>>> Documentation, including a Œhello world¹-style example, is in the
>>> UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other issues
>>> so that we may add unit testing to the shell code in those respective
>>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
Understood about the tests.
--Chris Nauroth
On 5/15/16, 7:30 AM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com>
>>wrote:
>>
>> +1 (binding)
>>
>> -Tried a dry-run merge of HADOOP-12930 to trunk.
>> -Successfully built distro on Windows.
>> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
>> commands through Cygwin.
>> -Reviewed documentation.
>>
>> Allen, thank you for the contribution. Would you please attach a full
>> patch to HADOOP-12930 to check pre-commit results?
>
>
> Nope. The whole reason this was done as a branch with multiple patches
>was to prevent Jenkins from getting overwhelmed since it would trigger
>full unit tests on pretty much the entire code base….
>
>> While testing this, I discovered a bug in the distro build for Windows.
>> Could someone please code review my patch on HADOOP-13149?
>
> Done!
>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>>
>>>
>>> Hey gang!
>>>
>>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT)
>>>to
>>> merge the HADOOP-12930 feature branch into trunk. This branch was
>>> developed exclusively by me as per the discussion two months ago as a
>>>way
>>> to make what would be a rather large patch hopefully easier to review.
>>> The vast majority of the branch is code movement in the same file,
>>> additional license headers, maven assembly hooks for distribution, and
>>> variable renames. Not a whole lot of new code, but a big diff file
>>> none-the-less.
>>>
>>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹
>>>commands
>>> to allow for subcommands to be added or modified at runtime. This
>>>allows
>>> for individual users or entire sites to tweak the execution environment
>>> to suit their local needs. For example, it has been a practice for
>>>some
>>> locations to change the distcp jar out for a custom one. Using this
>>> functionality, it is possible that the Œhadoop distcp¹ command could
>>>run
>>> the local version without overwriting the bundled jar and for existing
>>> documentation (read: results from Internet searches) to work as written
>>> without modification. This has the potential to be a huge win,
>>>especially
>>> for:
>>>
>>> * advanced end users looking to supplement the Apache Hadoop
>>>experience
>>> * operations teams that may be able to leverage existing
>>>documentation
>>> without having to remain local ³exception² docs
>>> * development groups wanting an easy way to trial experimental
>>>features
>>>
>>> Additionally, this branch includes the following, related changes:
>>>
>>> * Adds the first unit tests for the Œhadoop¹ command
>>> * Adds the infrastructure for hdfs script testing and the first unit
>>> test for the Œhdfs¹ command
>>> * Modifies the hadoop-tools components to be dynamic rather than hard
>>> coded
>>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>>> consistent with other bundled profiles, including the ones introduced
>>>in
>>> this branch
>>>
>>> Documentation, including a Œhello world¹-style example, is in the
>>> UnixShellGuide markdown file. (Of course!)
>>>
>>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>>
>>> Thanks!
>>>
>>> P.S.,
>>>
>>> There are still two open sub-tasks. These are blocked by other issues
>>> so that we may add unit testing to the shell code in those respective
>>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>>
>>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> +1 (binding)
>
> -Tried a dry-run merge of HADOOP-12930 to trunk.
> -Successfully built distro on Windows.
> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
> commands through Cygwin.
> -Reviewed documentation.
>
> Allen, thank you for the contribution. Would you please attach a full
> patch to HADOOP-12930 to check pre-commit results?
Nope. The whole reason this was done as a branch with multiple patches was to prevent Jenkins from getting overwhelmed since it would trigger full unit tests on pretty much the entire code base….
> While testing this, I discovered a bug in the distro build for Windows.
> Could someone please code review my patch on HADOOP-13149?
Done!
>
> --Chris Nauroth
>
>
>
>
> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>>
>> Hey gang!
>>
>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>> merge the HADOOP-12930 feature branch into trunk. This branch was
>> developed exclusively by me as per the discussion two months ago as a way
>> to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>
>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>> to allow for subcommands to be added or modified at runtime. This allows
>> for individual users or entire sites to tweak the execution environment
>> to suit their local needs. For example, it has been a practice for some
>> locations to change the distcp jar out for a custom one. Using this
>> functionality, it is possible that the Œhadoop distcp¹ command could run
>> the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>
>> * advanced end users looking to supplement the Apache Hadoop experience
>> * operations teams that may be able to leverage existing documentation
>> without having to remain local ³exception² docs
>> * development groups wanting an easy way to trial experimental features
>>
>> Additionally, this branch includes the following, related changes:
>>
>> * Adds the first unit tests for the Œhadoop¹ command
>> * Adds the infrastructure for hdfs script testing and the first unit
>> test for the Œhdfs¹ command
>> * Modifies the hadoop-tools components to be dynamic rather than hard
>> coded
>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>> consistent with other bundled profiles, including the ones introduced in
>> this branch
>>
>> Documentation, including a Œhello world¹-style example, is in the
>> UnixShellGuide markdown file. (Of course!)
>>
>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>
>> Thanks!
>>
>> P.S.,
>>
>> There are still two open sub-tasks. These are blocked by other issues
>> so that we may add unit testing to the shell code in those respective
>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> +1 (binding)
>
> -Tried a dry-run merge of HADOOP-12930 to trunk.
> -Successfully built distro on Windows.
> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
> commands through Cygwin.
> -Reviewed documentation.
>
> Allen, thank you for the contribution. Would you please attach a full
> patch to HADOOP-12930 to check pre-commit results?
Nope. The whole reason this was done as a branch with multiple patches was to prevent Jenkins from getting overwhelmed since it would trigger full unit tests on pretty much the entire code base….
> While testing this, I discovered a bug in the distro build for Windows.
> Could someone please code review my patch on HADOOP-13149?
Done!
>
> --Chris Nauroth
>
>
>
>
> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>>
>> Hey gang!
>>
>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>> merge the HADOOP-12930 feature branch into trunk. This branch was
>> developed exclusively by me as per the discussion two months ago as a way
>> to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>
>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>> to allow for subcommands to be added or modified at runtime. This allows
>> for individual users or entire sites to tweak the execution environment
>> to suit their local needs. For example, it has been a practice for some
>> locations to change the distcp jar out for a custom one. Using this
>> functionality, it is possible that the Œhadoop distcp¹ command could run
>> the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>
>> * advanced end users looking to supplement the Apache Hadoop experience
>> * operations teams that may be able to leverage existing documentation
>> without having to remain local ³exception² docs
>> * development groups wanting an easy way to trial experimental features
>>
>> Additionally, this branch includes the following, related changes:
>>
>> * Adds the first unit tests for the Œhadoop¹ command
>> * Adds the infrastructure for hdfs script testing and the first unit
>> test for the Œhdfs¹ command
>> * Modifies the hadoop-tools components to be dynamic rather than hard
>> coded
>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>> consistent with other bundled profiles, including the ones introduced in
>> this branch
>>
>> Documentation, including a Œhello world¹-style example, is in the
>> UnixShellGuide markdown file. (Of course!)
>>
>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>
>> Thanks!
>>
>> P.S.,
>>
>> There are still two open sub-tasks. These are blocked by other issues
>> so that we may add unit testing to the shell code in those respective
>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> +1 (binding)
>
> -Tried a dry-run merge of HADOOP-12930 to trunk.
> -Successfully built distro on Windows.
> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
> commands through Cygwin.
> -Reviewed documentation.
>
> Allen, thank you for the contribution. Would you please attach a full
> patch to HADOOP-12930 to check pre-commit results?
Nope. The whole reason this was done as a branch with multiple patches was to prevent Jenkins from getting overwhelmed since it would trigger full unit tests on pretty much the entire code base….
> While testing this, I discovered a bug in the distro build for Windows.
> Could someone please code review my patch on HADOOP-13149?
Done!
>
> --Chris Nauroth
>
>
>
>
> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>>
>> Hey gang!
>>
>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>> merge the HADOOP-12930 feature branch into trunk. This branch was
>> developed exclusively by me as per the discussion two months ago as a way
>> to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>
>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>> to allow for subcommands to be added or modified at runtime. This allows
>> for individual users or entire sites to tweak the execution environment
>> to suit their local needs. For example, it has been a practice for some
>> locations to change the distcp jar out for a custom one. Using this
>> functionality, it is possible that the Œhadoop distcp¹ command could run
>> the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>
>> * advanced end users looking to supplement the Apache Hadoop experience
>> * operations teams that may be able to leverage existing documentation
>> without having to remain local ³exception² docs
>> * development groups wanting an easy way to trial experimental features
>>
>> Additionally, this branch includes the following, related changes:
>>
>> * Adds the first unit tests for the Œhadoop¹ command
>> * Adds the infrastructure for hdfs script testing and the first unit
>> test for the Œhdfs¹ command
>> * Modifies the hadoop-tools components to be dynamic rather than hard
>> coded
>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>> consistent with other bundled profiles, including the ones introduced in
>> this branch
>>
>> Documentation, including a Œhello world¹-style example, is in the
>> UnixShellGuide markdown file. (Of course!)
>>
>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>
>> Thanks!
>>
>> P.S.,
>>
>> There are still two open sub-tasks. These are blocked by other issues
>> so that we may add unit testing to the shell code in those respective
>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 14, 2016, at 3:11 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> +1 (binding)
>
> -Tried a dry-run merge of HADOOP-12930 to trunk.
> -Successfully built distro on Windows.
> -Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
> commands through Cygwin.
> -Reviewed documentation.
>
> Allen, thank you for the contribution. Would you please attach a full
> patch to HADOOP-12930 to check pre-commit results?
Nope. The whole reason this was done as a branch with multiple patches was to prevent Jenkins from getting overwhelmed since it would trigger full unit tests on pretty much the entire code base….
> While testing this, I discovered a bug in the distro build for Windows.
> Could someone please code review my patch on HADOOP-13149?
Done!
>
> --Chris Nauroth
>
>
>
>
> On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
>>
>> Hey gang!
>>
>> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>> merge the HADOOP-12930 feature branch into trunk. This branch was
>> developed exclusively by me as per the discussion two months ago as a way
>> to make what would be a rather large patch hopefully easier to review.
>> The vast majority of the branch is code movement in the same file,
>> additional license headers, maven assembly hooks for distribution, and
>> variable renames. Not a whole lot of new code, but a big diff file
>> none-the-less.
>>
>> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>> to allow for subcommands to be added or modified at runtime. This allows
>> for individual users or entire sites to tweak the execution environment
>> to suit their local needs. For example, it has been a practice for some
>> locations to change the distcp jar out for a custom one. Using this
>> functionality, it is possible that the Œhadoop distcp¹ command could run
>> the local version without overwriting the bundled jar and for existing
>> documentation (read: results from Internet searches) to work as written
>> without modification. This has the potential to be a huge win, especially
>> for:
>>
>> * advanced end users looking to supplement the Apache Hadoop experience
>> * operations teams that may be able to leverage existing documentation
>> without having to remain local ³exception² docs
>> * development groups wanting an easy way to trial experimental features
>>
>> Additionally, this branch includes the following, related changes:
>>
>> * Adds the first unit tests for the Œhadoop¹ command
>> * Adds the infrastructure for hdfs script testing and the first unit
>> test for the Œhdfs¹ command
>> * Modifies the hadoop-tools components to be dynamic rather than hard
>> coded
>> * Renames the shell profiles for hdfs, mapred, and yarn to be
>> consistent with other bundled profiles, including the ones introduced in
>> this branch
>>
>> Documentation, including a Œhello world¹-style example, is in the
>> UnixShellGuide markdown file. (Of course!)
>>
>> I am at ApacheCon this week if anyone wants to discuss in-depth.
>>
>> Thanks!
>>
>> P.S.,
>>
>> There are still two open sub-tasks. These are blocked by other issues
>> so that we may add unit testing to the shell code in those respective
>> areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
+1 (binding)
-Tried a dry-run merge of HADOOP-12930 to trunk.
-Successfully built distro on Windows.
-Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
commands through Cygwin.
-Reviewed documentation.
Allen, thank you for the contribution. Would you please attach a full
patch to HADOOP-12930 to check pre-commit results?
While testing this, I discovered a bug in the distro build for Windows.
Could someone please code review my patch on HADOOP-13149?
--Chris Nauroth
On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
> Hey gang!
>
> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>merge the HADOOP-12930 feature branch into trunk. This branch was
>developed exclusively by me as per the discussion two months ago as a way
>to make what would be a rather large patch hopefully easier to review.
>The vast majority of the branch is code movement in the same file,
>additional license headers, maven assembly hooks for distribution, and
>variable renames. Not a whole lot of new code, but a big diff file
>none-the-less.
>
> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>to allow for subcommands to be added or modified at runtime. This allows
>for individual users or entire sites to tweak the execution environment
>to suit their local needs. For example, it has been a practice for some
>locations to change the distcp jar out for a custom one. Using this
>functionality, it is possible that the Œhadoop distcp¹ command could run
>the local version without overwriting the bundled jar and for existing
>documentation (read: results from Internet searches) to work as written
>without modification. This has the potential to be a huge win, especially
>for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation
>without having to remain local ³exception² docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the Œhadoop¹ command
> * Adds the infrastructure for hdfs script testing and the first unit
>test for the Œhdfs¹ command
> * Modifies the hadoop-tools components to be dynamic rather than hard
>coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be
>consistent with other bundled profiles, including the ones introduced in
>this branch
>
> Documentation, including a Œhello world¹-style example, is in the
>UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
>P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues
>so that we may add unit testing to the shell code in those respective
>areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
+1 (binding)
-Tried a dry-run merge of HADOOP-12930 to trunk.
-Successfully built distro on Windows.
-Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
commands through Cygwin.
-Reviewed documentation.
Allen, thank you for the contribution. Would you please attach a full
patch to HADOOP-12930 to check pre-commit results?
While testing this, I discovered a bug in the distro build for Windows.
Could someone please code review my patch on HADOOP-13149?
--Chris Nauroth
On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
> Hey gang!
>
> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>merge the HADOOP-12930 feature branch into trunk. This branch was
>developed exclusively by me as per the discussion two months ago as a way
>to make what would be a rather large patch hopefully easier to review.
>The vast majority of the branch is code movement in the same file,
>additional license headers, maven assembly hooks for distribution, and
>variable renames. Not a whole lot of new code, but a big diff file
>none-the-less.
>
> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>to allow for subcommands to be added or modified at runtime. This allows
>for individual users or entire sites to tweak the execution environment
>to suit their local needs. For example, it has been a practice for some
>locations to change the distcp jar out for a custom one. Using this
>functionality, it is possible that the Œhadoop distcp¹ command could run
>the local version without overwriting the bundled jar and for existing
>documentation (read: results from Internet searches) to work as written
>without modification. This has the potential to be a huge win, especially
>for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation
>without having to remain local ³exception² docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the Œhadoop¹ command
> * Adds the infrastructure for hdfs script testing and the first unit
>test for the Œhdfs¹ command
> * Modifies the hadoop-tools components to be dynamic rather than hard
>coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be
>consistent with other bundled profiles, including the ones introduced in
>this branch
>
> Documentation, including a Œhello world¹-style example, is in the
>UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
>P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues
>so that we may add unit testing to the shell code in those respective
>areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
[RESULT] Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
Vote passes.
1 = +1 non-binding
4 = +1 binding
I’ll squash and commit here in a sec.
Thanks everyone!
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
[RESULT] Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
Vote passes.
1 = +1 non-binding
4 = +1 binding
I’ll squash and commit here in a sec.
Thanks everyone!
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Sean Busbey <bu...@cloudera.com>.
+1 (non-binding)
reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.
tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.
On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
> Hey gang!
>
> I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch was developed exclusively by me as per the discussion two months ago as a way to make what would be a rather large patch hopefully easier to review. The vast majority of the branch is code movement in the same file, additional license headers, maven assembly hooks for distribution, and variable renames. Not a whole lot of new code, but a big diff file none-the-less.
>
> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ commands to allow for subcommands to be added or modified at runtime. This allows for individual users or entire sites to tweak the execution environment to suit their local needs. For example, it has been a practice for some locations to change the distcp jar out for a custom one. Using this functionality, it is possible that the ‘hadoop distcp’ command could run the local version without overwriting the bundled jar and for existing documentation (read: results from Internet searches) to work as written without modification. This has the potential to be a huge win, especially for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation without having to remain local “exception” docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the ‘hadoop’ command
> * Adds the infrastructure for hdfs script testing and the first unit test for the ‘hdfs’ command
> * Modifies the hadoop-tools components to be dynamic rather than hard coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be consistent with other bundled profiles, including the ones introduced in this branch
>
> Documentation, including a ‘hello world’-style example, is in the UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
> P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues so that we may add unit testing to the shell code in those respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
--
busbey
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
[RESULT] Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Allen Wittenauer <aw...@apache.org>.
Vote passes.
1 = +1 non-binding
4 = +1 binding
I’ll squash and commit here in a sec.
Thanks everyone!
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Sean Busbey <bu...@cloudera.com>.
+1 (non-binding)
reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.
tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.
On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
> Hey gang!
>
> I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch was developed exclusively by me as per the discussion two months ago as a way to make what would be a rather large patch hopefully easier to review. The vast majority of the branch is code movement in the same file, additional license headers, maven assembly hooks for distribution, and variable renames. Not a whole lot of new code, but a big diff file none-the-less.
>
> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ commands to allow for subcommands to be added or modified at runtime. This allows for individual users or entire sites to tweak the execution environment to suit their local needs. For example, it has been a practice for some locations to change the distcp jar out for a custom one. Using this functionality, it is possible that the ‘hadoop distcp’ command could run the local version without overwriting the bundled jar and for existing documentation (read: results from Internet searches) to work as written without modification. This has the potential to be a huge win, especially for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation without having to remain local “exception” docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the ‘hadoop’ command
> * Adds the infrastructure for hdfs script testing and the first unit test for the ‘hdfs’ command
> * Modifies the hadoop-tools components to be dynamic rather than hard coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be consistent with other bundled profiles, including the ones introduced in this branch
>
> Documentation, including a ‘hello world’-style example, is in the UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
> P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues so that we may add unit testing to the shell code in those respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
--
busbey
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Sean Busbey <bu...@cloudera.com>.
+1 (non-binding)
reviewed everything, filed an additional subtask for a very trivial
typo in the docs. should be fine to make a full issue after close and
then fix.
tried merging locally, tried running through new shell tests (both
with and without bats installed), tried making an example custom
command (valid and malformed). everything looks great.
On Mon, May 9, 2016 at 1:26 PM, Allen Wittenauer <aw...@apache.org> wrote:
>
> Hey gang!
>
> I’d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to merge the HADOOP-12930 feature branch into trunk. This branch was developed exclusively by me as per the discussion two months ago as a way to make what would be a rather large patch hopefully easier to review. The vast majority of the branch is code movement in the same file, additional license headers, maven assembly hooks for distribution, and variable renames. Not a whole lot of new code, but a big diff file none-the-less.
>
> This branch modifies the ‘hadoop’, ‘hdfs’, ‘mapred’, and ‘yarn’ commands to allow for subcommands to be added or modified at runtime. This allows for individual users or entire sites to tweak the execution environment to suit their local needs. For example, it has been a practice for some locations to change the distcp jar out for a custom one. Using this functionality, it is possible that the ‘hadoop distcp’ command could run the local version without overwriting the bundled jar and for existing documentation (read: results from Internet searches) to work as written without modification. This has the potential to be a huge win, especially for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation without having to remain local “exception” docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the ‘hadoop’ command
> * Adds the infrastructure for hdfs script testing and the first unit test for the ‘hdfs’ command
> * Modifies the hadoop-tools components to be dynamic rather than hard coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be consistent with other bundled profiles, including the ones introduced in this branch
>
> Documentation, including a ‘hello world’-style example, is in the UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
> P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues so that we may add unit testing to the shell code in those respective areas. I’ll covert to full issues after HADOOP-12930 is closed.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
--
busbey
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
+1 (binding)
-Tried a dry-run merge of HADOOP-12930 to trunk.
-Successfully built distro on Windows.
-Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
commands through Cygwin.
-Reviewed documentation.
Allen, thank you for the contribution. Would you please attach a full
patch to HADOOP-12930 to check pre-commit results?
While testing this, I discovered a bug in the distro build for Windows.
Could someone please code review my patch on HADOOP-13149?
--Chris Nauroth
On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
> Hey gang!
>
> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>merge the HADOOP-12930 feature branch into trunk. This branch was
>developed exclusively by me as per the discussion two months ago as a way
>to make what would be a rather large patch hopefully easier to review.
>The vast majority of the branch is code movement in the same file,
>additional license headers, maven assembly hooks for distribution, and
>variable renames. Not a whole lot of new code, but a big diff file
>none-the-less.
>
> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>to allow for subcommands to be added or modified at runtime. This allows
>for individual users or entire sites to tweak the execution environment
>to suit their local needs. For example, it has been a practice for some
>locations to change the distcp jar out for a custom one. Using this
>functionality, it is possible that the Œhadoop distcp¹ command could run
>the local version without overwriting the bundled jar and for existing
>documentation (read: results from Internet searches) to work as written
>without modification. This has the potential to be a huge win, especially
>for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation
>without having to remain local ³exception² docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the Œhadoop¹ command
> * Adds the infrastructure for hdfs script testing and the first unit
>test for the Œhdfs¹ command
> * Modifies the hadoop-tools components to be dynamic rather than hard
>coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be
>consistent with other bundled profiles, including the ones introduced in
>this branch
>
> Documentation, including a Œhello world¹-style example, is in the
>UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
>P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues
>so that we may add unit testing to the shell code in those respective
>areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch HADOOP-12930
Posted by Chris Nauroth <cn...@hortonworks.com>.
+1 (binding)
-Tried a dry-run merge of HADOOP-12930 to trunk.
-Successfully built distro on Windows.
-Ran "hdfs namenode", "hdfs datanode", and various interactive hdfs
commands through Cygwin.
-Reviewed documentation.
Allen, thank you for the contribution. Would you please attach a full
patch to HADOOP-12930 to check pre-commit results?
While testing this, I discovered a bug in the distro build for Windows.
Could someone please code review my patch on HADOOP-13149?
--Chris Nauroth
On 5/9/16, 1:26 PM, "Allen Wittenauer" <aw...@apache.org> wrote:
>
> Hey gang!
>
> I¹d like to call a vote to run for 7 days (ending May 16 at 13:30 PT) to
>merge the HADOOP-12930 feature branch into trunk. This branch was
>developed exclusively by me as per the discussion two months ago as a way
>to make what would be a rather large patch hopefully easier to review.
>The vast majority of the branch is code movement in the same file,
>additional license headers, maven assembly hooks for distribution, and
>variable renames. Not a whole lot of new code, but a big diff file
>none-the-less.
>
> This branch modifies the Œhadoop¹, Œhdfs¹, Œmapred¹, and Œyarn¹ commands
>to allow for subcommands to be added or modified at runtime. This allows
>for individual users or entire sites to tweak the execution environment
>to suit their local needs. For example, it has been a practice for some
>locations to change the distcp jar out for a custom one. Using this
>functionality, it is possible that the Œhadoop distcp¹ command could run
>the local version without overwriting the bundled jar and for existing
>documentation (read: results from Internet searches) to work as written
>without modification. This has the potential to be a huge win, especially
>for:
>
> * advanced end users looking to supplement the Apache Hadoop experience
> * operations teams that may be able to leverage existing documentation
>without having to remain local ³exception² docs
> * development groups wanting an easy way to trial experimental features
>
> Additionally, this branch includes the following, related changes:
>
> * Adds the first unit tests for the Œhadoop¹ command
> * Adds the infrastructure for hdfs script testing and the first unit
>test for the Œhdfs¹ command
> * Modifies the hadoop-tools components to be dynamic rather than hard
>coded
> * Renames the shell profiles for hdfs, mapred, and yarn to be
>consistent with other bundled profiles, including the ones introduced in
>this branch
>
> Documentation, including a Œhello world¹-style example, is in the
>UnixShellGuide markdown file. (Of course!)
>
> I am at ApacheCon this week if anyone wants to discuss in-depth.
>
> Thanks!
>
>P.S.,
>
> There are still two open sub-tasks. These are blocked by other issues
>so that we may add unit testing to the shell code in those respective
>areas. I¹ll covert to full issues after HADOOP-12930 is closed.
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org