You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Peter Cseh <ge...@cloudera.com> on 2017/08/10 17:21:42 UTC
Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs
-----
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java b507c79d44aa2095caf532a5f4d19d20c1a30630
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/1/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
> > Line 66 (original), 65 (patched)
> > <https://reviews.apache.org/r/61560/diff/1/?file=1794986#file1794986line66>
> >
> > Have we verified that HCat/HMS is okay with changing the alias? I don't know if maybe they were doing something funny with it.
>
> Peter Cseh wrote:
> I know there is a reference for it in Sqoop (https://issues.apache.org/jira/browse/SQOOP-3177) , but It does not seem to work anyways.
I'm reverting the token alias changes
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
-----------------------------------------------------------
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
> > Line 66 (original), 65 (patched)
> > <https://reviews.apache.org/r/61560/diff/1/?file=1794986#file1794986line66>
> >
> > Have we verified that HCat/HMS is okay with changing the alias? I don't know if maybe they were doing something funny with it.
I know there is a reference for it in Sqoop (https://issues.apache.org/jira/browse/SQOOP-3177) , but It does not seem to work anyways.
> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java
> > Line 82 (original), 81 (patched)
> > <https://reviews.apache.org/r/61560/diff/1/?file=1794988#file1794988line82>
> >
> > Have we verified that HS2 is okay with changing the alias? I don't know if maybe they were doing something funny with it.
I could not find a reference for this alias in Hive's codebase.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
-----------------------------------------------------------
On Aug. 10, 2017, 5:21 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 5:21 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java b507c79d44aa2095caf532a5f4d19d20c1a30630
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/1/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 141 (patched)
> > <https://reviews.apache.org/r/61560/diff/1/?file=1794991#file1794991line141>
> >
> > Probably need a space " " between these?
Sure, in my next patch :)
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
-----------------------------------------------------------
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
> > Line 66 (original), 65 (patched)
> > <https://reviews.apache.org/r/61560/diff/1/?file=1794986#file1794986line66>
> >
> > Have we verified that HCat/HMS is okay with changing the alias? I don't know if maybe they were doing something funny with it.
>
> Peter Cseh wrote:
> I know there is a reference for it in Sqoop (https://issues.apache.org/jira/browse/SQOOP-3177) , but It does not seem to work anyways.
>
> Peter Cseh wrote:
> I'm reverting the token alias changes
No, I'm keeping them. See Hive2Credentials for reasons
> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java
> > Line 82 (original), 81 (patched)
> > <https://reviews.apache.org/r/61560/diff/1/?file=1794988#file1794988line82>
> >
> > Have we verified that HS2 is okay with changing the alias? I don't know if maybe they were doing something funny with it.
>
> Peter Cseh wrote:
> I could not find a reference for this alias in Hive's codebase.
We have used the alias " Text fauxAlias = new Text(tk.getKind() + "_" + tk.getService());" before OYA.
https://github.com/apache/oozie/blob/53b1d1e4392cfd35d10860e3b1437273d909a316/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1178-L1196
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
-----------------------------------------------------------
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Aug. 22, 2017, 8:34 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
> > Lines 81 (patched)
> > <https://reviews.apache.org/r/61560/diff/1/?file=1794985#file1794985line81>
> >
> > Is the current time necessary? I think kind and service should be enough (IIRC that's what we use today somewhere). We shouldn't ever need to different tokens with the same kind and service as far as I can tell.
getService() often resulted in an empty string during my testing.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
-----------------------------------------------------------
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review183516
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
Lines 81 (patched)
<https://reviews.apache.org/r/61560/#comment259524>
Is the current time necessary? I think kind and service should be enough (IIRC that's what we use today somewhere). We shouldn't ever need to different tokens with the same kind and service as far as I can tell.
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 66 (original), 65 (patched)
<https://reviews.apache.org/r/61560/#comment259526>
Have we verified that HCat/HMS is okay with changing the alias? I don't know if maybe they were doing something funny with it.
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java
Line 82 (original), 81 (patched)
<https://reviews.apache.org/r/61560/#comment259527>
Have we verified that HS2 is okay with changing the alias? I don't know if maybe they were doing something funny with it.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 141 (patched)
<https://reviews.apache.org/r/61560/#comment259525>
Probably need a space " " between these?
- Robert Kanter
On Aug. 10, 2017, 5:21 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 5:21 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java b507c79d44aa2095caf532a5f4d19d20c1a30630
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/1/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 5, 2017, 5:05 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 555 (patched)
> > <https://reviews.apache.org/r/61560/diff/2/?file=1814414#file1814414line557>
> >
> > Do we always want to get an MR token? Certainly not for Spark or Shell, but Java, Hive (Hive-on-Spark), and Pig (Pig-on-Spark), are more complicated. I guess it's simpler to just always add the MR token.
>
> Peter Cseh wrote:
> I think we should always try to get one. It might be better just to log a warning if we failed instead of failing the action.
I'd rather implement the warning part in a follow-up jira and keep it as is now.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184556
-----------------------------------------------------------
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 5, 2017, 5:05 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 555 (patched)
> > <https://reviews.apache.org/r/61560/diff/2/?file=1814414#file1814414line557>
> >
> > Do we always want to get an MR token? Certainly not for Spark or Shell, but Java, Hive (Hive-on-Spark), and Pig (Pig-on-Spark), are more complicated. I guess it's simpler to just always add the MR token.
I think we should always try to get one. It might be better just to log a warning if we failed instead of failing the action.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184556
-----------------------------------------------------------
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 5, 2017, 5:05 p.m., Robert Kanter wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 196 (patched)
> > <https://reviews.apache.org/r/61560/diff/2/?file=1814419#file1814419line214>
> >
> > Aren't we already in a doAs? It looks like this is so we can remove the AMRM token. I wonder if instead of doing a nested doAs here, we could instead simply remove the AMRM token from the current UGI? If not, then it's fine to leave this as-is.
Without the AM_RM token we can't register and unregister the application in the RM. There may be a cleaner way to do this, but we'll need two doAs calls. With the AM_RM token customer code could request more containers and do such things we don't really want them to do.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184556
-----------------------------------------------------------
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184556
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 555 (patched)
<https://reviews.apache.org/r/61560/#comment260724>
Do we always want to get an MR token? Certainly not for Spark or Shell, but Java, Hive (Hive-on-Spark), and Pig (Pig-on-Spark), are more complicated. I guess it's simpler to just always add the MR token.
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 559 (patched)
<https://reviews.apache.org/r/61560/#comment260725>
Add an RM_DELEGATION_TOKEN...
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 584 (patched)
<https://reviews.apache.org/r/61560/#comment260726>
Add an MR_DELEGATION_TOKEN...
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 605 (patched)
<https://reviews.apache.org/r/61560/#comment260727>
It's good to say this was copied from YARNRunner, but maybe that should be the second sentence. The first sentence should say what this does.
"Gets an MR_DELEEGATION_TOKEN from the Job History Server. Copied from YARNRunner in Hadoop."
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 616 (patched)
<https://reviews.apache.org/r/61560/#comment260728>
Why do we call this to get the renewer instead of using the same code that the other tokens are using?
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 627 (patched)
<https://reviews.apache.org/r/61560/#comment260729>
Same as with the javadoc for getDelegationTokenFromHS - we should have a more descriptive first sentence.
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 653-654 (patched)
<https://reviews.apache.org/r/61560/#comment260730>
While this is true (though HDFS HA vs not HA doesn't matter), and that was a tricky issue, I'm not sure this is the right thing to say here. The main reason to get an HDFS token is so that the action can use HDFS, right? Maybe something like:
"Add a HDFS_DELEGATION_TOKEN to the @{link Credentials} provided. This is also important to ensure that log aggregation works correctly from the NM"
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
Lines 25 (patched)
<https://reviews.apache.org/r/61560/#comment260731>
This change is unrelated.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Line 120 (original), 125 (patched)
<https://reviews.apache.org/r/61560/#comment260732>
I would add a comment mentioning something about the whole AM being executed inside of a doAs.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 196 (patched)
<https://reviews.apache.org/r/61560/#comment260733>
Aren't we already in a doAs? It looks like this is so we can remove the AMRM token. I wonder if instead of doing a nested doAs here, we could instead simply remove the AMRM token from the current UGI? If not, then it's fine to leave this as-is.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Line 354 (original), 381 (patched)
<https://reviews.apache.org/r/61560/#comment260734>
Looks like we can delete this comment
- Robert Kanter
On Sept. 5, 2017, 1:56 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 1:56 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184537
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/ErrorCode.java
Lines 192 (patched)
<https://reviews.apache.org/r/61560/#comment260698>
Indentation
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 544 (patched)
<https://reviews.apache.org/r/61560/#comment260703>
If you added JavaDoc for private methods, pls also add one for this method too
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 553 (patched)
<https://reviews.apache.org/r/61560/#comment260712>
To help current and future Oozie developers, it could be useful to add short comments about why we need these tokens. HDFS is probably obvious, but RM/JHS is not (RM = to be able to fetch child jobs in the AM, JHS = so that we can submit MR jobs).
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 567 (patched)
<https://reviews.apache.org/r/61560/#comment260704>
Typo: "Resoucre"
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 569 (patched)
<https://reviews.apache.org/r/61560/#comment260702>
Indentation
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 591 (patched)
<https://reviews.apache.org/r/61560/#comment260701>
Indentation
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Line 118 (original), 123 (patched)
<https://reviews.apache.org/r/61560/#comment260705>
Explain this comment (I mean, add more info why you think we should keep it here)
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 135 (patched)
<https://reviews.apache.org/r/61560/#comment260711>
Please check if ApplicationConstants.Environment.CONTAINER_ID is available in Hadoop 2.6.0.
If not, then pls extract this to a final field.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 162 (patched)
<https://reviews.apache.org/r/61560/#comment260707>
Space
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 195 (patched)
<https://reviews.apache.org/r/61560/#comment260708>
Explain why we remove RM/AM token here.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 316 (patched)
<https://reviews.apache.org/r/61560/#comment260710>
Spaces
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
Lines 139 (patched)
<https://reviews.apache.org/r/61560/#comment260709>
I feel the need the extract this property key to a public static final field.
- Peter Bacsko
On szept. 5, 2017, 1:56 du, Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated szept. 5, 2017, 1:56 du)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/2/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 6, 2017, 4:29 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 558-570 (patched)
> > <https://reviews.apache.org/r/61560/diff/3/?file=1816365#file1816365line560>
> >
> > I'd extract those three methods, and this one, to a new service called `HadoopTokenService`. It's best readable and tested then.
Will look into this. I don't think I will turn it into an actual Oozie Service. I don't really thing there is much I can test here at the Unit level.
> On Sept. 6, 2017, 4:29 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 584-594 (patched)
> > <https://reviews.apache.org/r/61560/diff/3/?file=1816365#file1816365line586>
> >
> > Getting the `Text`, and calling `credentials.addToken()` can be refactored to a common class, and to get the token for the `Text` can be done within three different `Callable<Token>` calls for the three methods.
I don't see a way to do this while increasing readability.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184696
-----------------------------------------------------------
On Sept. 6, 2017, 2:25 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2017, 2:25 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/3/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Bacsko <pb...@cloudera.com>.
> On szept. 6, 2017, 4:29 du, András Piros wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
> > Lines 558-570 (patched)
> > <https://reviews.apache.org/r/61560/diff/3/?file=1816365#file1816365line560>
> >
> > I'd extract those three methods, and this one, to a new service called `HadoopTokenService`. It's best readable and tested then.
>
> Peter Cseh wrote:
> Will look into this. I don't think I will turn it into an actual Oozie Service. I don't really thing there is much I can test here at the Unit level.
It's not necessarily a bad idea, although the code path we have here is fairly obvious. Still, it's good to have something that gives us some coverage.
I think we can create a class like DelegationTokenRetriever, which receives YarnClient, JHS proxy (or YarnRPC) and FS in the constructor. Then it's easy to unit test it.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184696
-----------------------------------------------------------
On szept. 6, 2017, 2:25 du, Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated szept. 6, 2017, 2:25 du)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/3/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by András Piros <an...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184696
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 558-570 (patched)
<https://reviews.apache.org/r/61560/#comment260886>
I'd extract those three methods, and this one, to a new service called `HadoopTokenService`. It's best readable and tested then.
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 584-594 (patched)
<https://reviews.apache.org/r/61560/#comment260888>
Getting the `Text`, and calling `credentials.addToken()` can be refactored to a common class, and to get the token for the `Text` can be done within three different `Callable<Token>` calls for the three methods.
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 607-617 (patched)
<https://reviews.apache.org/r/61560/#comment260889>
Getting the `Text`, and calling `credentials.addToken()` can be refactored to a common class, and to get the token for the `Text` can be done within three different `Callable<Token>` calls for the three methods.
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 681-688 (patched)
<https://reviews.apache.org/r/61560/#comment260890>
Getting the `Text`, and calling `credentials.addToken()` can be refactored to a common class, and to get the token for the `Text` can be done within three different `Callable<Token>` calls for the three methods.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 148-149 (original), 155-169 (patched)
<https://reviews.apache.org/r/61560/#comment260891>
Should be extracted to another method.
- András Piros
On Sept. 6, 2017, 2:25 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2017, 2:25 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/3/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 12, 2017, 2:12 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java
> > Lines 91 (patched)
> > <https://reviews.apache.org/r/61560/diff/4/?file=1820161#file1820161line91>
> >
> > Not sure if this is an issue, but wondering why a proxy user + doAs is necessary here. Oozie is expected to be authenticated and should be able to retrieve a token without any sort of proxying.
I've just copied this part over from ClientCache. I guess you have to impersonate the user you have to get the tokens to for JHS.
> On Sept. 12, 2017, 2:12 p.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
> > Lines 325 (patched)
> > <https://reviews.apache.org/r/61560/diff/4/?file=1820170#file1820170line350>
> >
> > Is this null-check necessary?
No, I've removed it.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185177
-----------------------------------------------------------
On Sept. 12, 2017, 1:40 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 12, 2017, 1:40 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/4/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185177
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java
Lines 88 (patched)
<https://reviews.apache.org/r/61560/#comment261472>
Nit: @Override
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 1 (patched)
<https://reviews.apache.org/r/61560/#comment261473>
Add ASF license header
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 12 (patched)
<https://reviews.apache.org/r/61560/#comment261474>
Space before "{"
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 25 (patched)
<https://reviews.apache.org/r/61560/#comment261476>
Line exceeds 132 chars
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java
Lines 1 (patched)
<https://reviews.apache.org/r/61560/#comment261477>
Add ASF license header
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java
Lines 1 (patched)
<https://reviews.apache.org/r/61560/#comment261478>
Add ASF license header
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java
Lines 40 (patched)
<https://reviews.apache.org/r/61560/#comment261479>
Line exceeds 132 chars
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java
Lines 90 (patched)
<https://reviews.apache.org/r/61560/#comment261480>
>132 chars here as well
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java
Lines 91 (patched)
<https://reviews.apache.org/r/61560/#comment261482>
Not sure if this is an issue, but wondering why a proxy user + doAs is necessary here. Oozie is expected to be authenticated and should be able to retrieve a token without any sort of proxying.
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1090 (original), 1043 (patched)
<https://reviews.apache.org/r/61560/#comment261483>
Nit: how about addHadoopCredentialPropertiesToActionConf() ?
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java
Lines 1 (patched)
<https://reviews.apache.org/r/61560/#comment261484>
Add ASH header license
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java
Lines 28 (patched)
<https://reviews.apache.org/r/61560/#comment261485>
>132 chars
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java
Lines 33 (patched)
<https://reviews.apache.org/r/61560/#comment261486>
>132 chars
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java
Lines 1 (patched)
<https://reviews.apache.org/r/61560/#comment261487>
Add ASF header
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 325 (patched)
<https://reviews.apache.org/r/61560/#comment261488>
Is this null-check necessary?
- Peter Bacsko
On szept. 12, 2017, 1:40 du, Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated szept. 12, 2017, 1:40 du)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/4/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 18, 2017, 3:25 p.m., Attila Sasvari wrote:
> > core/src/main/java/org/apache/oozie/ErrorCode.java
> > Lines 92 (patched)
> > <https://reviews.apache.org/r/61560/diff/7/?file=1828257#file1828257line92>
> >
> > typo
Where?
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185551
-----------------------------------------------------------
On Sept. 17, 2017, 7:06 a.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2017, 7:06 a.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
> core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/7/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 18, 2017, 3:25 p.m., Attila Sasvari wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java
> > Line 69 (original), 69 (patched)
> > <https://reviews.apache.org/r/61560/diff/7/?file=1828278#file1828278line69>
> >
> > Could you explain why it is necessary to kill child yarn jobs? If it is the first job, then it does not look necessary
It's a common pattern across *Mains to call this. I don't know why it's not inside ShellMain, but you're right, this is not the patch to put it in there.
> On Sept. 18, 2017, 3:25 p.m., Attila Sasvari wrote:
> > sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
> > Line 234 (original), 233 (patched)
> > <https://reviews.apache.org/r/61560/diff/7/?file=1828280#file1828280line236>
> >
> > what do you test here? Is the testname ``testLauncherJobConfCannotBeLoaded`` still correct?
We're reading the confign in LauncherAM.main, but there is no way we can inject the localFsOperationsMock in there so I'm calling the method that should throw the exception.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185551
-----------------------------------------------------------
On Sept. 17, 2017, 7:06 a.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2017, 7:06 a.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
> core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/7/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Robert Kanter <rk...@cloudera.com>.
> On Sept. 18, 2017, 3:25 p.m., Attila Sasvari wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java
> > Line 69 (original), 69 (patched)
> > <https://reviews.apache.org/r/61560/diff/7/?file=1828278#file1828278line69>
> >
> > Could you explain why it is necessary to kill child yarn jobs? If it is the first job, then it does not look necessary
>
> Peter Cseh wrote:
> It's a common pattern across *Mains to call this. I don't know why it's not inside ShellMain, but you're right, this is not the patch to put it in there.
The purpose of this is for the case where the launcher gets restarted by Yarn (which can happen for a variety of reasons). Oozie can't resume child jobs that were in the middle other than the MR action because it has no idea what it's doing. We decided that the best thing to do would be to kill all the children and start over - that's what this does. Otherwise, you'd end up with duplicate jobs running, which could cause problems. Oozie doesn't do this in the Shell Action (and maybe also the Java Action) because Oozie doesn't manage the child jobs from these action types - that's up to the user.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185551
-----------------------------------------------------------
On Sept. 19, 2017, 1:59 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 19, 2017, 1:59 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 9d1afb51976fd8ebef4d63cda1ea2ad58de32566
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 02e60c01db2221e7e107762caa4d0beb3fe53ca5
> core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
> core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 3e6ffc90cc53debac678ac79a83864eaad596730
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/10/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185551
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/ErrorCode.java
Lines 92 (patched)
<https://reviews.apache.org/r/61560/#comment261858>
typo
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java
Lines 82 (patched)
<https://reviews.apache.org/r/61560/#comment261871>
getDelegationTokenFromJHS
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1092 (original), 1045 (patched)
<https://reviews.apache.org/r/61560/#comment261864>
extract method
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 177 (patched)
<https://reviews.apache.org/r/61560/#comment261869>
``Token \"<token>\" is not removed : <decoded_identifier>`` or something would help to understand what is happenning
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java
Line 69 (original), 69 (patched)
<https://reviews.apache.org/r/61560/#comment261878>
Could you explain why it is necessary to kill child yarn jobs? If it is the first job, then it does not look necessary
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java
Line 234 (original), 233 (patched)
<https://reviews.apache.org/r/61560/#comment261868>
what do you test here? Is the testname ``testLauncherJobConfCannotBeLoaded`` still correct?
- Attila Sasvari
On Sept. 17, 2017, 7:06 a.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2017, 7:06 a.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
> core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/7/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185729
-----------------------------------------------------------
Ship it!
Ship It!
- Attila Sasvari
On Sept. 19, 2017, 1:59 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 19, 2017, 1:59 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 9d1afb51976fd8ebef4d63cda1ea2ad58de32566
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 02e60c01db2221e7e107762caa4d0beb3fe53ca5
> core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
> core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 3e6ffc90cc53debac678ac79a83864eaad596730
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/10/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185785
-----------------------------------------------------------
Ship it!
Ship It!
- Peter Bacsko
On szept. 19, 2017, 1:59 du, Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated szept. 19, 2017, 1:59 du)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 9d1afb51976fd8ebef4d63cda1ea2ad58de32566
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 02e60c01db2221e7e107762caa4d0beb3fe53ca5
> core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
> core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 3e6ffc90cc53debac678ac79a83864eaad596730
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/10/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 19, 2017, 1:59 p.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 9d1afb51976fd8ebef4d63cda1ea2ad58de32566
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 02e60c01db2221e7e107762caa4d0beb3fe53ca5
core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 3e6ffc90cc53debac678ac79a83864eaad596730
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/10/
Changes: https://reviews.apache.org/r/61560/diff/9-10/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 19, 2017, 8:14 a.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 9d1afb51976fd8ebef4d63cda1ea2ad58de32566
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 02e60c01db2221e7e107762caa4d0beb3fe53ca5
core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 3e6ffc90cc53debac678ac79a83864eaad596730
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/9/
Changes: https://reviews.apache.org/r/61560/diff/8-9/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 19, 2017, 7:33 a.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/8/
Changes: https://reviews.apache.org/r/61560/diff/7-8/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 17, 2017, 7:06 a.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/7/
Changes: https://reviews.apache.org/r/61560/diff/6-7/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 16, 2017, 8:18 p.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java a7d6c18a645b50fc35657515480da9967fa74e2e
core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java cf77f18c4bdbe95ea66f63aa115ce6f34c90c855
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/6/
Changes: https://reviews.apache.org/r/61560/diff/5-6/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 13, 2017, 1:06 p.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/5/
Changes: https://reviews.apache.org/r/61560/diff/4-5/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 12, 2017, 5:29 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java
> > Lines 23 (patched)
> > <https://reviews.apache.org/r/61560/diff/4/?file=1820158#file1820158line23>
> >
> > Do we still need the JT?
I've just copied over the code from HadoopAccessorService. I guess we can drop it.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185217
-----------------------------------------------------------
On Sept. 12, 2017, 1:40 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 12, 2017, 1:40 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/4/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review185217
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 15 (patched)
<https://reviews.apache.org/r/61560/#comment261502>
Add _an_ HDFS_DELEGATION_TOKEN
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 36 (patched)
<https://reviews.apache.org/r/61560/#comment261503>
LOG.debug("...");
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java
Lines 23 (patched)
<https://reviews.apache.org/r/61560/#comment261504>
Do we still need the JT?
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java
Lines 31 (patched)
<https://reviews.apache.org/r/61560/#comment261506>
This Jvadoc comment seems to have gotten lost...
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java
Lines 19 (patched)
<https://reviews.apache.org/r/61560/#comment261508>
The syntax is actually
````
{@link Credentials}
````
(The '@' goes inside)
If done correctly, the IDE should make it a clickable link if you hold CMD
- Robert Kanter
On Sept. 12, 2017, 1:40 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 12, 2017, 1:40 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
> core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/4/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 12, 2017, 1:40 p.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProperties.java 20f93ce57618a529b1bd83b398add2cd1d96b36e
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 52abbf17afa71e702d7871d10a4a190a4d21e47b
core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HadoopTokenHelper.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JHSCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
core/src/main/java/org/apache/oozie/action/hadoop/YarnRMCredentials.java PRE-CREATION
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/action/hadoop/TestHadoopTokenHelper.java PRE-CREATION
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/4/
Changes: https://reviews.apache.org/r/61560/diff/3-4/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 7, 2017, 10:13 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/ErrorCode.java
> > Lines 192-193 (patched)
> > <https://reviews.apache.org/r/61560/diff/3/?file=1816359#file1816359line192>
> >
> > In both of these, won''t should be won't
Actually, org.apache.oozie.TestErrorCode. testEscapedSingleQuotes() will fail if I write won't here.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184891
-----------------------------------------------------------
On Sept. 6, 2017, 2:25 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2017, 2:25 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/3/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/#review184891
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/ErrorCode.java
Lines 192-193 (patched)
<https://reviews.apache.org/r/61560/#comment261088>
In both of these, won''t should be won't
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
Lines 26 (patched)
<https://reviews.apache.org/r/61560/#comment261089>
I don't think we want/need this import here...
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
Lines 545 (patched)
<https://reviews.apache.org/r/61560/#comment261092>
This is a little silly, but let's change all references of "HistoryServer" to either "Job History Server" or "JHS".
Looking forward, I don't want us to get confused with the ATS, which is often called the History Server (it was named AHS - Application History Server at one point). And there's ATSv1, ATSv1.5, and ATSv2 to make things more complicated :)
- Robert Kanter
On Sept. 6, 2017, 2:25 p.m., Peter Cseh wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61560/
> -----------------------------------------------------------
>
> (Updated Sept. 6, 2017, 2:25 p.m.)
>
>
> Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-2909
> https://issues.apache.org/jira/browse/OOZIE-2909
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Changing UGI calls and cleaning up things around requesting tokens.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
> core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
> core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
> core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
> core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
>
>
> Diff: https://reviews.apache.org/r/61560/diff/3/
>
>
> Testing
> -------
>
> We're running tests on a Kerberized cluster with these changes.
>
>
> Thanks,
>
> Peter Cseh
>
>
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 6, 2017, 2:25 p.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/3/
Changes: https://reviews.apache.org/r/61560/diff/2-3/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh
Re: Review Request 61560: OOZIE-2909 LauncherAM: rewrite UGI calls
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61560/
-----------------------------------------------------------
(Updated Sept. 5, 2017, 1:56 p.m.)
Review request for oozie, Attila Sasvari, Peter Bacsko, Robert Kanter, and Rohini Palaniswamy.
Bugs: OOZIE-2909
https://issues.apache.org/jira/browse/OOZIE-2909
Repository: oozie-git
Description
-------
Changing UGI calls and cleaning up things around requesting tokens.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/ErrorCode.java 662e1edc9c4b23b3606c751bf5ed4b531ee7ac62
core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java 5ca8d3e80942840a5bba51a06694350929044042
core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 9804c7b673a899cd9c778850c9a64731cc642fb3
core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 4add5f14cb20ab77c755b81eff939335c0c5fddc
core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 0b495f75842b0033a3337ef6728a334bf5651770
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 187cee2290ba4bbe358913d28a052d9c75f2369a
core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 960c2f9750062c98fdf7a2b456d3668069e07ca5
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java e6c9d04653be984bdd5aa77bee5f28b187e3de5d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java b4cbb4b1d9b52aaebfb562c02edc91e15a2a9a2e
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java 874d371664ced3ae19a7b5d615c67d4c72e7cfdd
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java bde7f1daf123d663d47f5d2fb2350cfa1ac33ed8
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java 68c0f4bafe0322ff0d1007297cd6e8f38c92020a
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java 37af3dd9ed43f4504ff3909484237e2d05665c6c
Diff: https://reviews.apache.org/r/61560/diff/2/
Changes: https://reviews.apache.org/r/61560/diff/1-2/
Testing
-------
We're running tests on a Kerberized cluster with these changes.
Thanks,
Peter Cseh