You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2015/11/14 00:11:41 UTC
Review Request 40315: HIVE-12341 LLAP security
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 01cd731
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127510#file1127510line2361>
> >
> > Don't think the default value - "*" - has any significance here. Replace by null - to avoid confusion.
>
> Sergey Shelukhin wrote:
> Following in the footsteps of slider
>
> Siddharth Seth wrote:
> Can we drop the * in favor of a null. (or no default if Hive supports that).
Why?
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java, line 129
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127516#file1127516line129>
> >
> > This could be moved into it's own protocol (but listening on the same server).
> >
> > The methods so far are for access from the AM.
> >
> > getTokens is to be used by Clients.
> >
> > What that also allows is for the annotations to change.
> > getTokens() - protected by Kerberos, and cannot be obtained using a token.
> > Remaining methods - require a token.
>
> Sergey Shelukhin wrote:
> why add an extra protocol? it seems like most services don't handle tokens like this. HDFS gives out tokens as part of normal interface.
>
> Siddharth Seth wrote:
> HDFS also doesn't have a separation of API vs server side jars.
>
> Reason for separation is that they're very different operations - one relates to exuecuting and tracking work, the other to access. Consider the case where there is a central service which is responsible for handing out these delegation tokens (rotation, etc etc). That will definitely not implement the submitWork APIs, and the daemons cannot implement the getTokns at that point.
>
> Doesn't need to be done rightnow, and can be changed later since this API is private for the moment (It is private, right?)
>
> Re-opened primarily for the "Is this private" bit.
Added new protocol. Will test after the weekend.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107706
-----------------------------------------------------------
On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 11:11 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127510#file1127510line2361>
> >
> > Don't think the default value - "*" - has any significance here. Replace by null - to avoid confusion.
Following in the footsteps of slider
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java, line 71
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127512#file1127512line71>
> >
> > Does a renewer for a token type have to be specified ?
renewer is set elsewhere
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java, line 129
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127516#file1127516line129>
> >
> > This could be moved into it's own protocol (but listening on the same server).
> >
> > The methods so far are for access from the AM.
> >
> > getTokens is to be used by Clients.
> >
> > What that also allows is for the annotations to change.
> > getTokens() - protected by Kerberos, and cannot be obtained using a token.
> > Remaining methods - require a token.
why add an extra protocol? it seems like most services don't handle tokens like this. HDFS gives out tokens as part of normal interface.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java, line 134
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127517#file1127517line134>
> >
> > Sanity checks for the values. Empty strings are not allowed.
these are passed as defaults to ZK config getters; then, they are checked when attempting to log in
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java, line 251
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127517#file1127517line251>
> >
> > YARN can take care of renewing delegation tokens - assuming the service supports it (i.e. the ZKSecretManager on one of the LLAP instances or a direct connection to ZK from the RM - but that isn't a good idea).
> > Eventually, I believe the renweer would need to change to the RM service user.
Hmm... not sure how this would work. Can you file a follow-up JIRA? By MR logic, the renewer would be a central job manager, e.g. HS2
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java, line 26
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127519#file1127519line26>
> >
> > How is the default value picked up ? (definitely not from the hive conf)
> > OR
> > What is the default value - "*" or " ".
> > I'm not sure how other services handle this - but this can be set to " " by default on secure clusters, and "*" on non-secure clusters.
>From the conf passed to refreshServiceAcl it looks like.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java, line 32
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127519#file1127519line32>
> >
> > clone not required.
that looks like what other services do
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java, line 53
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127520#file1127520line53>
> >
> > This would matter when running under HiveServer ? or is the synchronization in LlapIoProxy taking care of this ?
shouldn't matter
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java, line 105
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127525#file1127525line105>
> >
> > Stop logging the token.
This part doesn't contain any secrets, changed to debug
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java, line 511
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127525#file1127525line511>
> >
> > Required for each host separately ? Setting the host may not be required.
I think it is. Server has to run the principal with _HOST for hadoop IPC to work
On Nov. 24, 2015, 3:56 a.m., Sergey Shelukhin wrote:
> > Haven't looked at the details of the ZKSecretManager - but it looks like the Tokens issued by any of the LLAP instances can be used by an application to communicate with all other instances.
> > Also, are the tokens the same for different applications ?
Yes, and no.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107706
-----------------------------------------------------------
On Nov. 16, 2015, 7:45 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 16, 2015, 7:45 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 838f25c
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Siddharth Seth <ss...@apache.org>.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127510#file1127510line2361>
> >
> > Don't think the default value - "*" - has any significance here. Replace by null - to avoid confusion.
>
> Sergey Shelukhin wrote:
> Following in the footsteps of slider
Can we drop the * in favor of a null. (or no default if Hive supports that).
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java, line 71
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127512#file1127512line71>
> >
> > Does a renewer for a token type have to be specified ?
>
> Sergey Shelukhin wrote:
> renewer is set elsewhere
Not the user which does the renewal. Implementation of the Renewer interface. See TrivialRenewer in Token.java.
Not sure if one is required for each token kind - would be safer to add one which accepts the LLAP token kind and says it's not managed for now.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java, line 129
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127516#file1127516line129>
> >
> > This could be moved into it's own protocol (but listening on the same server).
> >
> > The methods so far are for access from the AM.
> >
> > getTokens is to be used by Clients.
> >
> > What that also allows is for the annotations to change.
> > getTokens() - protected by Kerberos, and cannot be obtained using a token.
> > Remaining methods - require a token.
>
> Sergey Shelukhin wrote:
> why add an extra protocol? it seems like most services don't handle tokens like this. HDFS gives out tokens as part of normal interface.
HDFS also doesn't have a separation of API vs server side jars.
Reason for separation is that they're very different operations - one relates to exuecuting and tracking work, the other to access. Consider the case where there is a central service which is responsible for handing out these delegation tokens (rotation, etc etc). That will definitely not implement the submitWork APIs, and the daemons cannot implement the getTokns at that point.
Doesn't need to be done rightnow, and can be changed later since this API is private for the moment (It is private, right?)
Re-opened primarily for the "Is this private" bit.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java, line 251
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127517#file1127517line251>
> >
> > YARN can take care of renewing delegation tokens - assuming the service supports it (i.e. the ZKSecretManager on one of the LLAP instances or a direct connection to ZK from the RM - but that isn't a good idea).
> > Eventually, I believe the renweer would need to change to the RM service user.
>
> Sergey Shelukhin wrote:
> Hmm... not sure how this would work. Can you file a follow-up JIRA? By MR logic, the renewer would be a central job manager, e.g. HS2
YARN (The RM) is given tokens for a job. It checks whether these tokens are managed or not (via the Renewer interface), and takes care of renewing them while the job is alive. That's what is done for HDFS delegation tokens anyway - renewed by the RM every day while the job is running.
The same could be done for LLAP - but this is for later.
Opening a jira to track renewal of tokens.
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java, line 26
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127519#file1127519line26>
> >
> > How is the default value picked up ? (definitely not from the hive conf)
> > OR
> > What is the default value - "*" or " ".
> > I'm not sure how other services handle this - but this can be set to " " by default on secure clusters, and "*" on non-secure clusters.
>
> Sergey Shelukhin wrote:
> From the conf passed to refreshServiceAcl it looks like.
What if the fiels is not set ?
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java, line 32
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127519#file1127519line32>
> >
> > clone not required.
>
> Sergey Shelukhin wrote:
> that looks like what other services do
Looking at yarn services, this isn't required. A final array is being returned - this was likely changed to get past findbugs warnings, and ignored in YARN
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java, line 53
> > <https://reviews.apache.org/r/40315/diff/2/?file=1127520#file1127520line53>
> >
> > This would matter when running under HiveServer ? or is the synchronization in LlapIoProxy taking care of this ?
>
> Sergey Shelukhin wrote:
> shouldn't matter
why ? Multiple threads launching AMs and accessing Proxy.getTokens()
- Siddharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107706
-----------------------------------------------------------
On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 11:11 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Siddharth Seth <ss...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107706
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2348)
<https://reviews.apache.org/r/40315/#comment166963>
Description needs fixing.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 2351)
<https://reviews.apache.org/r/40315/#comment166964>
Don't think the default value - "*" - has any significance here. Replace by null - to avoid confusion.
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java (line 76)
<https://reviews.apache.org/r/40315/#comment166965>
Rename class to LLAPProxy ? It's no longer limited to IO only.
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java (line 44)
<https://reviews.apache.org/r/40315/#comment166966>
This and readFields - eventually should be implemented using a Protobuf payload. Allows the token to evolve during rolling upgrades.
Separate jira.
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java (line 71)
<https://reviews.apache.org/r/40315/#comment166967>
Does a renewer for a token type have to be specified ?
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java (line 128)
<https://reviews.apache.org/r/40315/#comment166968>
This could be moved into it's own protocol (but listening on the same server).
The methods so far are for access from the AM.
getTokens is to be used by Clients.
What that also allows is for the annotations to change.
getTokens() - protected by Kerberos, and cannot be obtained using a token.
Remaining methods - require a token.
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java (line 129)
<https://reviews.apache.org/r/40315/#comment166969>
Rename to getDelegationToken ?
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java (line 79)
<https://reviews.apache.org/r/40315/#comment166970>
throws IOException not required.
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java (line 133)
<https://reviews.apache.org/r/40315/#comment166971>
Sanity checks for the values. Empty strings are not allowed.
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java (line 157)
<https://reviews.apache.org/r/40315/#comment166972>
Avoid using the ZK property names. Instead, the properties defined for LLAP should be used.
(ZK properties could leak in from some other service which uses the same SecretManager)
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java (line 164)
<https://reviews.apache.org/r/40315/#comment166973>
New instances of Configuration if doing a set (this config is shared by several services)
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java (line 250)
<https://reviews.apache.org/r/40315/#comment166974>
YARN can take care of renewing delegation tokens - assuming the service supports it (i.e. the ZKSecretManager on one of the LLAP instances or a direct connection to ZK from the RM - but that isn't a good idea).
Eventually, I believe the renweer would need to change to the RM service user.
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java (line 26)
<https://reviews.apache.org/r/40315/#comment166975>
How is the default value picked up ? (definitely not from the hive conf)
OR
What is the default value - "*" or " ".
I'm not sure how other services handle this - but this can be set to " " by default on secure clusters, and "*" on non-secure clusters.
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java (line 32)
<https://reviews.apache.org/r/40315/#comment166976>
clone not required.
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java (line 53)
<https://reviews.apache.org/r/40315/#comment166977>
This would matter when running under HiveServer ? or is the synchronization in LlapIoProxy taking care of this ?
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java (line 59)
<https://reviews.apache.org/r/40315/#comment166978>
final. Also retryPolicy.
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java (line 67)
<https://reviews.apache.org/r/40315/#comment166979>
make configurable ?
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java (line 94)
<https://reviews.apache.org/r/40315/#comment166980>
Lots of TODOs - fix / convert to jira with a reference to the jira number ?
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java (line 96)
<https://reviews.apache.org/r/40315/#comment166981>
Stop logging the token.
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java (line 100)
<https://reviews.apache.org/r/40315/#comment166983>
Can be accessed via TaskCommunicatorContext. More on this in SessionState.
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java (line 105)
<https://reviews.apache.org/r/40315/#comment166982>
Stop logging the token.
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java (line 511)
<https://reviews.apache.org/r/40315/#comment166984>
Required for each host separately ? Setting the host may not be required.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java (line 319)
<https://reviews.apache.org/r/40315/#comment166985>
Emailing the token to the AM in the Configuration is very avoidable.
The token can be provided to Tez while creating the TezClient. TezClient will make this available to the TaskCommunicator via the TaskCommunicatorContext.getCredentials().
See TokenCache.get/setSessionToken.
The static string defined in HiveConf to send this token could be shortened and moved out of HiveConf.
Haven't looked at the details of the ZKSecretManager - but it looks like the Tokens issued by any of the LLAP instances can be used by an application to communicate with all other instances.
Also, are the tokens the same for different applications ?
- Siddharth Seth
On Nov. 16, 2015, 7:45 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 16, 2015, 7:45 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 838f25c
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Siddharth Seth <ss...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107870
-----------------------------------------------------------
Token and credentials usage looks good to me. Looking at other review comments in a bit.
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java (line 100)
<https://reviews.apache.org/r/40315/#comment167199>
Precondition check for security==true and token!=null
- Siddharth Seth
On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 11:11 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Siddharth Seth <ss...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107877
-----------------------------------------------------------
- Siddharth Seth
On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 11:11 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Lefty Leverenz <le...@gmail.com>.
> On Nov. 30, 2015, 7:49 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 2377-2378
> > <https://reviews.apache.org/r/40315/diff/6/?file=1147130#file1147130line2377>
> >
> > Should this have a TimeValidator (like the next two Spark parameters, lines 2372-2376)?
> >
> > Also, the default is 14 * 24 * 3600 but the comment says DelegationTokenManager default is 1 week instead of 2 weeks, so is DelegationTokenManager something different from this parameter?
>
> Sergey Shelukhin wrote:
> Yes, that is the hadoop delegation token manager. This is a comment for the background of how the ballpark for the default value was chosen.
Okay, thanks Sergey.
- Lefty
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108296
-----------------------------------------------------------
On Nov. 30, 2015, 7:50 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2015, 7:50 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 4525ab9
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On Nov. 30, 2015, 7:49 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 2377-2378
> > <https://reviews.apache.org/r/40315/diff/6/?file=1147130#file1147130line2377>
> >
> > Should this have a TimeValidator (like the next two Spark parameters, lines 2372-2376)?
> >
> > Also, the default is 14 * 24 * 3600 but the comment says DelegationTokenManager default is 1 week instead of 2 weeks, so is DelegationTokenManager something different from this parameter?
Yes, that is the hadoop delegation token manager. This is a comment for the background of how the ballpark for the default value was chosen.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108296
-----------------------------------------------------------
On Nov. 26, 2015, 1:38 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2015, 1:38 a.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 4525ab9
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Lefty Leverenz <le...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108296
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2357 - 2358)
<https://reviews.apache.org/r/40315/#comment167688>
Spell out ZooKeeper in the description.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2359 - 2361)
<https://reviews.apache.org/r/40315/#comment167689>
Spell out ZooKeeper in the description.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2362 - 2363)
<https://reviews.apache.org/r/40315/#comment167692>
Spell out ZooKeeper in the description.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 2367 - 2368)
<https://reviews.apache.org/r/40315/#comment167690>
Should this have a TimeValidator (like the next two Spark parameters, lines 2372-2376)?
Also, the default is 14 * 24 * 3600 but the comment says DelegationTokenManager default is 1 week instead of 2 weeks, so is DelegationTokenManager something different from this parameter?
- Lefty Leverenz
On Nov. 26, 2015, 1:38 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2015, 1:38 a.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 4525ab9
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Lefty Leverenz <le...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review108467
-----------------------------------------------------------
Ship it!
Ship It!
- Lefty Leverenz
On Nov. 30, 2015, 7:50 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2015, 7:50 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 4525ab9
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Siddharth Seth <ss...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review109407
-----------------------------------------------------------
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java (line 47)
<https://reviews.apache.org/r/40315/#comment168902>
rename to getManagementPort ?
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java (line 156)
<https://reviews.apache.org/r/40315/#comment168905>
Not for this jira. While the daemons serve the management protocol - the management protocol and daemon protocol could potentially run on the same server with the same port. This saveson the number of handler threads which end up being created.
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java (line 159)
<https://reviews.apache.org/r/40315/#comment168908>
In the interim - a separate configuration for management handlers would help, which could be set to a low values - maybe 2.
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java (line 66)
<https://reviews.apache.org/r/40315/#comment168911>
New jira to make this configurable
- Siddharth Seth
On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2015, 9:40 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java f116de4
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java a085427
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java 2673ad7
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java dbdf571
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java b93650d
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 8144165
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 52ba360
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java bf8a673
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Siddharth Seth <ss...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review109414
-----------------------------------------------------------
Ship it!
Looks good. Small nits in the last patch - mostly around creating new jiras.
The compiled proto file seems to be missing from the RB patch. Assuming that'll be part of the commit.
- Siddharth Seth
On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2015, 9:40 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java f116de4
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java a085427
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java 2673ad7
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java dbdf571
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java b93650d
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 8144165
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 52ba360
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java bf8a673
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Siddharth Seth <ss...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review109413
-----------------------------------------------------------
Ship it!
Ship It!
- Siddharth Seth
On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2015, 9:40 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java f116de4
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java a085427
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java 2673ad7
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java dbdf571
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java b93650d
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 8144165
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 52ba360
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java bf8a673
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a
> ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3
> ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Dec. 5, 2015, 9:40 p.m.)
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java f116de4
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java a085427
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java 2673ad7
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java dbdf571
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java b93650d
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 8144165
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 52ba360
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java bf8a673
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1a9469a
ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Dec. 3, 2015, 11:07 p.m.)
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f96071
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java f116de4
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java 34e0682
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapYarnRegistryImpl.java d474b6f
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 4525ab9
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 7c38dc3
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Nov. 30, 2015, 7:50 p.m.)
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 4525ab9
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Nov. 26, 2015, 1:38 a.m.)
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java db942b0
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapManagementProtocolBlockingPB.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapManagementProtocolClientImpl.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/MiniLlapCluster.java 4525ab9
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Nov. 24, 2015, 11:11 p.m.)
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Nov. 24, 2015, 11:08 p.m.)
Review request for hive, Gopal V and Siddharth Seth.
Changes
-------
same as previous minus the generated code
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Nov. 24, 2015, 11:06 p.m.)
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java b044df9
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95
ql/src/java/org/apache/hadoop/hive/ql/exec/GlobalWorkMapFactory.java 59ee347
ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java 3d9771a
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 914b4e7
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java efcf88c
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3feab1a
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/#review107670
-----------------------------------------------------------
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java (line 31)
<https://reviews.apache.org/r/40315/#comment166900>
TODO: need to clone, same below
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java (line 96)
<https://reviews.apache.org/r/40315/#comment166902>
TODO debug
- Sergey Shelukhin
On Nov. 16, 2015, 7:45 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40315/
> -----------------------------------------------------------
>
> (Updated Nov. 16, 2015, 7:45 p.m.)
>
>
> Review request for hive, Gopal V and Siddharth Seth.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 838f25c
> llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
> llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
> llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
> llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
> llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
> ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
> serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
>
> Diff: https://reviews.apache.org/r/40315/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 40315: HIVE-12341 LLAP security
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40315/
-----------------------------------------------------------
(Updated Nov. 16, 2015, 7:45 p.m.)
Review request for hive, Gopal V and Siddharth Seth.
Repository: hive-git
Description
-------
see JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 838f25c
llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIoProxy.java 4c31e32
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java PRE-CREATION
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/LlapDaemonProtocolBlockingPB.java 5ad2344
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 98b1ccd
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolClientImpl.java 4b13277
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemonProtocolServerImpl.java 784c631
llap-server/src/java/org/apache/hadoop/hive/llap/protocol/LlapTaskUmbilicalProtocol.java fae7654
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapDaemonPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapServerSecurityInfo.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapTokenSelector.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java d327fc0
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapUmbilicalPolicyProvider.java PRE-CREATION
llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/TaskCommunicator.java 33e998c
llap-server/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo PRE-CREATION
llap-server/src/protobuf/LlapDaemonProtocol.proto 0ba6acf
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java 8d45c95
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9ab3e98
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java 07f26be
serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java 9269ff4
Diff: https://reviews.apache.org/r/40315/diff/
Testing
-------
Thanks,
Sergey Shelukhin