adding new tokens to existing Hconnection instances

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi,

The short question: 
Is there any way to update delegation tokens of an existing active HConnection instance?

Long story:
This is a follow up to http://osdir.com/ml/general/2014-08/msg27210.html. To recap storm is trying to get delegation tokens from Hbase on behalf of a user who is trying to run a storm topology  and then distribute these tokens to all the worker that would run the user topology. I was able to get delegation tokens using TokenUtil.obtainAndCacheToken(hbaseConf, proxyUser) and then read the token from the user credentials. I was hoping on worker host the user code will just add these tokens to the User’s subject object and then call createConnection(Configuration conf, User user).  

This seem to work fine until the token expires. Because Hbase do not support token renewal , we have a renewal scheme where master just asks for new tokens at regular interval and then pushes it to worker which again adds it to ugi’s subject. 

During the code review of above feature it was pointed out that HConnection implementation only contacts the UGI during initial connection establishment and then caches it. This means even if UGI is updated by adding new tokens the connection will not see these changes and will end up using old expired tokens. I could not actually verify the behavior because token expiry is 7 days(anyway to change this?) and my token.cancel() methods are failing.  

I looked at RPCClient and HConnectionImplementation, and they both seem to have a user instance which is set to the user instance passed during “createConnection" call.  The only place the token are accessed are during construction of Connection objects in RPCClient. Have I missed something obvious here or there is no other alternative when token expires other then abandoning all objects and connections and recreating a Connection instance?

Thanks
Parth
-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: adding new tokens to existing Hconnection instances

[Gary Helmling <gh...@gmail.com>]

Authentication is only performed during RPC connection setup.  So
there isn't really a concept of token expiration for an existing RPC
connection.  The connection will be authenticated (will not expire)
for as long as it's held open.  When it's closed and re-opened, it
should pick up the latest tokens associated with the UGI.  So I think
this should work as expected, as long you you are adding the new
tokens to the existing UGI.

By the way, when testing this, you can set the value for
"hbase.auth.token.max.lifetime" to a smaller value (say 3600000 for
one hour) in your HBase configuration.  This would make it easier to
manually test riding over a token expiration.

On Wed, Sep 10, 2014 at 11:06 AM, Parth Brahmbhatt
<pb...@hortonworks.com> wrote:
> Hi,
>
> The short question:
> Is there any way to update delegation tokens of an existing active HConnection instance?
>
> Long story:
> This is a follow up to http://osdir.com/ml/general/2014-08/msg27210.html. To recap storm is trying to get delegation tokens from Hbase on behalf of a user who is trying to run a storm topology  and then distribute these tokens to all the worker that would run the user topology. I was able to get delegation tokens using TokenUtil.obtainAndCacheToken(hbaseConf, proxyUser) and then read the token from the user credentials. I was hoping on worker host the user code will just add these tokens to the User’s subject object and then call createConnection(Configuration conf, User user).
>
> This seem to work fine until the token expires. Because Hbase do not support token renewal , we have a renewal scheme where master just asks for new tokens at regular interval and then pushes it to worker which again adds it to ugi’s subject.
>
> During the code review of above feature it was pointed out that HConnection implementation only contacts the UGI during initial connection establishment and then caches it. This means even if UGI is updated by adding new tokens the connection will not see these changes and will end up using old expired tokens. I could not actually verify the behavior because token expiry is 7 days(anyway to change this?) and my token.cancel() methods are failing.
>
> I looked at RPCClient and HConnectionImplementation, and they both seem to have a user instance which is set to the user instance passed during “createConnection" call.  The only place the token are accessed are during construction of Connection objects in RPCClient. Have I missed something obvious here or there is no other alternative when token expires other then abandoning all objects and connections and recreating a Connection instance?
>
> Thanks
> Parth
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.