You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2016/06/19 05:26:20 UTC
how to increase lifetime of hdfs delegation tokens ?
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Raja,
Do you have admin access to the cluster? If so look for the
properties yarn.resourcemanager.delegation.token.max-lifetime and
dfs.namenode.delegation.token.max-lifetime in your Hadoop configuration.
They should currently be set to 7 days. This is what is making the tokens
expire after 7days. For testing you can change it to half hour for example
and restart hadoop services. You can now see without waiting for a long
time if the application is getting new tokens and running with them.
Thanks
On Tue, Jul 5, 2016 at 2:23 PM, Raja.Aravapalli <Ra...@target.com>
wrote:
>
> Hi Pramod,
>
> As suggested in step2, I added the properties you provided below, and ran
> a new job, to see the debug lines "number of tokens: " and "updated
> token: "
>
> And, as you said, I can see those lines in the application log now..
>
> Does this mean, the renewal is working as expected ? So, will my
> application run continously even after 7days now ? We have only changed
> refersh times here… how does this ensure the application runs indefinite
> time…
>
>
>
>
> Regards,
> Raja.
>
> From: "Raja.Aravapalli" <Ra...@target.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Saturday, July 2, 2016 at 9:52 AM
>
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
>
> Thanks Pramod, I will these suggestions and let you know. Thanks a lot.
>
>
> Regards,
> Raja.
>
> From: Pramod Immaneni <pr...@datatorrent.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Friday, July 1, 2016 at 8:36 PM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
> Hi Raja,
>
> Some questions for you and also some options for you to verify without
> waiting for a long time.
>
> 1. Do you see a warning message like ""No keytab specified for refreshing
> tokens, application may not be able to run indefinitely" when application
> is being launched from command line.
>
> 2. For testing, set the parameters below in your dt-site.xml and launch
> your application. This will make the application think that the tokens are
> only valid for 5 minutes and within 5 * 0.7 = 3.5 minutes (token refresh
> factor is 0.7) the application should try to get new tokens. It should
> print the debug lines "number of tokens: " and "updated token: " in the
> application master logs. The application master logs are in the log file of
> the first container of the application. Let me know if you see those log
> lines.
>
> <property>
> <name>dt.resourcemanager.delegation.token.max-lifetime</name>
> <value>300000</value>
> </property>
>
> <property>
> <name>dt.namenode.delegation.token.max-lifetime</name>
> <value>300000</value>
> </property>
>
> <property>
> <name>dt.attr.DEBUG</name>
> <value>true</value>
> </property>
>
> For more information about application auto-fetching new tokens read here
>
> https://github.com/apache/apex-core/blob/master/docs/security.md
>
> Thanks
>
> On Fri, Jul 1, 2016 at 1:08 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>> Thanks a lot Pramod. Will wait for your response.
>>
>>
>> Regards,
>> Raja.
>>
>> From: Pramod Immaneni <pr...@datatorrent.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Friday, July 1, 2016 at 10:56 AM
>>
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>> Hi Raja,
>>
>> Let me look at this and get back to you.
>>
>> Thanks
>>
>> On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <
>> Raja.Aravapalli@target.com> wrote:
>>
>>>
>>> Can someone pls help me, how can I ensure, my apex application doesn’t
>>> fail after 7days…
>>>
>>> Thanks a lot.
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: "Raja.Aravapalli" <Ra...@target.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Thursday, June 30, 2016 at 6:06 AM
>>>
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>>
>>> Hi,
>>>
>>> I triggered my application by specifying properties, “
>>> dt.authentication.principal” & “dt.authentication.keytab” , BUT, did
>>> not specify the property “dt.authentication.store.keytab”.
>>>
>>> I also observed the keytab is copied to hdfs location
>>> “/user/<user>/datatorrent”. But, still my apex application failed after
>>> 7days!!!
>>>
>>> I am setting these properties in “properties.xml” file!
>>>
>>> How can I ensure my settings are working correct. Having waiting for
>>> 7days to learn its failure is a very tough thing. Hope there should be some
>>> other alternatives.
>>>
>>> Can someone pls help me fix this …. Thanks a lot !!
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: "Raja.Aravapalli" <Ra...@target.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Monday, June 20, 2016 at 5:43 PM
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>>
>>> Sure Pramod. Please respond on this mail chain when you get to know..
>>>
>>> Thanks very much.
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: Pramod Immaneni <pr...@datatorrent.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Monday, June 20, 2016 at 4:54 PM
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>> Raja,
>>>
>>> I believe it would. I will check and get back to you but the easiest way
>>> for you to check is that the file should appear in HDFS under
>>> /user/<username>/datatorrent with the same filename as it is in your local
>>> filesystem.
>>>
>>> Thanks
>>>
>>> On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <
>>> Raja.Aravapalli@target.com> wrote:
>>>
>>>>
>>>> Thanks for the response Pramod.
>>>>
>>>> My quick question is, I see we should mention these properties in
>>>> dt-site.xml !! I am not sure about dt-site.xml, all I am using is only
>>>> properites.xml file, which I am using to pass some configuration to
>>>> application.
>>>> Can I set these in properties.xml file and it will still work ?
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>> From: Pramod Immaneni <pr...@datatorrent.com>
>>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Date: Monday, June 20, 2016 at 4:32 PM
>>>>
>>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>>
>>>> Hi Raja,
>>>>
>>>> Yes the keytab would be copied over to HDFS and reused for getting a
>>>> new token before the old one expires. By default it is 7 days. If it is
>>>> different in your cluster please set the
>>>> properties dt.resourcemanager.delegation.token.max-lifetime and
>>>> dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't
>>>> the default keytab to be copied over into HDFS and reused you can specify
>>>> your own keytab file for fetching a new token by putting it in HDFS and
>>>> specifying the property dt.authentication.store.keytab.All this is
>>>> described in the document that Thomas sent over.
>>>>
>>>> Thanks
>>>>
>>>> On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <
>>>> Raja.Aravapalli@target.com> wrote:
>>>>
>>>>>
>>>>> Hi Thomas,
>>>>>
>>>>> To ensure auto renewal of delegation tokens life time, Can I use the
>>>>> the below properties in properties.xml file ?
>>>>>
>>>>> <property>
>>>>> <name>dt.authentication.principal</name>
>>>>> <value>kerberos-principal-of-user</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>dt.authentication.keytab</name>
>>>>> <value>absolute-path-to-keytab-file</value>
>>>>> </property>
>>>>>
>>>>>
>>>>> FYI,
>>>>> I am launching application from Apex CLI! And till this time I haven’t
>>>>> used the above properties when launching apex applications in our secure
>>>>> hadoop environment, still they worked fine without any issues, but failing
>>>>> after 7days!!
>>>>>
>>>>> If I set the above properties in properties.xml, will that do
>>>>> auto-renewal and run successfully without any issues of failing again due
>>>>> to delegation token lifetime expiry ??
>>>>>
>>>>> Please advise.
>>>>>
>>>>>
>>>>> Thanks a lot in advance.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Raja.
>>>>>
>>>>> From: "Raja.Aravapalli" <Ra...@target.com>
>>>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>>>> Date: Sunday, June 19, 2016 at 3:30 PM
>>>>>
>>>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>>>
>>>>>
>>>>> Thanks a lot Thomas.
>>>>>
>>>>> Will take this as reference and test our application. Great!
>>>>>
>>>>>
>>>>> Regards,
>>>>> Raja.
>>>>>
>>>>> From: Thomas Weise <th...@gmail.com>
>>>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>>>> Date: Sunday, June 19, 2016 at 2:01 PM
>>>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>>>
>>>>> Token expiration working as expected!
>>>>>
>>>>> Please have a look on how to extend or refresh it:
>>>>>
>>>>>
>>>>> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
>>>>> Raja.Aravapalli@target.com> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> My Apex application failed exactly after running 7days in our
>>>>>> distributed hadoop environment, with delegation token expiry!!
>>>>>>
>>>>>> Can someone pls help me with details, on how I can increase the
>>>>>> delegation token time to lifetime or any other process running in parallel
>>>>>> to renew the tokens ?
>>>>>>
>>>>>> *Exception details below:*
>>>>>>
>>>>>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
>>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks a lot in advance.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Raja.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Hi Pramod,
As suggested in step2, I added the properties you provided below, and ran a new job, to see the debug lines "number of tokens: " and "updated token: "
And, as you said, I can see those lines in the application log now..
Does this mean, the renewal is working as expected ? So, will my application run continously even after 7days now ? We have only changed refersh times here… how does this ensure the application runs indefinite time…
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Saturday, July 2, 2016 at 9:52 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks Pramod, I will these suggestions and let you know. Thanks a lot.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Friday, July 1, 2016 at 8:36 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Some questions for you and also some options for you to verify without waiting for a long time.
1. Do you see a warning message like ""No keytab specified for refreshing tokens, application may not be able to run indefinitely" when application is being launched from command line.
2. For testing, set the parameters below in your dt-site.xml and launch your application. This will make the application think that the tokens are only valid for 5 minutes and within 5 * 0.7 = 3.5 minutes (token refresh factor is 0.7) the application should try to get new tokens. It should print the debug lines "number of tokens: " and "updated token: " in the application master logs. The application master logs are in the log file of the first container of the application. Let me know if you see those log lines.
<property>
<name>dt.resourcemanager.delegation.token.max-lifetime</name>
<value>300000</value>
</property>
<property>
<name>dt.namenode.delegation.token.max-lifetime</name>
<value>300000</value>
</property>
<property>
<name>dt.attr.DEBUG</name>
<value>true</value>
</property>
For more information about application auto-fetching new tokens read here
https://github.com/apache/apex-core/blob/master/docs/security.md
Thanks
On Fri, Jul 1, 2016 at 1:08 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks a lot Pramod. Will wait for your response.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Friday, July 1, 2016 at 10:56 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Let me look at this and get back to you.
Thanks
On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Can someone pls help me, how can I ensure, my apex application doesn’t fail after 7days…
Thanks a lot.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Thursday, June 30, 2016 at 6:06 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi,
I triggered my application by specifying properties, “dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”.
I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”. But, still my apex application failed after 7days!!!
I am setting these properties in “properties.xml” file!
How can I ensure my settings are working correct. Having waiting for 7days to learn its failure is a very tough thing. Hope there should be some other alternatives.
Can someone pls help me fix this …. Thanks a lot !!
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 5:43 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Sure Pramod. Please respond on this mail chain when you get to know..
Thanks very much.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Raja,
I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem.
Thanks
On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks for the response Pramod.
My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application.
Can I set these in properties.xml file and it will still work ?
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks Pramod, I will these suggestions and let you know. Thanks a lot.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Friday, July 1, 2016 at 8:36 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Some questions for you and also some options for you to verify without waiting for a long time.
1. Do you see a warning message like ""No keytab specified for refreshing tokens, application may not be able to run indefinitely" when application is being launched from command line.
2. For testing, set the parameters below in your dt-site.xml and launch your application. This will make the application think that the tokens are only valid for 5 minutes and within 5 * 0.7 = 3.5 minutes (token refresh factor is 0.7) the application should try to get new tokens. It should print the debug lines "number of tokens: " and "updated token: " in the application master logs. The application master logs are in the log file of the first container of the application. Let me know if you see those log lines.
<property>
<name>dt.resourcemanager.delegation.token.max-lifetime</name>
<value>300000</value>
</property>
<property>
<name>dt.namenode.delegation.token.max-lifetime</name>
<value>300000</value>
</property>
<property>
<name>dt.attr.DEBUG</name>
<value>true</value>
</property>
For more information about application auto-fetching new tokens read here
https://github.com/apache/apex-core/blob/master/docs/security.md
Thanks
On Fri, Jul 1, 2016 at 1:08 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks a lot Pramod. Will wait for your response.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Friday, July 1, 2016 at 10:56 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Let me look at this and get back to you.
Thanks
On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Can someone pls help me, how can I ensure, my apex application doesn’t fail after 7days…
Thanks a lot.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Thursday, June 30, 2016 at 6:06 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi,
I triggered my application by specifying properties, “dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”.
I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”. But, still my apex application failed after 7days!!!
I am setting these properties in “properties.xml” file!
How can I ensure my settings are working correct. Having waiting for 7days to learn its failure is a very tough thing. Hope there should be some other alternatives.
Can someone pls help me fix this …. Thanks a lot !!
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 5:43 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Sure Pramod. Please respond on this mail chain when you get to know..
Thanks very much.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Raja,
I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem.
Thanks
On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks for the response Pramod.
My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application.
Can I set these in properties.xml file and it will still work ?
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Raja,
Some questions for you and also some options for you to verify without
waiting for a long time.
1. Do you see a warning message like ""No keytab specified for refreshing
tokens, application may not be able to run indefinitely" when application
is being launched from command line.
2. For testing, set the parameters below in your dt-site.xml and launch
your application. This will make the application think that the tokens are
only valid for 5 minutes and within 5 * 0.7 = 3.5 minutes (token refresh
factor is 0.7) the application should try to get new tokens. It should
print the debug lines "number of tokens: " and "updated token: " in the
application master logs. The application master logs are in the log file of
the first container of the application. Let me know if you see those log
lines.
<property>
<name>dt.resourcemanager.delegation.token.max-lifetime</name>
<value>300000</value>
</property>
<property>
<name>dt.namenode.delegation.token.max-lifetime</name>
<value>300000</value>
</property>
<property>
<name>dt.attr.DEBUG</name>
<value>true</value>
</property>
For more information about application auto-fetching new tokens read here
https://github.com/apache/apex-core/blob/master/docs/security.md
Thanks
On Fri, Jul 1, 2016 at 1:08 PM, Raja.Aravapalli <Ra...@target.com>
wrote:
>
> Thanks a lot Pramod. Will wait for your response.
>
>
> Regards,
> Raja.
>
> From: Pramod Immaneni <pr...@datatorrent.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Friday, July 1, 2016 at 10:56 AM
>
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
> Hi Raja,
>
> Let me look at this and get back to you.
>
> Thanks
>
> On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>> Can someone pls help me, how can I ensure, my apex application doesn’t
>> fail after 7days…
>>
>> Thanks a lot.
>>
>>
>> Regards,
>> Raja.
>>
>> From: "Raja.Aravapalli" <Ra...@target.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Thursday, June 30, 2016 at 6:06 AM
>>
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>>
>> Hi,
>>
>> I triggered my application by specifying properties, “
>> dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not
>> specify the property “dt.authentication.store.keytab”.
>>
>> I also observed the keytab is copied to hdfs location
>> “/user/<user>/datatorrent”. But, still my apex application failed after
>> 7days!!!
>>
>> I am setting these properties in “properties.xml” file!
>>
>> How can I ensure my settings are working correct. Having waiting for
>> 7days to learn its failure is a very tough thing. Hope there should be some
>> other alternatives.
>>
>> Can someone pls help me fix this …. Thanks a lot !!
>>
>>
>> Regards,
>> Raja.
>>
>> From: "Raja.Aravapalli" <Ra...@target.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Monday, June 20, 2016 at 5:43 PM
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>>
>> Sure Pramod. Please respond on this mail chain when you get to know..
>>
>> Thanks very much.
>>
>>
>> Regards,
>> Raja.
>>
>> From: Pramod Immaneni <pr...@datatorrent.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Monday, June 20, 2016 at 4:54 PM
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>> Raja,
>>
>> I believe it would. I will check and get back to you but the easiest way
>> for you to check is that the file should appear in HDFS under
>> /user/<username>/datatorrent with the same filename as it is in your local
>> filesystem.
>>
>> Thanks
>>
>> On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <
>> Raja.Aravapalli@target.com> wrote:
>>
>>>
>>> Thanks for the response Pramod.
>>>
>>> My quick question is, I see we should mention these properties in
>>> dt-site.xml !! I am not sure about dt-site.xml, all I am using is only
>>> properites.xml file, which I am using to pass some configuration to
>>> application.
>>> Can I set these in properties.xml file and it will still work ?
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: Pramod Immaneni <pr...@datatorrent.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Monday, June 20, 2016 at 4:32 PM
>>>
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>> Hi Raja,
>>>
>>> Yes the keytab would be copied over to HDFS and reused for getting a new
>>> token before the old one expires. By default it is 7 days. If it is
>>> different in your cluster please set the
>>> properties dt.resourcemanager.delegation.token.max-lifetime and
>>> dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't
>>> the default keytab to be copied over into HDFS and reused you can specify
>>> your own keytab file for fetching a new token by putting it in HDFS and
>>> specifying the property dt.authentication.store.keytab.All this is
>>> described in the document that Thomas sent over.
>>>
>>> Thanks
>>>
>>> On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <
>>> Raja.Aravapalli@target.com> wrote:
>>>
>>>>
>>>> Hi Thomas,
>>>>
>>>> To ensure auto renewal of delegation tokens life time, Can I use the
>>>> the below properties in properties.xml file ?
>>>>
>>>> <property>
>>>> <name>dt.authentication.principal</name>
>>>> <value>kerberos-principal-of-user</value>
>>>> </property>
>>>> <property>
>>>> <name>dt.authentication.keytab</name>
>>>> <value>absolute-path-to-keytab-file</value>
>>>> </property>
>>>>
>>>>
>>>> FYI,
>>>> I am launching application from Apex CLI! And till this time I haven’t
>>>> used the above properties when launching apex applications in our secure
>>>> hadoop environment, still they worked fine without any issues, but failing
>>>> after 7days!!
>>>>
>>>> If I set the above properties in properties.xml, will that do
>>>> auto-renewal and run successfully without any issues of failing again due
>>>> to delegation token lifetime expiry ??
>>>>
>>>> Please advise.
>>>>
>>>>
>>>> Thanks a lot in advance.
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>> From: "Raja.Aravapalli" <Ra...@target.com>
>>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Date: Sunday, June 19, 2016 at 3:30 PM
>>>>
>>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>>
>>>>
>>>> Thanks a lot Thomas.
>>>>
>>>> Will take this as reference and test our application. Great!
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>> From: Thomas Weise <th...@gmail.com>
>>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Date: Sunday, June 19, 2016 at 2:01 PM
>>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>>
>>>> Token expiration working as expected!
>>>>
>>>> Please have a look on how to extend or refresh it:
>>>>
>>>>
>>>> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
>>>> Raja.Aravapalli@target.com> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> My Apex application failed exactly after running 7days in our
>>>>> distributed hadoop environment, with delegation token expiry!!
>>>>>
>>>>> Can someone pls help me with details, on how I can increase the
>>>>> delegation token time to lifetime or any other process running in parallel
>>>>> to renew the tokens ?
>>>>>
>>>>> *Exception details below:*
>>>>>
>>>>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>>>>
>>>>>
>>>>>
>>>>> Thanks a lot in advance.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Raja.
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks a lot Pramod. Will wait for your response.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Friday, July 1, 2016 at 10:56 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Let me look at this and get back to you.
Thanks
On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Can someone pls help me, how can I ensure, my apex application doesn’t fail after 7days…
Thanks a lot.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Thursday, June 30, 2016 at 6:06 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi,
I triggered my application by specifying properties, “dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”.
I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”. But, still my apex application failed after 7days!!!
I am setting these properties in “properties.xml” file!
How can I ensure my settings are working correct. Having waiting for 7days to learn its failure is a very tough thing. Hope there should be some other alternatives.
Can someone pls help me fix this …. Thanks a lot !!
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 5:43 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Sure Pramod. Please respond on this mail chain when you get to know..
Thanks very much.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Raja,
I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem.
Thanks
On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks for the response Pramod.
My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application.
Can I set these in properties.xml file and it will still work ?
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Raja,
Let me look at this and get back to you.
Thanks
On Thu, Jun 30, 2016 at 11:20 PM, Raja.Aravapalli <
Raja.Aravapalli@target.com> wrote:
>
> Can someone pls help me, how can I ensure, my apex application doesn’t
> fail after 7days…
>
> Thanks a lot.
>
>
> Regards,
> Raja.
>
> From: "Raja.Aravapalli" <Ra...@target.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Thursday, June 30, 2016 at 6:06 AM
>
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
>
> Hi,
>
> I triggered my application by specifying properties, “
> dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not
> specify the property “dt.authentication.store.keytab”.
>
> I also observed the keytab is copied to hdfs location
> “/user/<user>/datatorrent”. But, still my apex application failed after
> 7days!!!
>
> I am setting these properties in “properties.xml” file!
>
> How can I ensure my settings are working correct. Having waiting for 7days
> to learn its failure is a very tough thing. Hope there should be some other
> alternatives.
>
> Can someone pls help me fix this …. Thanks a lot !!
>
>
> Regards,
> Raja.
>
> From: "Raja.Aravapalli" <Ra...@target.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Monday, June 20, 2016 at 5:43 PM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
>
> Sure Pramod. Please respond on this mail chain when you get to know..
>
> Thanks very much.
>
>
> Regards,
> Raja.
>
> From: Pramod Immaneni <pr...@datatorrent.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Monday, June 20, 2016 at 4:54 PM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
> Raja,
>
> I believe it would. I will check and get back to you but the easiest way
> for you to check is that the file should appear in HDFS under
> /user/<username>/datatorrent with the same filename as it is in your local
> filesystem.
>
> Thanks
>
> On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>> Thanks for the response Pramod.
>>
>> My quick question is, I see we should mention these properties in
>> dt-site.xml !! I am not sure about dt-site.xml, all I am using is only
>> properites.xml file, which I am using to pass some configuration to
>> application.
>> Can I set these in properties.xml file and it will still work ?
>>
>>
>> Regards,
>> Raja.
>>
>> From: Pramod Immaneni <pr...@datatorrent.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Monday, June 20, 2016 at 4:32 PM
>>
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>> Hi Raja,
>>
>> Yes the keytab would be copied over to HDFS and reused for getting a new
>> token before the old one expires. By default it is 7 days. If it is
>> different in your cluster please set the
>> properties dt.resourcemanager.delegation.token.max-lifetime and
>> dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't
>> the default keytab to be copied over into HDFS and reused you can specify
>> your own keytab file for fetching a new token by putting it in HDFS and
>> specifying the property dt.authentication.store.keytab.All this is
>> described in the document that Thomas sent over.
>>
>> Thanks
>>
>> On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <
>> Raja.Aravapalli@target.com> wrote:
>>
>>>
>>> Hi Thomas,
>>>
>>> To ensure auto renewal of delegation tokens life time, Can I use the the
>>> below properties in properties.xml file ?
>>>
>>> <property>
>>> <name>dt.authentication.principal</name>
>>> <value>kerberos-principal-of-user</value>
>>> </property>
>>> <property>
>>> <name>dt.authentication.keytab</name>
>>> <value>absolute-path-to-keytab-file</value>
>>> </property>
>>>
>>>
>>> FYI,
>>> I am launching application from Apex CLI! And till this time I haven’t
>>> used the above properties when launching apex applications in our secure
>>> hadoop environment, still they worked fine without any issues, but failing
>>> after 7days!!
>>>
>>> If I set the above properties in properties.xml, will that do
>>> auto-renewal and run successfully without any issues of failing again due
>>> to delegation token lifetime expiry ??
>>>
>>> Please advise.
>>>
>>>
>>> Thanks a lot in advance.
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: "Raja.Aravapalli" <Ra...@target.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Sunday, June 19, 2016 at 3:30 PM
>>>
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>>
>>> Thanks a lot Thomas.
>>>
>>> Will take this as reference and test our application. Great!
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: Thomas Weise <th...@gmail.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Sunday, June 19, 2016 at 2:01 PM
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>> Token expiration working as expected!
>>>
>>> Please have a look on how to extend or refresh it:
>>>
>>>
>>> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
>>> Raja.Aravapalli@target.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> My Apex application failed exactly after running 7days in our
>>>> distributed hadoop environment, with delegation token expiry!!
>>>>
>>>> Can someone pls help me with details, on how I can increase the
>>>> delegation token time to lifetime or any other process running in parallel
>>>> to renew the tokens ?
>>>>
>>>> *Exception details below:*
>>>>
>>>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>>>
>>>>
>>>>
>>>> Thanks a lot in advance.
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>>
>>>
>>
>
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Aniruddha Thombare <an...@datatorrent.com>.
Hi,
You can set the flag on MIT kerberos server with following in your kdc.conf:
default_principal_flags = *+renewable*, +YOUR_OTHER_FLAGS....
Also, take a look at:
http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_sg_sec_troubleshooting.html
Thanks,
Aniruddha
On Fri, Jul 1, 2016 at 12:02 PM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:
> Hi,
>
> I believe, you should also set renewable flag for those tickets for the
> principal.
>
> MIT Kerberos documents for the same:
>
> http://web.mit.edu/Kerberos/krb5-1.13/doc/user/tkt_mgmt.html
>
> @Pramod, Please correct me if this is wrong...
>
> Thanks,
>
> A
>
> _____________________________________
> Sent with difficulty, I mean handheld ;)
> On 1 Jul 2016 11:50 am, "Raja.Aravapalli" <Ra...@target.com>
> wrote:
>
>>
>> Can someone pls help me, how can I ensure, my apex application doesn’t
>> fail after 7days…
>>
>> Thanks a lot.
>>
>>
>> Regards,
>> Raja.
>>
>> From: "Raja.Aravapalli" <Ra...@target.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Thursday, June 30, 2016 at 6:06 AM
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>>
>> Hi,
>>
>> I triggered my application by specifying properties, “
>> dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not
>> specify the property “dt.authentication.store.keytab”.
>>
>> I also observed the keytab is copied to hdfs location
>> “/user/<user>/datatorrent”. But, still my apex application failed after
>> 7days!!!
>>
>> I am setting these properties in “properties.xml” file!
>>
>> How can I ensure my settings are working correct. Having waiting for
>> 7days to learn its failure is a very tough thing. Hope there should be some
>> other alternatives.
>>
>> Can someone pls help me fix this …. Thanks a lot !!
>>
>>
>> Regards,
>> Raja.
>>
>> From: "Raja.Aravapalli" <Ra...@target.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Monday, June 20, 2016 at 5:43 PM
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>>
>> Sure Pramod. Please respond on this mail chain when you get to know..
>>
>> Thanks very much.
>>
>>
>> Regards,
>> Raja.
>>
>> From: Pramod Immaneni <pr...@datatorrent.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Monday, June 20, 2016 at 4:54 PM
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>> Raja,
>>
>> I believe it would. I will check and get back to you but the easiest way
>> for you to check is that the file should appear in HDFS under
>> /user/<username>/datatorrent with the same filename as it is in your local
>> filesystem.
>>
>> Thanks
>>
>> On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <
>> Raja.Aravapalli@target.com> wrote:
>>
>>>
>>> Thanks for the response Pramod.
>>>
>>> My quick question is, I see we should mention these properties in
>>> dt-site.xml !! I am not sure about dt-site.xml, all I am using is only
>>> properites.xml file, which I am using to pass some configuration to
>>> application.
>>> Can I set these in properties.xml file and it will still work ?
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: Pramod Immaneni <pr...@datatorrent.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Monday, June 20, 2016 at 4:32 PM
>>>
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>> Hi Raja,
>>>
>>> Yes the keytab would be copied over to HDFS and reused for getting a new
>>> token before the old one expires. By default it is 7 days. If it is
>>> different in your cluster please set the
>>> properties dt.resourcemanager.delegation.token.max-lifetime and
>>> dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't
>>> the default keytab to be copied over into HDFS and reused you can specify
>>> your own keytab file for fetching a new token by putting it in HDFS and
>>> specifying the property dt.authentication.store.keytab.All this is
>>> described in the document that Thomas sent over.
>>>
>>> Thanks
>>>
>>> On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <
>>> Raja.Aravapalli@target.com> wrote:
>>>
>>>>
>>>> Hi Thomas,
>>>>
>>>> To ensure auto renewal of delegation tokens life time, Can I use the
>>>> the below properties in properties.xml file ?
>>>>
>>>> <property>
>>>> <name>dt.authentication.principal</name>
>>>> <value>kerberos-principal-of-user</value>
>>>> </property>
>>>> <property>
>>>> <name>dt.authentication.keytab</name>
>>>> <value>absolute-path-to-keytab-file</value>
>>>> </property>
>>>>
>>>>
>>>> FYI,
>>>> I am launching application from Apex CLI! And till this time I haven’t
>>>> used the above properties when launching apex applications in our secure
>>>> hadoop environment, still they worked fine without any issues, but failing
>>>> after 7days!!
>>>>
>>>> If I set the above properties in properties.xml, will that do
>>>> auto-renewal and run successfully without any issues of failing again due
>>>> to delegation token lifetime expiry ??
>>>>
>>>> Please advise.
>>>>
>>>>
>>>> Thanks a lot in advance.
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>> From: "Raja.Aravapalli" <Ra...@target.com>
>>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Date: Sunday, June 19, 2016 at 3:30 PM
>>>>
>>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>>
>>>>
>>>> Thanks a lot Thomas.
>>>>
>>>> Will take this as reference and test our application. Great!
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>> From: Thomas Weise <th...@gmail.com>
>>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Date: Sunday, June 19, 2016 at 2:01 PM
>>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>>
>>>> Token expiration working as expected!
>>>>
>>>> Please have a look on how to extend or refresh it:
>>>>
>>>>
>>>> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
>>>> Raja.Aravapalli@target.com> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> My Apex application failed exactly after running 7days in our
>>>>> distributed hadoop environment, with delegation token expiry!!
>>>>>
>>>>> Can someone pls help me with details, on how I can increase the
>>>>> delegation token time to lifetime or any other process running in parallel
>>>>> to renew the tokens ?
>>>>>
>>>>> *Exception details below:*
>>>>>
>>>>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>>>>
>>>>>
>>>>>
>>>>> Thanks a lot in advance.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Raja.
>>>>>
>>>>>
>>>>
>>>
>>
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Aniruddha Thombare <an...@datatorrent.com>.
Hi,
I believe, you should also set renewable flag for those tickets for the
principal.
MIT Kerberos documents for the same:
http://web.mit.edu/Kerberos/krb5-1.13/doc/user/tkt_mgmt.html
@Pramod, Please correct me if this is wrong...
Thanks,
A
_____________________________________
Sent with difficulty, I mean handheld ;)
On 1 Jul 2016 11:50 am, "Raja.Aravapalli" <Ra...@target.com>
wrote:
>
> Can someone pls help me, how can I ensure, my apex application doesn’t
> fail after 7days…
>
> Thanks a lot.
>
>
> Regards,
> Raja.
>
> From: "Raja.Aravapalli" <Ra...@target.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Thursday, June 30, 2016 at 6:06 AM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
>
> Hi,
>
> I triggered my application by specifying properties, “
> dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not
> specify the property “dt.authentication.store.keytab”.
>
> I also observed the keytab is copied to hdfs location
> “/user/<user>/datatorrent”. But, still my apex application failed after
> 7days!!!
>
> I am setting these properties in “properties.xml” file!
>
> How can I ensure my settings are working correct. Having waiting for 7days
> to learn its failure is a very tough thing. Hope there should be some other
> alternatives.
>
> Can someone pls help me fix this …. Thanks a lot !!
>
>
> Regards,
> Raja.
>
> From: "Raja.Aravapalli" <Ra...@target.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Monday, June 20, 2016 at 5:43 PM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
>
> Sure Pramod. Please respond on this mail chain when you get to know..
>
> Thanks very much.
>
>
> Regards,
> Raja.
>
> From: Pramod Immaneni <pr...@datatorrent.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Monday, June 20, 2016 at 4:54 PM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
> Raja,
>
> I believe it would. I will check and get back to you but the easiest way
> for you to check is that the file should appear in HDFS under
> /user/<username>/datatorrent with the same filename as it is in your local
> filesystem.
>
> Thanks
>
> On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>> Thanks for the response Pramod.
>>
>> My quick question is, I see we should mention these properties in
>> dt-site.xml !! I am not sure about dt-site.xml, all I am using is only
>> properites.xml file, which I am using to pass some configuration to
>> application.
>> Can I set these in properties.xml file and it will still work ?
>>
>>
>> Regards,
>> Raja.
>>
>> From: Pramod Immaneni <pr...@datatorrent.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Monday, June 20, 2016 at 4:32 PM
>>
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>> Hi Raja,
>>
>> Yes the keytab would be copied over to HDFS and reused for getting a new
>> token before the old one expires. By default it is 7 days. If it is
>> different in your cluster please set the
>> properties dt.resourcemanager.delegation.token.max-lifetime and
>> dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't
>> the default keytab to be copied over into HDFS and reused you can specify
>> your own keytab file for fetching a new token by putting it in HDFS and
>> specifying the property dt.authentication.store.keytab.All this is
>> described in the document that Thomas sent over.
>>
>> Thanks
>>
>> On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <
>> Raja.Aravapalli@target.com> wrote:
>>
>>>
>>> Hi Thomas,
>>>
>>> To ensure auto renewal of delegation tokens life time, Can I use the the
>>> below properties in properties.xml file ?
>>>
>>> <property>
>>> <name>dt.authentication.principal</name>
>>> <value>kerberos-principal-of-user</value>
>>> </property>
>>> <property>
>>> <name>dt.authentication.keytab</name>
>>> <value>absolute-path-to-keytab-file</value>
>>> </property>
>>>
>>>
>>> FYI,
>>> I am launching application from Apex CLI! And till this time I haven’t
>>> used the above properties when launching apex applications in our secure
>>> hadoop environment, still they worked fine without any issues, but failing
>>> after 7days!!
>>>
>>> If I set the above properties in properties.xml, will that do
>>> auto-renewal and run successfully without any issues of failing again due
>>> to delegation token lifetime expiry ??
>>>
>>> Please advise.
>>>
>>>
>>> Thanks a lot in advance.
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: "Raja.Aravapalli" <Ra...@target.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Sunday, June 19, 2016 at 3:30 PM
>>>
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>>
>>> Thanks a lot Thomas.
>>>
>>> Will take this as reference and test our application. Great!
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>> From: Thomas Weise <th...@gmail.com>
>>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Date: Sunday, June 19, 2016 at 2:01 PM
>>> To: "users@apex.apache.org" <us...@apex.apache.org>
>>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>>
>>> Token expiration working as expected!
>>>
>>> Please have a look on how to extend or refresh it:
>>>
>>>
>>> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
>>> Raja.Aravapalli@target.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> My Apex application failed exactly after running 7days in our
>>>> distributed hadoop environment, with delegation token expiry!!
>>>>
>>>> Can someone pls help me with details, on how I can increase the
>>>> delegation token time to lifetime or any other process running in parallel
>>>> to renew the tokens ?
>>>>
>>>> *Exception details below:*
>>>>
>>>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>>>
>>>>
>>>>
>>>> Thanks a lot in advance.
>>>>
>>>>
>>>> Regards,
>>>> Raja.
>>>>
>>>>
>>>
>>
>
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Can someone pls help me, how can I ensure, my apex application doesn’t fail after 7days…
Thanks a lot.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Thursday, June 30, 2016 at 6:06 AM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi,
I triggered my application by specifying properties, “dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”.
I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”. But, still my apex application failed after 7days!!!
I am setting these properties in “properties.xml” file!
How can I ensure my settings are working correct. Having waiting for 7days to learn its failure is a very tough thing. Hope there should be some other alternatives.
Can someone pls help me fix this …. Thanks a lot !!
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 5:43 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Sure Pramod. Please respond on this mail chain when you get to know..
Thanks very much.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Raja,
I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem.
Thanks
On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks for the response Pramod.
My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application.
Can I set these in properties.xml file and it will still work ?
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Hi,
I triggered my application by specifying properties, “dt.authentication.principal” & “dt.authentication.keytab” , BUT, did not specify the property “dt.authentication.store.keytab”.
I also observed the keytab is copied to hdfs location “/user/<user>/datatorrent”. But, still my apex application failed after 7days!!!
I am setting these properties in “properties.xml” file!
How can I ensure my settings are working correct. Having waiting for 7days to learn its failure is a very tough thing. Hope there should be some other alternatives.
Can someone pls help me fix this …. Thanks a lot !!
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 5:43 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Sure Pramod. Please respond on this mail chain when you get to know..
Thanks very much.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Raja,
I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem.
Thanks
On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks for the response Pramod.
My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application.
Can I set these in properties.xml file and it will still work ?
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Sure Pramod. Please respond on this mail chain when you get to know..
Thanks very much.
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:54 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Raja,
I believe it would. I will check and get back to you but the easiest way for you to check is that the file should appear in HDFS under /user/<username>/datatorrent with the same filename as it is in your local filesystem.
Thanks
On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks for the response Pramod.
My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application.
Can I set these in properties.xml file and it will still work ?
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Raja,
I believe it would. I will check and get back to you but the easiest way
for you to check is that the file should appear in HDFS under
/user/<username>/datatorrent with the same filename as it is in your local
filesystem.
Thanks
On Mon, Jun 20, 2016 at 2:40 PM, Raja.Aravapalli <Raja.Aravapalli@target.com
> wrote:
>
> Thanks for the response Pramod.
>
> My quick question is, I see we should mention these properties in
> dt-site.xml !! I am not sure about dt-site.xml, all I am using is only
> properites.xml file, which I am using to pass some configuration to
> application.
> Can I set these in properties.xml file and it will still work ?
>
>
> Regards,
> Raja.
>
> From: Pramod Immaneni <pr...@datatorrent.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Monday, June 20, 2016 at 4:32 PM
>
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
> Hi Raja,
>
> Yes the keytab would be copied over to HDFS and reused for getting a new
> token before the old one expires. By default it is 7 days. If it is
> different in your cluster please set the
> properties dt.resourcemanager.delegation.token.max-lifetime and
> dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't
> the default keytab to be copied over into HDFS and reused you can specify
> your own keytab file for fetching a new token by putting it in HDFS and
> specifying the property dt.authentication.store.keytab.All this is
> described in the document that Thomas sent over.
>
> Thanks
>
> On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>> Hi Thomas,
>>
>> To ensure auto renewal of delegation tokens life time, Can I use the the
>> below properties in properties.xml file ?
>>
>> <property>
>> <name>dt.authentication.principal</name>
>> <value>kerberos-principal-of-user</value>
>> </property>
>> <property>
>> <name>dt.authentication.keytab</name>
>> <value>absolute-path-to-keytab-file</value>
>> </property>
>>
>>
>> FYI,
>> I am launching application from Apex CLI! And till this time I haven’t
>> used the above properties when launching apex applications in our secure
>> hadoop environment, still they worked fine without any issues, but failing
>> after 7days!!
>>
>> If I set the above properties in properties.xml, will that do
>> auto-renewal and run successfully without any issues of failing again due
>> to delegation token lifetime expiry ??
>>
>> Please advise.
>>
>>
>> Thanks a lot in advance.
>>
>>
>> Regards,
>> Raja.
>>
>> From: "Raja.Aravapalli" <Ra...@target.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Sunday, June 19, 2016 at 3:30 PM
>>
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>>
>> Thanks a lot Thomas.
>>
>> Will take this as reference and test our application. Great!
>>
>>
>> Regards,
>> Raja.
>>
>> From: Thomas Weise <th...@gmail.com>
>> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
>> Date: Sunday, June 19, 2016 at 2:01 PM
>> To: "users@apex.apache.org" <us...@apex.apache.org>
>> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>>
>> Token expiration working as expected!
>>
>> Please have a look on how to extend or refresh it:
>>
>>
>> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
>>
>> Thanks,
>> Thomas
>>
>>
>> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
>> Raja.Aravapalli@target.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> My Apex application failed exactly after running 7days in our
>>> distributed hadoop environment, with delegation token expiry!!
>>>
>>> Can someone pls help me with details, on how I can increase the
>>> delegation token time to lifetime or any other process running in parallel
>>> to renew the tokens ?
>>>
>>> *Exception details below:*
>>>
>>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>>
>>>
>>>
>>> Thanks a lot in advance.
>>>
>>>
>>> Regards,
>>> Raja.
>>>
>>>
>>
>
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks for the response Pramod.
My quick question is, I see we should mention these properties in dt-site.xml !! I am not sure about dt-site.xml, all I am using is only properites.xml file, which I am using to pass some configuration to application.
Can I set these in properties.xml file and it will still work ?
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, June 20, 2016 at 4:32 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new token before the old one expires. By default it is 7 days. If it is different in your cluster please set the properties dt.resourcemanager.delegation.token.max-lifetime and dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't the default keytab to be copied over into HDFS and reused you can specify your own keytab file for fetching a new token by putting it in HDFS and specifying the property dt.authentication.store.keytab.All this is described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Raja,
Yes the keytab would be copied over to HDFS and reused for getting a new
token before the old one expires. By default it is 7 days. If it is
different in your cluster please set the
properties dt.resourcemanager.delegation.token.max-lifetime and
dt.namenode.delegation.token.max-lifetime in dt-site.xml. Also if you don't
the default keytab to be copied over into HDFS and reused you can specify
your own keytab file for fetching a new token by putting it in HDFS and
specifying the property dt.authentication.store.keytab.All this is
described in the document that Thomas sent over.
Thanks
On Mon, Jun 20, 2016 at 1:54 PM, Raja.Aravapalli <Raja.Aravapalli@target.com
> wrote:
>
> Hi Thomas,
>
> To ensure auto renewal of delegation tokens life time, Can I use the the
> below properties in properties.xml file ?
>
> <property>
> <name>dt.authentication.principal</name>
> <value>kerberos-principal-of-user</value>
> </property>
> <property>
> <name>dt.authentication.keytab</name>
> <value>absolute-path-to-keytab-file</value>
> </property>
>
>
> FYI,
> I am launching application from Apex CLI! And till this time I haven’t
> used the above properties when launching apex applications in our secure
> hadoop environment, still they worked fine without any issues, but failing
> after 7days!!
>
> If I set the above properties in properties.xml, will that do auto-renewal
> and run successfully without any issues of failing again due to delegation
> token lifetime expiry ??
>
> Please advise.
>
>
> Thanks a lot in advance.
>
>
> Regards,
> Raja.
>
> From: "Raja.Aravapalli" <Ra...@target.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Sunday, June 19, 2016 at 3:30 PM
>
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
>
> Thanks a lot Thomas.
>
> Will take this as reference and test our application. Great!
>
>
> Regards,
> Raja.
>
> From: Thomas Weise <th...@gmail.com>
> Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
> Date: Sunday, June 19, 2016 at 2:01 PM
> To: "users@apex.apache.org" <us...@apex.apache.org>
> Subject: Re: how to increase lifetime of hdfs delegation tokens ?
>
> Token expiration working as expected!
>
> Please have a look on how to extend or refresh it:
>
>
> https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
>
> Thanks,
> Thomas
>
>
> On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>> Hi,
>>
>> My Apex application failed exactly after running 7days in our distributed
>> hadoop environment, with delegation token expiry!!
>>
>> Can someone pls help me with details, on how I can increase the
>> delegation token time to lifetime or any other process running in parallel
>> to renew the tokens ?
>>
>> *Exception details below:*
>>
>> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
>> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>
>>
>>
>> Thanks a lot in advance.
>>
>>
>> Regards,
>> Raja.
>>
>>
>
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Hi Thomas,
To ensure auto renewal of delegation tokens life time, Can I use the the below properties in properties.xml file ?
<property>
<name>dt.authentication.principal</name>
<value>kerberos-principal-of-user</value>
</property>
<property>
<name>dt.authentication.keytab</name>
<value>absolute-path-to-keytab-file</value>
</property>
FYI,
I am launching application from Apex CLI! And till this time I haven’t used the above properties when launching apex applications in our secure hadoop environment, still they worked fine without any issues, but failing after 7days!!
If I set the above properties in properties.xml, will that do auto-renewal and run successfully without any issues of failing again due to delegation token lifetime expiry ??
Please advise.
Thanks a lot in advance.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 3:30 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks a lot Thomas.
Will take this as reference and test our application. Great!
Regards,
Raja.
From: Thomas Weise <th...@gmail.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Sunday, June 19, 2016 at 2:01 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: Re: how to increase lifetime of hdfs delegation tokens ?
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
My Apex application failed exactly after running 7days in our distributed hadoop environment, with delegation token expiry!!
Can someone pls help me with details, on how I can increase the delegation token time to lifetime or any other process running in parallel to renew the tokens ?
Exception details below:
ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
Thanks a lot in advance.
Regards,
Raja.
Re: how to increase lifetime of hdfs delegation tokens ?
Posted by Thomas Weise <th...@gmail.com>.
Token expiration working as expected!
Please have a look on how to extend or refresh it:
https://github.com/apache/apex-core/blob/master/docs/security.md#token-refresh
Thanks,
Thomas
On Sat, Jun 18, 2016 at 10:26 PM, Raja.Aravapalli <
Raja.Aravapalli@target.com> wrote:
>
> Hi,
>
> My Apex application failed exactly after running 7days in our distributed
> hadoop environment, with delegation token expiry!!
>
> Can someone pls help me with details, on how I can increase the delegation
> token time to lifetime or any other process running in parallel to renew
> the tokens ?
>
> *Exception details below:*
>
> ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(954)) - Failed to close inode 11111111
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 111111 for XXXXXX) is expired
> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>
>
>
> Thanks a lot in advance.
>
>
> Regards,
> Raja.
>
>