You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Pat White <pa...@oath.com> on 2018/12/19 21:50:33 UTC
Question regarding krb tgt renewal for Hive processors and connection pools
Hi Folks,
Using kerberos auth in Nifi clusters communicating with hdfs and for hive
access, the ticket life is 24 hours. Hdfs works fine, however we're seeing
issues with hive where the tgt doesn't seem to renew, or fetch a new
ticket, as the 24hr limit approaches. Hence, hive access works fine until
the 24hrs expires and then fails to authenticate. For example, a
SelectHiveQL processor using the Hive Database Connection Pooling Service
will work for 24 hours after a cluster restart but then fail with:
org.ietf.jgss.GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)
Enabled krb debugging, which shows the ticket is found but no renew, or new
fetch attempt, seems to have been made. Krb docs discuss
setting javax.security.auth.useSubjectCredsOnly=false in order to allow the
underlying mechanism to obtain credentials, however the bootstrap.conf
explicitly sets this to 'true', to inhibit JAAS from using any fallback
methods to authenticate.
Trying an experiment with useSubjectCredsOnly=false but would appreciate if
anyone has some guidance on this, how to get hive's connection pools to
renew tgt or fetch a new ticket ? Thank you.
patw
Re: Question regarding krb tgt renewal for Hive processors and
connection pools
Posted by Pat White <pa...@oath.com>.
No worries Jeff, the fix and the help were great, thanks again for
providing the PR.
patw
On Sun, Dec 23, 2018 at 12:01 PM Jeff <jt...@gmail.com> wrote:
> Pat,
>
> I apologize for not seeing this thread until today! I'm glad there was a
> commit available for you to cherry-pick and resolve your issue. Also,
> thanks to Shawn and Bryan the helpful comments!
>
> - Jeff
>
> On Sun, Dec 23, 2018 at 10:13 AM Pat White <pa...@oath.com> wrote:
>
>> Update, cherrypicking the fix from NIFI-5134 into 1.6.0 looks good to
>> resolve hive connectionpool tgt renew/fetch issue we're seeing.
>> Thanks again to Shawn and Bryan for the pointers, and to Jeff for the
>> original PR.
>>
>> patw
>>
>> On Wed, Dec 19, 2018 at 5:22 PM Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>>> It’s nifi-5134 that fixes this issue. Prior to that the hive connection
>>> pool did not renew its Kerberos ticket correctly.
>>>
>>> Sent from my iPhone
>>>
>>> On Dec 19, 2018, at 5:15 PM, Pat White <pa...@oath.com> wrote:
>>>
>>> Thanks much Bryan and Shawn, we're currently on 1.6.0 with some
>>> cherrypicks from 1.8.0 jiras.
>>> Will check the archives as mentioned, thanks again.
>>>
>>> patw
>>>
>>> On Wed, Dec 19, 2018 at 4:45 PM Shawn Weeks <sw...@weeksconsulting.us>
>>> wrote:
>>>
>>>> There is a bug for this but I’m not sure which release fixed it.
>>>> Something after 1.5 I think. The patch is in the hortonworks hdf 3.1.2
>>>> release.
>>>>
>>>> If you go search for me in the archives I mentioned it a few months
>>>> back.
>>>>
>>>> Thanks
>>>> Shawn
>>>>
>>>> Sent from my iPhone
>>>>
>>>> > On Dec 19, 2018, at 3:59 PM, Pat White <pa...@oath.com> wrote:
>>>> >
>>>> > Hi Folks,
>>>> >
>>>> > Using kerberos auth in Nifi clusters communicating with hdfs and for
>>>> hive access, the ticket life is 24 hours. Hdfs works fine, however we're
>>>> seeing issues with hive where the tgt doesn't seem to renew, or fetch a new
>>>> ticket, as the 24hr limit approaches. Hence, hive access works fine until
>>>> the 24hrs expires and then fails to authenticate. For example, a
>>>> SelectHiveQL processor using the Hive Database Connection Pooling Service
>>>> will work for 24 hours after a cluster restart but then fail with:
>>>> >
>>>> > org.ietf.jgss.GSSException: No valid credentials provided
>>>> > (Mechanism level: Failed to find any Kerberos tgt)
>>>> >
>>>> > Enabled krb debugging, which shows the ticket is found but no renew,
>>>> or new fetch attempt, seems to have been made. Krb docs discuss setting
>>>> javax.security.auth.useSubjectCredsOnly=false in order to allow the
>>>> underlying mechanism to obtain credentials, however the bootstrap.conf
>>>> explicitly sets this to 'true', to inhibit JAAS from using any fallback
>>>> methods to authenticate.
>>>> >
>>>> > Trying an experiment with useSubjectCredsOnly=false but would
>>>> appreciate if anyone has some guidance on this, how to get hive's
>>>> connection pools to renew tgt or fetch a new ticket ? Thank you.
>>>> >
>>>> > patw
>>>> >
>>>> >
>>>> >
>>>>
>>>
Re: Question regarding krb tgt renewal for Hive processors and
connection pools
Posted by Jeff <jt...@gmail.com>.
Pat,
I apologize for not seeing this thread until today! I'm glad there was a
commit available for you to cherry-pick and resolve your issue. Also,
thanks to Shawn and Bryan the helpful comments!
- Jeff
On Sun, Dec 23, 2018 at 10:13 AM Pat White <pa...@oath.com> wrote:
> Update, cherrypicking the fix from NIFI-5134 into 1.6.0 looks good to
> resolve hive connectionpool tgt renew/fetch issue we're seeing.
> Thanks again to Shawn and Bryan for the pointers, and to Jeff for the
> original PR.
>
> patw
>
> On Wed, Dec 19, 2018 at 5:22 PM Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>> It’s nifi-5134 that fixes this issue. Prior to that the hive connection
>> pool did not renew its Kerberos ticket correctly.
>>
>> Sent from my iPhone
>>
>> On Dec 19, 2018, at 5:15 PM, Pat White <pa...@oath.com> wrote:
>>
>> Thanks much Bryan and Shawn, we're currently on 1.6.0 with some
>> cherrypicks from 1.8.0 jiras.
>> Will check the archives as mentioned, thanks again.
>>
>> patw
>>
>> On Wed, Dec 19, 2018 at 4:45 PM Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>>> There is a bug for this but I’m not sure which release fixed it.
>>> Something after 1.5 I think. The patch is in the hortonworks hdf 3.1.2
>>> release.
>>>
>>> If you go search for me in the archives I mentioned it a few months
>>> back.
>>>
>>> Thanks
>>> Shawn
>>>
>>> Sent from my iPhone
>>>
>>> > On Dec 19, 2018, at 3:59 PM, Pat White <pa...@oath.com> wrote:
>>> >
>>> > Hi Folks,
>>> >
>>> > Using kerberos auth in Nifi clusters communicating with hdfs and for
>>> hive access, the ticket life is 24 hours. Hdfs works fine, however we're
>>> seeing issues with hive where the tgt doesn't seem to renew, or fetch a new
>>> ticket, as the 24hr limit approaches. Hence, hive access works fine until
>>> the 24hrs expires and then fails to authenticate. For example, a
>>> SelectHiveQL processor using the Hive Database Connection Pooling Service
>>> will work for 24 hours after a cluster restart but then fail with:
>>> >
>>> > org.ietf.jgss.GSSException: No valid credentials provided
>>> > (Mechanism level: Failed to find any Kerberos tgt)
>>> >
>>> > Enabled krb debugging, which shows the ticket is found but no renew,
>>> or new fetch attempt, seems to have been made. Krb docs discuss setting
>>> javax.security.auth.useSubjectCredsOnly=false in order to allow the
>>> underlying mechanism to obtain credentials, however the bootstrap.conf
>>> explicitly sets this to 'true', to inhibit JAAS from using any fallback
>>> methods to authenticate.
>>> >
>>> > Trying an experiment with useSubjectCredsOnly=false but would
>>> appreciate if anyone has some guidance on this, how to get hive's
>>> connection pools to renew tgt or fetch a new ticket ? Thank you.
>>> >
>>> > patw
>>> >
>>> >
>>> >
>>>
>>
Re: Question regarding krb tgt renewal for Hive processors and
connection pools
Posted by Pat White <pa...@oath.com>.
Update, cherrypicking the fix from NIFI-5134 into 1.6.0 looks good to
resolve hive connectionpool tgt renew/fetch issue we're seeing.
Thanks again to Shawn and Bryan for the pointers, and to Jeff for the
original PR.
patw
On Wed, Dec 19, 2018 at 5:22 PM Shawn Weeks <sw...@weeksconsulting.us>
wrote:
> It’s nifi-5134 that fixes this issue. Prior to that the hive connection
> pool did not renew its Kerberos ticket correctly.
>
> Sent from my iPhone
>
> On Dec 19, 2018, at 5:15 PM, Pat White <pa...@oath.com> wrote:
>
> Thanks much Bryan and Shawn, we're currently on 1.6.0 with some
> cherrypicks from 1.8.0 jiras.
> Will check the archives as mentioned, thanks again.
>
> patw
>
> On Wed, Dec 19, 2018 at 4:45 PM Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>> There is a bug for this but I’m not sure which release fixed it.
>> Something after 1.5 I think. The patch is in the hortonworks hdf 3.1.2
>> release.
>>
>> If you go search for me in the archives I mentioned it a few months back.
>>
>> Thanks
>> Shawn
>>
>> Sent from my iPhone
>>
>> > On Dec 19, 2018, at 3:59 PM, Pat White <pa...@oath.com> wrote:
>> >
>> > Hi Folks,
>> >
>> > Using kerberos auth in Nifi clusters communicating with hdfs and for
>> hive access, the ticket life is 24 hours. Hdfs works fine, however we're
>> seeing issues with hive where the tgt doesn't seem to renew, or fetch a new
>> ticket, as the 24hr limit approaches. Hence, hive access works fine until
>> the 24hrs expires and then fails to authenticate. For example, a
>> SelectHiveQL processor using the Hive Database Connection Pooling Service
>> will work for 24 hours after a cluster restart but then fail with:
>> >
>> > org.ietf.jgss.GSSException: No valid credentials provided
>> > (Mechanism level: Failed to find any Kerberos tgt)
>> >
>> > Enabled krb debugging, which shows the ticket is found but no renew, or
>> new fetch attempt, seems to have been made. Krb docs discuss setting
>> javax.security.auth.useSubjectCredsOnly=false in order to allow the
>> underlying mechanism to obtain credentials, however the bootstrap.conf
>> explicitly sets this to 'true', to inhibit JAAS from using any fallback
>> methods to authenticate.
>> >
>> > Trying an experiment with useSubjectCredsOnly=false but would
>> appreciate if anyone has some guidance on this, how to get hive's
>> connection pools to renew tgt or fetch a new ticket ? Thank you.
>> >
>> > patw
>> >
>> >
>> >
>>
>
Re: Question regarding krb tgt renewal for Hive processors and
connection pools
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
It’s nifi-5134 that fixes this issue. Prior to that the hive connection pool did not renew its Kerberos ticket correctly.
Sent from my iPhone
On Dec 19, 2018, at 5:15 PM, Pat White <pa...@oath.com>> wrote:
Thanks much Bryan and Shawn, we're currently on 1.6.0 with some cherrypicks from 1.8.0 jiras.
Will check the archives as mentioned, thanks again.
patw
On Wed, Dec 19, 2018 at 4:45 PM Shawn Weeks <sw...@weeksconsulting.us>> wrote:
There is a bug for this but I’m not sure which release fixed it. Something after 1.5 I think. The patch is in the hortonworks hdf 3.1.2 release.
If you go search for me in the archives I mentioned it a few months back.
Thanks
Shawn
Sent from my iPhone
> On Dec 19, 2018, at 3:59 PM, Pat White <pa...@oath.com>> wrote:
>
> Hi Folks,
>
> Using kerberos auth in Nifi clusters communicating with hdfs and for hive access, the ticket life is 24 hours. Hdfs works fine, however we're seeing issues with hive where the tgt doesn't seem to renew, or fetch a new ticket, as the 24hr limit approaches. Hence, hive access works fine until the 24hrs expires and then fails to authenticate. For example, a SelectHiveQL processor using the Hive Database Connection Pooling Service will work for 24 hours after a cluster restart but then fail with:
>
> org.ietf.jgss.GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)
>
> Enabled krb debugging, which shows the ticket is found but no renew, or new fetch attempt, seems to have been made. Krb docs discuss setting javax.security.auth.useSubjectCredsOnly=false in order to allow the underlying mechanism to obtain credentials, however the bootstrap.conf explicitly sets this to 'true', to inhibit JAAS from using any fallback methods to authenticate.
>
> Trying an experiment with useSubjectCredsOnly=false but would appreciate if anyone has some guidance on this, how to get hive's connection pools to renew tgt or fetch a new ticket ? Thank you.
>
> patw
>
>
>
Re: Question regarding krb tgt renewal for Hive processors and
connection pools
Posted by Pat White <pa...@oath.com>.
Thanks much Bryan and Shawn, we're currently on 1.6.0 with some cherrypicks
from 1.8.0 jiras.
Will check the archives as mentioned, thanks again.
patw
On Wed, Dec 19, 2018 at 4:45 PM Shawn Weeks <sw...@weeksconsulting.us>
wrote:
> There is a bug for this but I’m not sure which release fixed it. Something
> after 1.5 I think. The patch is in the hortonworks hdf 3.1.2 release.
>
> If you go search for me in the archives I mentioned it a few months back.
>
> Thanks
> Shawn
>
> Sent from my iPhone
>
> > On Dec 19, 2018, at 3:59 PM, Pat White <pa...@oath.com> wrote:
> >
> > Hi Folks,
> >
> > Using kerberos auth in Nifi clusters communicating with hdfs and for
> hive access, the ticket life is 24 hours. Hdfs works fine, however we're
> seeing issues with hive where the tgt doesn't seem to renew, or fetch a new
> ticket, as the 24hr limit approaches. Hence, hive access works fine until
> the 24hrs expires and then fails to authenticate. For example, a
> SelectHiveQL processor using the Hive Database Connection Pooling Service
> will work for 24 hours after a cluster restart but then fail with:
> >
> > org.ietf.jgss.GSSException: No valid credentials provided
> > (Mechanism level: Failed to find any Kerberos tgt)
> >
> > Enabled krb debugging, which shows the ticket is found but no renew, or
> new fetch attempt, seems to have been made. Krb docs discuss setting
> javax.security.auth.useSubjectCredsOnly=false in order to allow the
> underlying mechanism to obtain credentials, however the bootstrap.conf
> explicitly sets this to 'true', to inhibit JAAS from using any fallback
> methods to authenticate.
> >
> > Trying an experiment with useSubjectCredsOnly=false but would appreciate
> if anyone has some guidance on this, how to get hive's connection pools to
> renew tgt or fetch a new ticket ? Thank you.
> >
> > patw
> >
> >
> >
>
Re: Question regarding krb tgt renewal for Hive processors and
connection pools
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
There is a bug for this but I’m not sure which release fixed it. Something after 1.5 I think. The patch is in the hortonworks hdf 3.1.2 release.
If you go search for me in the archives I mentioned it a few months back.
Thanks
Shawn
Sent from my iPhone
> On Dec 19, 2018, at 3:59 PM, Pat White <pa...@oath.com> wrote:
>
> Hi Folks,
>
> Using kerberos auth in Nifi clusters communicating with hdfs and for hive access, the ticket life is 24 hours. Hdfs works fine, however we're seeing issues with hive where the tgt doesn't seem to renew, or fetch a new ticket, as the 24hr limit approaches. Hence, hive access works fine until the 24hrs expires and then fails to authenticate. For example, a SelectHiveQL processor using the Hive Database Connection Pooling Service will work for 24 hours after a cluster restart but then fail with:
>
> org.ietf.jgss.GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)
>
> Enabled krb debugging, which shows the ticket is found but no renew, or new fetch attempt, seems to have been made. Krb docs discuss setting javax.security.auth.useSubjectCredsOnly=false in order to allow the underlying mechanism to obtain credentials, however the bootstrap.conf explicitly sets this to 'true', to inhibit JAAS from using any fallback methods to authenticate.
>
> Trying an experiment with useSubjectCredsOnly=false but would appreciate if anyone has some guidance on this, how to get hive's connection pools to renew tgt or fetch a new ticket ? Thank you.
>
> patw
>
>
>
Re: Question regarding krb tgt renewal for Hive processors and
connection pools
Posted by Bryan Bende <bb...@gmail.com>.
Hi Pat,
I’m personally not that familiar with Hive, but for those that are, they
will probably need to know what version of NiFi you are using since some
bugs have been fixed along the way.
Thanks,
Bryan
On Wed, Dec 19, 2018 at 4:59 PM Pat White <pa...@oath.com> wrote:
> Hi Folks,
>
> Using kerberos auth in Nifi clusters communicating with hdfs and for hive
> access, the ticket life is 24 hours. Hdfs works fine, however we're seeing
> issues with hive where the tgt doesn't seem to renew, or fetch a new
> ticket, as the 24hr limit approaches. Hence, hive access works fine until
> the 24hrs expires and then fails to authenticate. For example, a
> SelectHiveQL processor using the Hive Database Connection Pooling Service
> will work for 24 hours after a cluster restart but then fail with:
>
> org.ietf.jgss.GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)
>
> Enabled krb debugging, which shows the ticket is found but no renew, or
> new fetch attempt, seems to have been made. Krb docs discuss
> setting javax.security.auth.useSubjectCredsOnly=false in order to allow the
> underlying mechanism to obtain credentials, however the bootstrap.conf
> explicitly sets this to 'true', to inhibit JAAS from using any fallback
> methods to authenticate.
>
> Trying an experiment with useSubjectCredsOnly=false but would appreciate
> if anyone has some guidance on this, how to get hive's connection pools to
> renew tgt or fetch a new ticket ? Thank you.
>
> patw
>
>
>
> --
Sent from Gmail Mobile