You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by James Srinivasan <ja...@gmail.com> on 2019/06/13 19:10:10 UTC
Kerberos Ticket Renewal (when not updating Hadoop user)
Hi all,
I'm finally getting around to fixing up some deprecation issues with
our use of Kerberos with Accumulo and GeoMesa
(https://github.com/locationtech/geomesa/). Because I didn't know any
better at the time, I used the KerberosToken ctor specifying that the
Hadoop user should be replaced. Combined with a thread to periodically
renew the ticket (calling
UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()),
this has worked nicely for us.
However, there are some unfortunate side effects of updating the
Hadoop user - for instance, subsequent HDFS operations use the new
user, who may not have the same permissions as the original user in a
Zeppelin-type notebook environment. Plus the replaceCurrentUser param
is deprecated and removed in Accumulo 2.0. So I'm keen on not
replacing the Hadoop user, but how do I handle ticket renewal?
Thanks very much,
James
Re: Kerberos Ticket Renewal (when not updating Hadoop user)
Posted by James Srinivasan <ja...@gmail.com>.
Hi all,
Do you know of any example open source projects that support Accumulo
using both username/password and Kerberos in a neat manner?
Thanks very much,
James
On Fri, 14 Jun 2019 at 15:02, Josh Elser <el...@apache.org> wrote:
>
> My point was that this is a problem you need to solve in GeoMesa and it
> doesn't matter how you go about doing it :). A proxy around Connector
> would be one way to do this, if that's what you're exposing to your users.
>
> Like I said earlier, you should be ensuring that any call which may
> result in an RPC is wrapped in a doAs().
>
> On 6/13/19 4:41 PM, Emilio Lahr-Vivaz wrote:
> > Hello,
> >
> > GeoMesa is a library (providing a GeoTools data store backed by
> > Accumulo, among other things), so there isn't a single entry point. We
> > could try to wrap every method call, but that would likely be complex
> > (the GeoTools API has a lot of surface area).
> >
> > Would it make sense to create a delegate proxy for the Accumulo
> > connection instance, and wrap all its methods with a doAs()? Sometimes
> > we create additional threads, or create a scanner but then pass it off
> > before reading it - would we need to wrap e.g. every call to next() on a
> > scanner, or would wrapping the connector call to create it be enough?
> >
> > Thanks,
> >
> > Emilio
> >
> > On 6/13/19 4:09 PM, Josh Elser wrote:
> >> Yes. Anything in which you're interacting with Accumulo (read-as,
> >> anything that's going to execute an RPC to talk to the Master or a
> >> TabletServer) needs to be wrapped in a doAs().
> >>
> >> What's often easiest is to wrap your "entry point" in a doAs().
> >>
> >> For example, if you had some class with a main:
> >>
> >> public class MyGeoMesa {
> >> public MyGeoMesa() {}
> >> public void do() {...}
> >> public static void main(String[] args) {
> >> (new MyGeoMesa()).do();
> >> }
> >> }
> >>
> >> You could turn that `do()` call into:
> >>
> >> UserGroupInformation user =
> >> UserGroupInformation.loginFromKeytabAndReturnUGI(..):
> >> user.doAs(new PrivilegedExecution<Void>() {
> >> public Void run() {
> >> (new MyGeoMesa()).do();
> >> return null;
> >> }
> >> });
> >>
> >> On 6/13/19 3:38 PM, James Srinivasan wrote:
> >>> Thanks, I think that helps. If I'm no longer updating the Hadoop user,
> >>> do I have to use doAs for any operation that touches Accumulo? I fear
> >>> there are lots...
> >>>
> >>> James
> >>>
> >>> On Thu, 13 Jun 2019 at 20:21, Josh Elser <el...@apache.org> wrote:
> >>>>
> >>>> Hi James,
> >>>>
> >>>> A thread calling checkTGTAndReloginFromKeytab() is still what you want
> >>>> for renewals. Just make sure you wrap that in a UGI.doAs() for the user
> >>>> whose ticket you want to renew.
> >>>>
> >>>> In general, you just want to wrap any Accumulo-related calls in a
> >>>> doAs()
> >>>> to avoid fallback onto the "loginUser" semantics that UGI has. Being
> >>>> explicit with a doAs() for the user you want to run some code as is the
> >>>> trick.
> >>>>
> >>>> Does that help?
> >>>>
> >>>> On 6/13/19 3:10 PM, James Srinivasan wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> I'm finally getting around to fixing up some deprecation issues with
> >>>>> our use of Kerberos with Accumulo and GeoMesa
> >>>>> (https://github.com/locationtech/geomesa/). Because I didn't know any
> >>>>> better at the time, I used the KerberosToken ctor specifying that the
> >>>>> Hadoop user should be replaced. Combined with a thread to periodically
> >>>>> renew the ticket (calling
> >>>>> UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()),
> >>>>> this has worked nicely for us.
> >>>>>
> >>>>> However, there are some unfortunate side effects of updating the
> >>>>> Hadoop user - for instance, subsequent HDFS operations use the new
> >>>>> user, who may not have the same permissions as the original user in a
> >>>>> Zeppelin-type notebook environment. Plus the replaceCurrentUser param
> >>>>> is deprecated and removed in Accumulo 2.0. So I'm keen on not
> >>>>> replacing the Hadoop user, but how do I handle ticket renewal?
> >>>>>
> >>>>> Thanks very much,
> >>>>>
> >>>>> James
> >>>>>
> >
Re: Kerberos Ticket Renewal (when not updating Hadoop user)
Posted by Josh Elser <el...@apache.org>.
My point was that this is a problem you need to solve in GeoMesa and it
doesn't matter how you go about doing it :). A proxy around Connector
would be one way to do this, if that's what you're exposing to your users.
Like I said earlier, you should be ensuring that any call which may
result in an RPC is wrapped in a doAs().
On 6/13/19 4:41 PM, Emilio Lahr-Vivaz wrote:
> Hello,
>
> GeoMesa is a library (providing a GeoTools data store backed by
> Accumulo, among other things), so there isn't a single entry point. We
> could try to wrap every method call, but that would likely be complex
> (the GeoTools API has a lot of surface area).
>
> Would it make sense to create a delegate proxy for the Accumulo
> connection instance, and wrap all its methods with a doAs()? Sometimes
> we create additional threads, or create a scanner but then pass it off
> before reading it - would we need to wrap e.g. every call to next() on a
> scanner, or would wrapping the connector call to create it be enough?
>
> Thanks,
>
> Emilio
>
> On 6/13/19 4:09 PM, Josh Elser wrote:
>> Yes. Anything in which you're interacting with Accumulo (read-as,
>> anything that's going to execute an RPC to talk to the Master or a
>> TabletServer) needs to be wrapped in a doAs().
>>
>> What's often easiest is to wrap your "entry point" in a doAs().
>>
>> For example, if you had some class with a main:
>>
>> public class MyGeoMesa {
>> public MyGeoMesa() {}
>> public void do() {...}
>> public static void main(String[] args) {
>> (new MyGeoMesa()).do();
>> }
>> }
>>
>> You could turn that `do()` call into:
>>
>> UserGroupInformation user =
>> UserGroupInformation.loginFromKeytabAndReturnUGI(..):
>> user.doAs(new PrivilegedExecution<Void>() {
>> public Void run() {
>> (new MyGeoMesa()).do();
>> return null;
>> }
>> });
>>
>> On 6/13/19 3:38 PM, James Srinivasan wrote:
>>> Thanks, I think that helps. If I'm no longer updating the Hadoop user,
>>> do I have to use doAs for any operation that touches Accumulo? I fear
>>> there are lots...
>>>
>>> James
>>>
>>> On Thu, 13 Jun 2019 at 20:21, Josh Elser <el...@apache.org> wrote:
>>>>
>>>> Hi James,
>>>>
>>>> A thread calling checkTGTAndReloginFromKeytab() is still what you want
>>>> for renewals. Just make sure you wrap that in a UGI.doAs() for the user
>>>> whose ticket you want to renew.
>>>>
>>>> In general, you just want to wrap any Accumulo-related calls in a
>>>> doAs()
>>>> to avoid fallback onto the "loginUser" semantics that UGI has. Being
>>>> explicit with a doAs() for the user you want to run some code as is the
>>>> trick.
>>>>
>>>> Does that help?
>>>>
>>>> On 6/13/19 3:10 PM, James Srinivasan wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm finally getting around to fixing up some deprecation issues with
>>>>> our use of Kerberos with Accumulo and GeoMesa
>>>>> (https://github.com/locationtech/geomesa/). Because I didn't know any
>>>>> better at the time, I used the KerberosToken ctor specifying that the
>>>>> Hadoop user should be replaced. Combined with a thread to periodically
>>>>> renew the ticket (calling
>>>>> UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()),
>>>>> this has worked nicely for us.
>>>>>
>>>>> However, there are some unfortunate side effects of updating the
>>>>> Hadoop user - for instance, subsequent HDFS operations use the new
>>>>> user, who may not have the same permissions as the original user in a
>>>>> Zeppelin-type notebook environment. Plus the replaceCurrentUser param
>>>>> is deprecated and removed in Accumulo 2.0. So I'm keen on not
>>>>> replacing the Hadoop user, but how do I handle ticket renewal?
>>>>>
>>>>> Thanks very much,
>>>>>
>>>>> James
>>>>>
>
Re: Kerberos Ticket Renewal (when not updating Hadoop user)
Posted by Emilio Lahr-Vivaz <el...@ccri.com>.
Hello,
GeoMesa is a library (providing a GeoTools data store backed by
Accumulo, among other things), so there isn't a single entry point. We
could try to wrap every method call, but that would likely be complex
(the GeoTools API has a lot of surface area).
Would it make sense to create a delegate proxy for the Accumulo
connection instance, and wrap all its methods with a doAs()? Sometimes
we create additional threads, or create a scanner but then pass it off
before reading it - would we need to wrap e.g. every call to next() on a
scanner, or would wrapping the connector call to create it be enough?
Thanks,
Emilio
On 6/13/19 4:09 PM, Josh Elser wrote:
> Yes. Anything in which you're interacting with Accumulo (read-as,
> anything that's going to execute an RPC to talk to the Master or a
> TabletServer) needs to be wrapped in a doAs().
>
> What's often easiest is to wrap your "entry point" in a doAs().
>
> For example, if you had some class with a main:
>
> public class MyGeoMesa {
> public MyGeoMesa() {}
> public void do() {...}
> public static void main(String[] args) {
> (new MyGeoMesa()).do();
> }
> }
>
> You could turn that `do()` call into:
>
> UserGroupInformation user =
> UserGroupInformation.loginFromKeytabAndReturnUGI(..):
> user.doAs(new PrivilegedExecution<Void>() {
> public Void run() {
> (new MyGeoMesa()).do();
> return null;
> }
> });
>
> On 6/13/19 3:38 PM, James Srinivasan wrote:
>> Thanks, I think that helps. If I'm no longer updating the Hadoop user,
>> do I have to use doAs for any operation that touches Accumulo? I fear
>> there are lots...
>>
>> James
>>
>> On Thu, 13 Jun 2019 at 20:21, Josh Elser <el...@apache.org> wrote:
>>>
>>> Hi James,
>>>
>>> A thread calling checkTGTAndReloginFromKeytab() is still what you want
>>> for renewals. Just make sure you wrap that in a UGI.doAs() for the user
>>> whose ticket you want to renew.
>>>
>>> In general, you just want to wrap any Accumulo-related calls in a
>>> doAs()
>>> to avoid fallback onto the "loginUser" semantics that UGI has. Being
>>> explicit with a doAs() for the user you want to run some code as is the
>>> trick.
>>>
>>> Does that help?
>>>
>>> On 6/13/19 3:10 PM, James Srinivasan wrote:
>>>> Hi all,
>>>>
>>>> I'm finally getting around to fixing up some deprecation issues with
>>>> our use of Kerberos with Accumulo and GeoMesa
>>>> (https://github.com/locationtech/geomesa/). Because I didn't know any
>>>> better at the time, I used the KerberosToken ctor specifying that the
>>>> Hadoop user should be replaced. Combined with a thread to periodically
>>>> renew the ticket (calling
>>>> UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()),
>>>> this has worked nicely for us.
>>>>
>>>> However, there are some unfortunate side effects of updating the
>>>> Hadoop user - for instance, subsequent HDFS operations use the new
>>>> user, who may not have the same permissions as the original user in a
>>>> Zeppelin-type notebook environment. Plus the replaceCurrentUser param
>>>> is deprecated and removed in Accumulo 2.0. So I'm keen on not
>>>> replacing the Hadoop user, but how do I handle ticket renewal?
>>>>
>>>> Thanks very much,
>>>>
>>>> James
>>>>
Re: Kerberos Ticket Renewal (when not updating Hadoop user)
Posted by Josh Elser <el...@apache.org>.
Yes. Anything in which you're interacting with Accumulo (read-as,
anything that's going to execute an RPC to talk to the Master or a
TabletServer) needs to be wrapped in a doAs().
What's often easiest is to wrap your "entry point" in a doAs().
For example, if you had some class with a main:
public class MyGeoMesa {
public MyGeoMesa() {}
public void do() {...}
public static void main(String[] args) {
(new MyGeoMesa()).do();
}
}
You could turn that `do()` call into:
UserGroupInformation user =
UserGroupInformation.loginFromKeytabAndReturnUGI(..):
user.doAs(new PrivilegedExecution<Void>() {
public Void run() {
(new MyGeoMesa()).do();
return null;
}
});
On 6/13/19 3:38 PM, James Srinivasan wrote:
> Thanks, I think that helps. If I'm no longer updating the Hadoop user,
> do I have to use doAs for any operation that touches Accumulo? I fear
> there are lots...
>
> James
>
> On Thu, 13 Jun 2019 at 20:21, Josh Elser <el...@apache.org> wrote:
>>
>> Hi James,
>>
>> A thread calling checkTGTAndReloginFromKeytab() is still what you want
>> for renewals. Just make sure you wrap that in a UGI.doAs() for the user
>> whose ticket you want to renew.
>>
>> In general, you just want to wrap any Accumulo-related calls in a doAs()
>> to avoid fallback onto the "loginUser" semantics that UGI has. Being
>> explicit with a doAs() for the user you want to run some code as is the
>> trick.
>>
>> Does that help?
>>
>> On 6/13/19 3:10 PM, James Srinivasan wrote:
>>> Hi all,
>>>
>>> I'm finally getting around to fixing up some deprecation issues with
>>> our use of Kerberos with Accumulo and GeoMesa
>>> (https://github.com/locationtech/geomesa/). Because I didn't know any
>>> better at the time, I used the KerberosToken ctor specifying that the
>>> Hadoop user should be replaced. Combined with a thread to periodically
>>> renew the ticket (calling
>>> UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()),
>>> this has worked nicely for us.
>>>
>>> However, there are some unfortunate side effects of updating the
>>> Hadoop user - for instance, subsequent HDFS operations use the new
>>> user, who may not have the same permissions as the original user in a
>>> Zeppelin-type notebook environment. Plus the replaceCurrentUser param
>>> is deprecated and removed in Accumulo 2.0. So I'm keen on not
>>> replacing the Hadoop user, but how do I handle ticket renewal?
>>>
>>> Thanks very much,
>>>
>>> James
>>>
Re: Kerberos Ticket Renewal (when not updating Hadoop user)
Posted by James Srinivasan <ja...@gmail.com>.
Thanks, I think that helps. If I'm no longer updating the Hadoop user,
do I have to use doAs for any operation that touches Accumulo? I fear
there are lots...
James
On Thu, 13 Jun 2019 at 20:21, Josh Elser <el...@apache.org> wrote:
>
> Hi James,
>
> A thread calling checkTGTAndReloginFromKeytab() is still what you want
> for renewals. Just make sure you wrap that in a UGI.doAs() for the user
> whose ticket you want to renew.
>
> In general, you just want to wrap any Accumulo-related calls in a doAs()
> to avoid fallback onto the "loginUser" semantics that UGI has. Being
> explicit with a doAs() for the user you want to run some code as is the
> trick.
>
> Does that help?
>
> On 6/13/19 3:10 PM, James Srinivasan wrote:
> > Hi all,
> >
> > I'm finally getting around to fixing up some deprecation issues with
> > our use of Kerberos with Accumulo and GeoMesa
> > (https://github.com/locationtech/geomesa/). Because I didn't know any
> > better at the time, I used the KerberosToken ctor specifying that the
> > Hadoop user should be replaced. Combined with a thread to periodically
> > renew the ticket (calling
> > UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()),
> > this has worked nicely for us.
> >
> > However, there are some unfortunate side effects of updating the
> > Hadoop user - for instance, subsequent HDFS operations use the new
> > user, who may not have the same permissions as the original user in a
> > Zeppelin-type notebook environment. Plus the replaceCurrentUser param
> > is deprecated and removed in Accumulo 2.0. So I'm keen on not
> > replacing the Hadoop user, but how do I handle ticket renewal?
> >
> > Thanks very much,
> >
> > James
> >
Re: Kerberos Ticket Renewal (when not updating Hadoop user)
Posted by Josh Elser <el...@apache.org>.
Hi James,
A thread calling checkTGTAndReloginFromKeytab() is still what you want
for renewals. Just make sure you wrap that in a UGI.doAs() for the user
whose ticket you want to renew.
In general, you just want to wrap any Accumulo-related calls in a doAs()
to avoid fallback onto the "loginUser" semantics that UGI has. Being
explicit with a doAs() for the user you want to run some code as is the
trick.
Does that help?
On 6/13/19 3:10 PM, James Srinivasan wrote:
> Hi all,
>
> I'm finally getting around to fixing up some deprecation issues with
> our use of Kerberos with Accumulo and GeoMesa
> (https://github.com/locationtech/geomesa/). Because I didn't know any
> better at the time, I used the KerberosToken ctor specifying that the
> Hadoop user should be replaced. Combined with a thread to periodically
> renew the ticket (calling
> UserGroupInformation.getCurrentUser.checkTGTAndReloginFromKeytab()),
> this has worked nicely for us.
>
> However, there are some unfortunate side effects of updating the
> Hadoop user - for instance, subsequent HDFS operations use the new
> user, who may not have the same permissions as the original user in a
> Zeppelin-type notebook environment. Plus the replaceCurrentUser param
> is deprecated and removed in Accumulo 2.0. So I'm keen on not
> replacing the Hadoop user, but how do I handle ticket renewal?
>
> Thanks very much,
>
> James
>