You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sentry.apache.org by Alexander Kolbasov <ak...@cloudera.com> on 2017/07/21 23:35:19 UTC
Splitting permissions and path updates
Lina’s email prompted an interesting thought - right now when the NameNode plugin isn’t happy and wants a full update we send it both permissions update and path update. Path update is very expensive while permissions update is usually much smaller.
It would be very useful to be able to send just full path update or just full perms update when there is a problem with just path or just permissions.
Any thoughts?
- Alex
Re: Splitting permissions and path updates
Posted by Alexander Kolbasov <ak...@cloudera.com>.
That's good. Permission updates - even full permission update shouldn't be
very expensive, so we may be more willing to send one when we detect
improper holes.
On Mon, Jul 24, 2017 at 5:59 PM, Na Li <li...@cloudera.com> wrote:
> Sasha,
>
> Yes.
>
> SentryHDFSServiceProcessor.get_authz_updates gets perm update and path
> update separately, and puts them into the response object.
>
> Thanks,
>
> Lina
>
> On Mon, Jul 24, 2017 at 10:48 AM, Alexander Kolbasov <ak...@cloudera.com>
> wrote:
>
> > Does it mean that without any changes the current code may send e.g. full
> > update for permissions and partial update for paths or visa versa?
> >
> > - Alex
> >
> > On Mon, Jul 24, 2017 at 5:36 PM, Na Li <li...@cloudera.com> wrote:
> >
> > > Sasha,
> > >
> > > When NameNode plugin asks for updates, it includes info for both
> > permission
> > > and path. However, the processing is separate. It is possible for
> Sentry
> > to
> > > send full snapshot of permission and delta change to HDFS. At Sentry,
> > perm
> > > and path processing share the same class, but they have their own
> > > instances.
> > >
> > > The current behavior can already satisfy your requirements. I have
> > > confirmed this with Sergio.
> > >
> > > You can see it in SentryPlugin.java at sentry server side.
> > >
> > > public void initialize(Configuration conf, SentryStore sentryStore)
> > > throws SentryPluginException {
> > > PermImageRetriever permImageRetriever = new
> > > PermImageRetriever(sentryStore);
> > > PathImageRetriever pathImageRetriever = new
> > > PathImageRetriever(sentryStore);
> > > PermDeltaRetriever permDeltaRetriever = new
> > > PermDeltaRetriever(sentryStore);
> > > PathDeltaRetriever pathDeltaRetriever = new
> > > PathDeltaRetriever(sentryStore);
> > > pathsUpdater = new DBUpdateForwarder<>(pathImageRetriever,
> > > pathDeltaRetriever); <- path has its own instance
> > > permsUpdater = new DBUpdateForwarder<>(permImageRetriever,
> > > permDeltaRetriever); <- perm has its own instance
> > > ...
> > > }
> > >
> > > Thanks,
> > >
> > > Lina
> > >
> > > On Fri, Jul 21, 2017 at 6:35 PM, Alexander Kolbasov <
> akolb@cloudera.com>
> > > wrote:
> > >
> > > > Lina’s email prompted an interesting thought - right now when the
> > > NameNode
> > > > plugin isn’t happy and wants a full update we send it both
> permissions
> > > > update and path update. Path update is very expensive while
> permissions
> > > > update is usually much smaller.
> > > >
> > > > It would be very useful to be able to send just full path update or
> > just
> > > > full perms update when there is a problem with just path or just
> > > > permissions.
> > > >
> > > > Any thoughts?
> > > >
> > > > - Alex
> > >
> >
>
Re: Splitting permissions and path updates
Posted by Na Li <li...@cloudera.com>.
Sasha,
Yes.
SentryHDFSServiceProcessor.get_authz_updates gets perm update and path
update separately, and puts them into the response object.
Thanks,
Lina
On Mon, Jul 24, 2017 at 10:48 AM, Alexander Kolbasov <ak...@cloudera.com>
wrote:
> Does it mean that without any changes the current code may send e.g. full
> update for permissions and partial update for paths or visa versa?
>
> - Alex
>
> On Mon, Jul 24, 2017 at 5:36 PM, Na Li <li...@cloudera.com> wrote:
>
> > Sasha,
> >
> > When NameNode plugin asks for updates, it includes info for both
> permission
> > and path. However, the processing is separate. It is possible for Sentry
> to
> > send full snapshot of permission and delta change to HDFS. At Sentry,
> perm
> > and path processing share the same class, but they have their own
> > instances.
> >
> > The current behavior can already satisfy your requirements. I have
> > confirmed this with Sergio.
> >
> > You can see it in SentryPlugin.java at sentry server side.
> >
> > public void initialize(Configuration conf, SentryStore sentryStore)
> > throws SentryPluginException {
> > PermImageRetriever permImageRetriever = new
> > PermImageRetriever(sentryStore);
> > PathImageRetriever pathImageRetriever = new
> > PathImageRetriever(sentryStore);
> > PermDeltaRetriever permDeltaRetriever = new
> > PermDeltaRetriever(sentryStore);
> > PathDeltaRetriever pathDeltaRetriever = new
> > PathDeltaRetriever(sentryStore);
> > pathsUpdater = new DBUpdateForwarder<>(pathImageRetriever,
> > pathDeltaRetriever); <- path has its own instance
> > permsUpdater = new DBUpdateForwarder<>(permImageRetriever,
> > permDeltaRetriever); <- perm has its own instance
> > ...
> > }
> >
> > Thanks,
> >
> > Lina
> >
> > On Fri, Jul 21, 2017 at 6:35 PM, Alexander Kolbasov <ak...@cloudera.com>
> > wrote:
> >
> > > Lina’s email prompted an interesting thought - right now when the
> > NameNode
> > > plugin isn’t happy and wants a full update we send it both permissions
> > > update and path update. Path update is very expensive while permissions
> > > update is usually much smaller.
> > >
> > > It would be very useful to be able to send just full path update or
> just
> > > full perms update when there is a problem with just path or just
> > > permissions.
> > >
> > > Any thoughts?
> > >
> > > - Alex
> >
>
Re: Splitting permissions and path updates
Posted by Alexander Kolbasov <ak...@cloudera.com>.
Does it mean that without any changes the current code may send e.g. full
update for permissions and partial update for paths or visa versa?
- Alex
On Mon, Jul 24, 2017 at 5:36 PM, Na Li <li...@cloudera.com> wrote:
> Sasha,
>
> When NameNode plugin asks for updates, it includes info for both permission
> and path. However, the processing is separate. It is possible for Sentry to
> send full snapshot of permission and delta change to HDFS. At Sentry, perm
> and path processing share the same class, but they have their own
> instances.
>
> The current behavior can already satisfy your requirements. I have
> confirmed this with Sergio.
>
> You can see it in SentryPlugin.java at sentry server side.
>
> public void initialize(Configuration conf, SentryStore sentryStore)
> throws SentryPluginException {
> PermImageRetriever permImageRetriever = new
> PermImageRetriever(sentryStore);
> PathImageRetriever pathImageRetriever = new
> PathImageRetriever(sentryStore);
> PermDeltaRetriever permDeltaRetriever = new
> PermDeltaRetriever(sentryStore);
> PathDeltaRetriever pathDeltaRetriever = new
> PathDeltaRetriever(sentryStore);
> pathsUpdater = new DBUpdateForwarder<>(pathImageRetriever,
> pathDeltaRetriever); <- path has its own instance
> permsUpdater = new DBUpdateForwarder<>(permImageRetriever,
> permDeltaRetriever); <- perm has its own instance
> ...
> }
>
> Thanks,
>
> Lina
>
> On Fri, Jul 21, 2017 at 6:35 PM, Alexander Kolbasov <ak...@cloudera.com>
> wrote:
>
> > Lina’s email prompted an interesting thought - right now when the
> NameNode
> > plugin isn’t happy and wants a full update we send it both permissions
> > update and path update. Path update is very expensive while permissions
> > update is usually much smaller.
> >
> > It would be very useful to be able to send just full path update or just
> > full perms update when there is a problem with just path or just
> > permissions.
> >
> > Any thoughts?
> >
> > - Alex
>
Re: Splitting permissions and path updates
Posted by Na Li <li...@cloudera.com>.
Sasha,
When NameNode plugin asks for updates, it includes info for both permission
and path. However, the processing is separate. It is possible for Sentry to
send full snapshot of permission and delta change to HDFS. At Sentry, perm
and path processing share the same class, but they have their own
instances.
The current behavior can already satisfy your requirements. I have
confirmed this with Sergio.
You can see it in SentryPlugin.java at sentry server side.
public void initialize(Configuration conf, SentryStore sentryStore)
throws SentryPluginException {
PermImageRetriever permImageRetriever = new
PermImageRetriever(sentryStore);
PathImageRetriever pathImageRetriever = new
PathImageRetriever(sentryStore);
PermDeltaRetriever permDeltaRetriever = new
PermDeltaRetriever(sentryStore);
PathDeltaRetriever pathDeltaRetriever = new
PathDeltaRetriever(sentryStore);
pathsUpdater = new DBUpdateForwarder<>(pathImageRetriever,
pathDeltaRetriever); <- path has its own instance
permsUpdater = new DBUpdateForwarder<>(permImageRetriever,
permDeltaRetriever); <- perm has its own instance
...
}
Thanks,
Lina
On Fri, Jul 21, 2017 at 6:35 PM, Alexander Kolbasov <ak...@cloudera.com>
wrote:
> Lina’s email prompted an interesting thought - right now when the NameNode
> plugin isn’t happy and wants a full update we send it both permissions
> update and path update. Path update is very expensive while permissions
> update is usually much smaller.
>
> It would be very useful to be able to send just full path update or just
> full perms update when there is a problem with just path or just
> permissions.
>
> Any thoughts?
>
> - Alex