You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/07/15 16:34:51 UTC
[GitHub] [accumulo] milleruntime opened a new issue #1659: Handle Internal pluggable components
milleruntime opened a new issue #1659:
URL: https://github.com/apache/accumulo/issues/1659
With the creation of the SPI package and introduction of Iterators into public API, significant progress has been made to corral the various internal pluggable components of Accumulo. It is difficult to maintain backwards compatibility of these components as they are scattered throughout the code. We recently bumped into an issue with the CompactionStrategy components in #1617 . This ticket is to create a list of the remaining components so we can figure out how to handle each of them. Ideally, they would get moved into either the SPI or API but it probably won't be that easy since most of them utilize internal types and will have to be deprecated and replaced with the newer external facing interfaces.
- org.apache.accumulo.server.fs.VolumeChooser (possibly VolumeManager?)
- org.apache.accumulo.server.tabletserver.MemoryManager
- org.apache.accumulo.server.security.handler classes (Authenticator, Authorizor, PermissionHandler)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-708522812
I am hoping #1721 can be a temporary fix for the security interfaces only to get replaced by #1221 . I found the branch that I started last year and am going to try and revisit it (https://github.com/milleruntime/accumulo/tree/sec-module).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime edited a comment on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime edited a comment on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-703807374
@cradal thanks for the help reviewing internal code but this ticket was more for handling code that currently is NOT part of the API or SPI. This was to deal with code that should either be part of the public API/SPI or removed from configuration properties, like I did with the MemoryManger. We had a property for users to write their own MemoryManager but the interface used server components that were not user friendly.
I think I will have to do the same with the interfaces in org.apache.accumulo.server.security.handler classes (Authenticator, Authorizor, PermissionHandler). These interfaces are called when Accumulo executes important security procedures (like creating users, initializing a system) but they really only work internally. It feels like they were added to add the ability for Kerberos to work with the default Zookeeper behavior. I don't know how we can rework these interfaces to extend this behavior without completely rewriting the interfaces to work in the SPI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-703807374
@cradal thanks for the help reviewing internal code but this ticket was more for handling code that currently is NOT part of the API or SPI. This was to deal with code that should either be part of the public API/SPI or removed from configuration properties, like I did with the MemoryManger. We had a property for users to write their own MemoryManager but the interface used server components that were not user friendly.
I think I will have to do the same with the interfaces in org.apache.accumulo.server.security.handler classes (Authenticator, Authorizor, PermissionHandler). These interfaces are called when Accumulo executes important security procedures (like creating users, initializing a system) but they really only work internally. It feels like they were added to add the ability for Kerberos to work with the default Zookeeper behavior. I don't know how we can rework these interfaces to extends this behavior without completely rewriting the interfaces to work in the SPI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-704972997
@ctubbsii what are your thoughts on moving the VolumeChoser to SPI? At first glance it doesn't look like there are any offending types across the interfaces, so it may just be a matter of relocation. Since the VolumeChoser doesn't directly affect user data (just where the data is stored) then it seems like the SPI is a more appropriate place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii edited a comment on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
ctubbsii edited a comment on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-705050279
> @ctubbsii what are your thoughts on moving the VolumeChoser to SPI? At first glance it doesn't look like there are any offending types across the interfaces, so it may just be a matter of relocation. Since the VolumeChoser doesn't directly affect user data (just where the data is stored) then it seems like the SPI is a more appropriate place.
I'm in favor, but would want to review its methods to be sure there's nothing we should polish first. Right now, it might be a bit Stringly typed, and we might want to consider using a proper 'Volume' type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-948841571
With the completion of the other tickets mentioned above, this is now completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-703846749
@keith-turner @ctubbsii I came up with one solution in #1721 to solve the issues with the security interfaces. It will break if a user actually has a custom security implementation so its not perfect but we maintain the Kerberos functionality and get those interfaces out of the properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-772628843
I was looking at VolumeChooser and it has some new methods added for 2.1. Thinking it would be a good candidate for moving to SPI, possibly only adding the new methods in the SPI. I am going to look into doing this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-705050279
> @ctubbsii what are your thoughts on moving the VolumeChoser to SPI? At first glance it doesn't look like there are any offending types across the interfaces, so it may just be a matter of relocation. Since the VolumeChoser doesn't directly affect user data (just where the data is stored) then it seems like the SPI is a more appropriate place.
I'm in favor, but would want to review it's methods to be sure there's nothing we should polish first. Right now, it might be a bit Stringly typed, and we might want to consider using a proper 'Volume' type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime edited a comment on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime edited a comment on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-703846749
@keith-turner @ctubbsii I came up with one solution in #1721 to solve the issues with the security interfaces. It will break if a user actually has a custom security implementation so its not perfect but we maintain the Kerberos functionality and get those interfaces out of the properties. It does that also without having to rewrite everything to work in the SPI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
keith-turner commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-658954000
The balancer is another one to consider. For some maybe instead of moving them SPI we could just remove the ability to configure them like the MemoryManager.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime closed issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime closed issue #1659:
URL: https://github.com/apache/accumulo/issues/1659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-705065036
> > @ctubbsii what are your thoughts on moving the VolumeChoser to SPI? At first glance it doesn't look like there are any offending types across the interfaces, so it may just be a matter of relocation. Since the VolumeChoser doesn't directly affect user data (just where the data is stored) then it seems like the SPI is a more appropriate place.
>
> I'm in favor, but would want to review its methods to be sure there's nothing we should polish first. Right now, it might be a bit Stringly typed, and we might want to consider using a proper 'Volume' type.
I did notice there are a lot of hadoop types. So that may be something to consider as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cradal commented on issue #1659: Handle Internal pluggable components
Posted by GitBox <gi...@apache.org>.
cradal commented on issue #1659:
URL: https://github.com/apache/accumulo/issues/1659#issuecomment-682979728
@milleruntime I reviewed .core classes and took some notes:
1) Reviewed appearance and usage of deprecated internal data types in core and discovered:
ConfiguratorBase uses the following deprecated types: RangeInputSplit, ClientConfiguration, DelegationTokenStub and ConnectorImpl.
ClientConfConverter uses ClientConfiguration.
The deprecation warnings are suppressed in ConfiguratorBase and ClientConfConverter. ConfiguratorBase and ClientConfConverter are still used in several classes, so they could be candidates for some clean them up.
2) Reviewed all the abstract classes and interfaces in .core and did not find any more that used deprecated internal types.
3) Found references to deprecated types in javadocs and pushed suggested updates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org