You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Michael Collado <co...@gmail.com> on 2024/11/05 17:42:44 UTC
Re: [DISCUSS] Apache Polaris powered by Quarkus
I added a table to the README with the differences that were called out. I
added some details that I think are worth understanding better. E.g., the
Json layout we added has specific custom functionality we wanted for
supporting key/value pairs and the micrometer annotation was added for some
custom support we wanted aside from what is supported with the default
dropwizard metrics support.
https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
I think it would be great if others could add to the table with features
that would be impacted by the migration and to call out the level of effort
in both dropwizard and quarkus.
Another possible consideration would be upgrading our Dropwizard dependency
to the latest development version. It may be the case that doing so would
address some of the targeted features with less effort in migrating.
Mike
On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi folks,
>
> Following the last community sync meeting, we create a branch
> demonstrating use of Quarkus to powered Apache Polaris:
>
> https://github.com/jbonofre/polaris/tree/QUARKUS
>
> We still have work to do, but you can already take a look and
> experiment (in the polaris-service-quarkus module).
>
> We added a README.md file to:
> 1. Highlight the main differences (in terms of code) between
> Dropwizard and Quarkus
> 2. To build and run Polaris powered by Quarkus
> 3. The list of TODO items
>
>
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
>
> If anyone wants to contribute on the branch before creating the PR,
> please let me know, I will add you as contributor on the branch.
>
> Any comments or questions are welcome !
>
> Thanks,
> Regards
> JB
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
I mean jakarta.enterprise.inject.Produces from jakarta.enterprise.cdi-api
4.1.0
On Wed, Nov 27, 2024 at 4:22 PM Michael Collado <co...@gmail.com>
wrote:
> Thanks Dmitri. The jakarta.inject api doesn't include the @Produces
> annotation - is it one that's included in the enterprise package used by
> Quarkus? I think that would eliminate some of the extra code and complexity
> in the current branch (e.g., a class can only implement one Factory<T>, but
> I could use @Produces on multiple methods in the same class).
>
> I made some (I think) substantial improvements in the application
> bootstrapping - the previous work doesn't work for the bootstrap or purge
> commands. I've posted a PR at https://github.com/apache/polaris/pull/493
> with the latest. I stripped out the cache stuff, as that's already
> implemented in the other PR I opened the other day. Take a look at this one
> and let me know what you think.
>
> On Tue, Nov 26, 2024 at 11:50 AM Dmitri Bourlatchkov
> <dm...@dremio.com.invalid> wrote:
>
> > Hi Mike,
> >
> > I had a brief, but end-to-end look at your latest HK2 branch. Overall, I
> > think it is a good refactoring.
> >
> > I see that it uses standard jakarta annotations in most places, which
> > should work fine with Quarkus too (as Alex already mentioned).
> >
> > I wonder whether implementing the HK2 Factory interface is actually
> > necessary. Would it be able to find `@Produces` annotations
> automatically?
> >
> > Let's proceed with the migration path doc (as discussed in other emails
> in
> > this thread).
> >
> > Cheers,
> > Dmitri.
> >
> >
> > On Tue, Nov 26, 2024 at 12:07 AM Michael Collado <collado.mike@gmail.com
> >
> > wrote:
> >
> > > FYI, I updated the branch to exclude HK2 dependencies from the core
> > module
> > > -
> > >
> > >
> >
> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> > > . The Factory implementations are defined in the polaris-service
> module.
> > >
> > > I also created another branch at
> > >
> > >
> >
> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> > > that merged the PR I have at
> https://github.com/apache/polaris/pull/465
> > > using the CDI annotations rather than the extra factories I had created
> > in
> > > the initial PR. This is still moving us toward supporting different
> > > implementations of the various Polaris*Manager interfaces - still
> relying
> > > on the current concrete PolarisMetaStoreManager, but we can easily
> extend
> > > the HK2 service-locator file to support different manager
> implementations
> > > in future PRs. (I prefer smaller incremental moves before we go with
> > > fully-swappable implementations).
> > >
> > > Mike
> > >
> > > On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <
> collado.mike@gmail.com>
> > > wrote:
> > >
> > > > Hey thanks for calling that out. It’s strictly a convenience change
> and
> > > > actually isn’t necessary. I added that in because, at the end of the
> > day,
> > > > most components need a PolarisMetaStoreManager, not a
> > > > MetaStoreManagerFactory. Registering it as a factory meant that
> > > components
> > > > could declare a dependency on the PolarisMetaStoreManager interface
> and
> > > it
> > > > would use the factory to create the right instance. Because the
> factory
> > > > method is annotated @RealmScope, the instance it creates is specific
> to
> > > the
> > > > realm. I could have done it in the PolarisApplication, as I did the
> > > > PolarisGrantManagerFactory at
> > > >
> > >
> >
> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > > > . I was just lazy when I did that one :) We can keep all the
> > hk2-specific
> > > > code out of the core module.
> > > >
> > > > I do want to point out where the @RealmScope differs from
> > @RequestScope -
> > > > the beans defined in @RequestScope are cleaned up at the end of the
> > > > request. The ones defined in the @RealmScope are reused across
> > requests.
> > > > This makes a difference with classes like the EntityCache that
> depends
> > on
> > > > being able to reuse, e.g., catalogs, but ensure the cache is specific
> > to
> > > > the current realm. We can, of course, pass around a bunch of
> > > > @ApplicationScope factories that return the realm-specific beans, but
> > > that
> > > > means we can't do things like add the PolarisGrantManager interface
> > > without
> > > > also adding a new realm-specific factory, even if the
> > PolarisGrantManager
> > > > implementation is really just the PolarisMetaStoreManager itself. I
> > > think,
> > > > ideally, we avoid declaring dependencies on factories and have
> > components
> > > > declare dependencies on the specific realm-scoped beans they need and
> > let
> > > > the CDI framework work out where the beans come from. This is
> basically
> > > > what I was doing with the MetaStoreManagerFactory interface - as I
> > said,
> > > I
> > > > was just being lazy by extending Factory directly.
> > > >
> > > > Mike
> > > >
> > > >
> > > >
> > > > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > > > <dm...@dremio.com.invalid> wrote:
> > > >
> > > >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
> > > missing
> > > >> the bigger picture.
> > > >>
> > > >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> > > >> Discoverable, but it extends the HK2 Factory class now. So we're
> > > basically
> > > >> trading one framework for another as a code-level runtime
> dependency.
> > I
> > > >> would really like it if we could avoid that.
> > > >>
> > > >> I think one of the key benefits of Quarkus is abstracting from
> > framework
> > > >> code in our class hierarchies. Granted, Quarkus requires certain Web
> > App
> > > >> frameworks, but that's at another level. As for CDI, our code under
> > > >> Quarkus
> > > >> would only have to have certain annotations, without having to
> > > >> extend/implement framework interfaces.
> > > >>
> > > >> Cheers,
> > > >> Dmitri.
> > > >>
> > > >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <
> > collado.mike@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a
> > CDI
> > > >> impl.
> > > >> > I only spent yesterday and today on this, so it's not complete,
> but
> > it
> > > >> is
> > > >> > functional and the tests pass :)
> > > >> >
> > > >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
> > > deleted
> > > >> all
> > > >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml
> config.
> > I
> > > >> > figured out how to use the Dropwizard Yaml to specify the
> > > >> > implementation of, e.g. the Authenticator and the
> > > >> MetaStoreManagerFactory,
> > > >> > but have the instances managed and injectable by HK2. The goal
> there
> > > >> was to
> > > >> > just keep the existing configuration format, but change the impl
> > under
> > > >> the
> > > >> > hood. I'm not married to the idea and I'm interested to see if the
> > > >> > jakarta.enterprise.inject.* annotations/interfaces that are used
> in
> > > the
> > > >> > Quarkus branch can make this simpler. However, I do think it would
> > > >> ideal if
> > > >> > we can get it working with the existing Yaml configuration, at
> least
> > > in
> > > >> the
> > > >> > short term.
> > > >> >
> > > >> > I did add support for a @RealmScope annotation to support
> > restricting
> > > >> items
> > > >> > to a given realm, such as the EntityCache and the
> > PolarisGrantManager.
> > > >> This
> > > >> > allowed me to do things like hide the grant lookups from the
> > > >> EntityCache so
> > > >> > that the Resolver doesn't have to pass around the
> > > ResolvedPolarisEntity,
> > > >> > but instead the grants are found from the cache without making it
> > > overt
> > > >> in
> > > >> > the PolarisAuthorizer API. This was one of my original goals with
> > > >> breaking
> > > >> > up the PolarisMetaStoreManager API into multiple interfaces. Right
> > > now,
> > > >> > everything still ties back to the configured
> PolarisMetaStoreManager
> > > >> > implementation, but eventually we can get to where the
> GrantManager,
> > > >> > CredentialVendor, etc. can all be swapped out for different
> > > >> > implementations.
> > > >> >
> > > >> > Please take a look at the changes at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > >> > and let me know what you think.
> > > >> >
> > > >> > Mike
> > > >> >
> > > >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> > > collado.mike@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > I added a table to the README with the differences that were
> > called
> > > >> out.
> > > >> > I
> > > >> > > added some details that I think are worth understanding better.
> > > E.g.,
> > > >> the
> > > >> > > Json layout we added has specific custom functionality we wanted
> > for
> > > >> > > supporting key/value pairs and the micrometer annotation was
> added
> > > for
> > > >> > some
> > > >> > > custom support we wanted aside from what is supported with the
> > > default
> > > >> > > dropwizard metrics support.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > >> > >
> > > >> > > I think it would be great if others could add to the table with
> > > >> features
> > > >> > > that would be impacted by the migration and to call out the
> level
> > of
> > > >> > effort
> > > >> > > in both dropwizard and quarkus.
> > > >> > >
> > > >> > > Another possible consideration would be upgrading our Dropwizard
> > > >> > > dependency to the latest development version. It may be the case
> > > that
> > > >> > doing
> > > >> > > so would address some of the targeted features with less effort
> in
> > > >> > > migrating.
> > > >> > >
> > > >> > > Mike
> > > >> > >
> > > >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> > > jb@nanthrax.net
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Hi folks,
> > > >> > >>
> > > >> > >> Following the last community sync meeting, we create a branch
> > > >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > >> > >>
> > > >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > >> > >>
> > > >> > >> We still have work to do, but you can already take a look and
> > > >> > >> experiment (in the polaris-service-quarkus module).
> > > >> > >>
> > > >> > >> We added a README.md file to:
> > > >> > >> 1. Highlight the main differences (in terms of code) between
> > > >> > >> Dropwizard and Quarkus
> > > >> > >> 2. To build and run Polaris powered by Quarkus
> > > >> > >> 3. The list of TODO items
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> >
> > > >>
> > >
> >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > >> > >>
> > > >> > >> If anyone wants to contribute on the branch before creating the
> > PR,
> > > >> > >> please let me know, I will add you as contributor on the
> branch.
> > > >> > >>
> > > >> > >> Any comments or questions are welcome !
> > > >> > >>
> > > >> > >> Thanks,
> > > >> > >> Regards
> > > >> > >> JB
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
Sorry, that one is for Web Apps
On Wed, Nov 27, 2024 at 5:07 PM Dmitri Bourlatchkov <
dmitri.bourlatchkov@dremio.com> wrote:
> I see jakarta.ws.rs.Produces in jakarta.ws.rs-api 3.0.0... Does HK2 have a
> lower version of jakarta?
>
> On Wed, Nov 27, 2024 at 4:22 PM Michael Collado <co...@gmail.com>
> wrote:
>
>> Thanks Dmitri. The jakarta.inject api doesn't include the @Produces
>> annotation - is it one that's included in the enterprise package used by
>> Quarkus? I think that would eliminate some of the extra code and
>> complexity
>> in the current branch (e.g., a class can only implement one Factory<T>,
>> but
>> I could use @Produces on multiple methods in the same class).
>>
>> I made some (I think) substantial improvements in the application
>> bootstrapping - the previous work doesn't work for the bootstrap or purge
>> commands. I've posted a PR at https://github.com/apache/polaris/pull/493
>> with the latest. I stripped out the cache stuff, as that's already
>> implemented in the other PR I opened the other day. Take a look at this
>> one
>> and let me know what you think.
>>
>> On Tue, Nov 26, 2024 at 11:50 AM Dmitri Bourlatchkov
>> <dm...@dremio.com.invalid> wrote:
>>
>> > Hi Mike,
>> >
>> > I had a brief, but end-to-end look at your latest HK2 branch. Overall, I
>> > think it is a good refactoring.
>> >
>> > I see that it uses standard jakarta annotations in most places, which
>> > should work fine with Quarkus too (as Alex already mentioned).
>> >
>> > I wonder whether implementing the HK2 Factory interface is actually
>> > necessary. Would it be able to find `@Produces` annotations
>> automatically?
>> >
>> > Let's proceed with the migration path doc (as discussed in other emails
>> in
>> > this thread).
>> >
>> > Cheers,
>> > Dmitri.
>> >
>> >
>> > On Tue, Nov 26, 2024 at 12:07 AM Michael Collado <
>> collado.mike@gmail.com>
>> > wrote:
>> >
>> > > FYI, I updated the branch to exclude HK2 dependencies from the core
>> > module
>> > > -
>> > >
>> > >
>> >
>> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
>> > > . The Factory implementations are defined in the polaris-service
>> module.
>> > >
>> > > I also created another branch at
>> > >
>> > >
>> >
>> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
>> > > that merged the PR I have at
>> https://github.com/apache/polaris/pull/465
>> > > using the CDI annotations rather than the extra factories I had
>> created
>> > in
>> > > the initial PR. This is still moving us toward supporting different
>> > > implementations of the various Polaris*Manager interfaces - still
>> relying
>> > > on the current concrete PolarisMetaStoreManager, but we can easily
>> extend
>> > > the HK2 service-locator file to support different manager
>> implementations
>> > > in future PRs. (I prefer smaller incremental moves before we go with
>> > > fully-swappable implementations).
>> > >
>> > > Mike
>> > >
>> > > On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <
>> collado.mike@gmail.com>
>> > > wrote:
>> > >
>> > > > Hey thanks for calling that out. It’s strictly a convenience change
>> and
>> > > > actually isn’t necessary. I added that in because, at the end of the
>> > day,
>> > > > most components need a PolarisMetaStoreManager, not a
>> > > > MetaStoreManagerFactory. Registering it as a factory meant that
>> > > components
>> > > > could declare a dependency on the PolarisMetaStoreManager interface
>> and
>> > > it
>> > > > would use the factory to create the right instance. Because the
>> factory
>> > > > method is annotated @RealmScope, the instance it creates is
>> specific to
>> > > the
>> > > > realm. I could have done it in the PolarisApplication, as I did the
>> > > > PolarisGrantManagerFactory at
>> > > >
>> > >
>> >
>> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
>> > > > . I was just lazy when I did that one :) We can keep all the
>> > hk2-specific
>> > > > code out of the core module.
>> > > >
>> > > > I do want to point out where the @RealmScope differs from
>> > @RequestScope -
>> > > > the beans defined in @RequestScope are cleaned up at the end of the
>> > > > request. The ones defined in the @RealmScope are reused across
>> > requests.
>> > > > This makes a difference with classes like the EntityCache that
>> depends
>> > on
>> > > > being able to reuse, e.g., catalogs, but ensure the cache is
>> specific
>> > to
>> > > > the current realm. We can, of course, pass around a bunch of
>> > > > @ApplicationScope factories that return the realm-specific beans,
>> but
>> > > that
>> > > > means we can't do things like add the PolarisGrantManager interface
>> > > without
>> > > > also adding a new realm-specific factory, even if the
>> > PolarisGrantManager
>> > > > implementation is really just the PolarisMetaStoreManager itself. I
>> > > think,
>> > > > ideally, we avoid declaring dependencies on factories and have
>> > components
>> > > > declare dependencies on the specific realm-scoped beans they need
>> and
>> > let
>> > > > the CDI framework work out where the beans come from. This is
>> basically
>> > > > what I was doing with the MetaStoreManagerFactory interface - as I
>> > said,
>> > > I
>> > > > was just being lazy by extending Factory directly.
>> > > >
>> > > > Mike
>> > > >
>> > > >
>> > > >
>> > > > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
>> > > > <dm...@dremio.com.invalid> wrote:
>> > > >
>> > > >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
>> > > missing
>> > > >> the bigger picture.
>> > > >>
>> > > >> Still, I see that MetaStoreManagerFactory no longer extends the DW
>> > > >> Discoverable, but it extends the HK2 Factory class now. So we're
>> > > basically
>> > > >> trading one framework for another as a code-level runtime
>> dependency.
>> > I
>> > > >> would really like it if we could avoid that.
>> > > >>
>> > > >> I think one of the key benefits of Quarkus is abstracting from
>> > framework
>> > > >> code in our class hierarchies. Granted, Quarkus requires certain
>> Web
>> > App
>> > > >> frameworks, but that's at another level. As for CDI, our code under
>> > > >> Quarkus
>> > > >> would only have to have certain annotations, without having to
>> > > >> extend/implement framework interfaces.
>> > > >>
>> > > >> Cheers,
>> > > >> Dmitri.
>> > > >>
>> > > >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <
>> > collado.mike@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a
>> > CDI
>> > > >> impl.
>> > > >> > I only spent yesterday and today on this, so it's not complete,
>> but
>> > it
>> > > >> is
>> > > >> > functional and the tests pass :)
>> > > >> >
>> > > >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
>> > > deleted
>> > > >> all
>> > > >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml
>> config.
>> > I
>> > > >> > figured out how to use the Dropwizard Yaml to specify the
>> > > >> > implementation of, e.g. the Authenticator and the
>> > > >> MetaStoreManagerFactory,
>> > > >> > but have the instances managed and injectable by HK2. The goal
>> there
>> > > >> was to
>> > > >> > just keep the existing configuration format, but change the impl
>> > under
>> > > >> the
>> > > >> > hood. I'm not married to the idea and I'm interested to see if
>> the
>> > > >> > jakarta.enterprise.inject.* annotations/interfaces that are used
>> in
>> > > the
>> > > >> > Quarkus branch can make this simpler. However, I do think it
>> would
>> > > >> ideal if
>> > > >> > we can get it working with the existing Yaml configuration, at
>> least
>> > > in
>> > > >> the
>> > > >> > short term.
>> > > >> >
>> > > >> > I did add support for a @RealmScope annotation to support
>> > restricting
>> > > >> items
>> > > >> > to a given realm, such as the EntityCache and the
>> > PolarisGrantManager.
>> > > >> This
>> > > >> > allowed me to do things like hide the grant lookups from the
>> > > >> EntityCache so
>> > > >> > that the Resolver doesn't have to pass around the
>> > > ResolvedPolarisEntity,
>> > > >> > but instead the grants are found from the cache without making it
>> > > overt
>> > > >> in
>> > > >> > the PolarisAuthorizer API. This was one of my original goals with
>> > > >> breaking
>> > > >> > up the PolarisMetaStoreManager API into multiple interfaces.
>> Right
>> > > now,
>> > > >> > everything still ties back to the configured
>> PolarisMetaStoreManager
>> > > >> > implementation, but eventually we can get to where the
>> GrantManager,
>> > > >> > CredentialVendor, etc. can all be swapped out for different
>> > > >> > implementations.
>> > > >> >
>> > > >> > Please take a look at the changes at
>> > > >> >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
>> > > >> > and let me know what you think.
>> > > >> >
>> > > >> > Mike
>> > > >> >
>> > > >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
>> > > collado.mike@gmail.com>
>> > > >> > wrote:
>> > > >> >
>> > > >> > > I added a table to the README with the differences that were
>> > called
>> > > >> out.
>> > > >> > I
>> > > >> > > added some details that I think are worth understanding better.
>> > > E.g.,
>> > > >> the
>> > > >> > > Json layout we added has specific custom functionality we
>> wanted
>> > for
>> > > >> > > supporting key/value pairs and the micrometer annotation was
>> added
>> > > for
>> > > >> > some
>> > > >> > > custom support we wanted aside from what is supported with the
>> > > default
>> > > >> > > dropwizard metrics support.
>> > > >> > >
>> > > >> > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
>> > > >> > >
>> > > >> > > I think it would be great if others could add to the table with
>> > > >> features
>> > > >> > > that would be impacted by the migration and to call out the
>> level
>> > of
>> > > >> > effort
>> > > >> > > in both dropwizard and quarkus.
>> > > >> > >
>> > > >> > > Another possible consideration would be upgrading our
>> Dropwizard
>> > > >> > > dependency to the latest development version. It may be the
>> case
>> > > that
>> > > >> > doing
>> > > >> > > so would address some of the targeted features with less
>> effort in
>> > > >> > > migrating.
>> > > >> > >
>> > > >> > > Mike
>> > > >> > >
>> > > >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
>> > > jb@nanthrax.net
>> > > >> >
>> > > >> > > wrote:
>> > > >> > >
>> > > >> > >> Hi folks,
>> > > >> > >>
>> > > >> > >> Following the last community sync meeting, we create a branch
>> > > >> > >> demonstrating use of Quarkus to powered Apache Polaris:
>> > > >> > >>
>> > > >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
>> > > >> > >>
>> > > >> > >> We still have work to do, but you can already take a look and
>> > > >> > >> experiment (in the polaris-service-quarkus module).
>> > > >> > >>
>> > > >> > >> We added a README.md file to:
>> > > >> > >> 1. Highlight the main differences (in terms of code) between
>> > > >> > >> Dropwizard and Quarkus
>> > > >> > >> 2. To build and run Polaris powered by Quarkus
>> > > >> > >> 3. The list of TODO items
>> > > >> > >>
>> > > >> > >>
>> > > >> > >>
>> > > >> >
>> > > >>
>> > >
>> >
>> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
>> > > >> > >>
>> > > >> > >> If anyone wants to contribute on the branch before creating
>> the
>> > PR,
>> > > >> > >> please let me know, I will add you as contributor on the
>> branch.
>> > > >> > >>
>> > > >> > >> Any comments or questions are welcome !
>> > > >> > >>
>> > > >> > >> Thanks,
>> > > >> > >> Regards
>> > > >> > >> JB
>> > > >> > >>
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
I see jakarta.ws.rs.Produces in jakarta.ws.rs-api 3.0.0... Does HK2 have a
lower version of jakarta?
On Wed, Nov 27, 2024 at 4:22 PM Michael Collado <co...@gmail.com>
wrote:
> Thanks Dmitri. The jakarta.inject api doesn't include the @Produces
> annotation - is it one that's included in the enterprise package used by
> Quarkus? I think that would eliminate some of the extra code and complexity
> in the current branch (e.g., a class can only implement one Factory<T>, but
> I could use @Produces on multiple methods in the same class).
>
> I made some (I think) substantial improvements in the application
> bootstrapping - the previous work doesn't work for the bootstrap or purge
> commands. I've posted a PR at https://github.com/apache/polaris/pull/493
> with the latest. I stripped out the cache stuff, as that's already
> implemented in the other PR I opened the other day. Take a look at this one
> and let me know what you think.
>
> On Tue, Nov 26, 2024 at 11:50 AM Dmitri Bourlatchkov
> <dm...@dremio.com.invalid> wrote:
>
> > Hi Mike,
> >
> > I had a brief, but end-to-end look at your latest HK2 branch. Overall, I
> > think it is a good refactoring.
> >
> > I see that it uses standard jakarta annotations in most places, which
> > should work fine with Quarkus too (as Alex already mentioned).
> >
> > I wonder whether implementing the HK2 Factory interface is actually
> > necessary. Would it be able to find `@Produces` annotations
> automatically?
> >
> > Let's proceed with the migration path doc (as discussed in other emails
> in
> > this thread).
> >
> > Cheers,
> > Dmitri.
> >
> >
> > On Tue, Nov 26, 2024 at 12:07 AM Michael Collado <collado.mike@gmail.com
> >
> > wrote:
> >
> > > FYI, I updated the branch to exclude HK2 dependencies from the core
> > module
> > > -
> > >
> > >
> >
> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> > > . The Factory implementations are defined in the polaris-service
> module.
> > >
> > > I also created another branch at
> > >
> > >
> >
> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> > > that merged the PR I have at
> https://github.com/apache/polaris/pull/465
> > > using the CDI annotations rather than the extra factories I had created
> > in
> > > the initial PR. This is still moving us toward supporting different
> > > implementations of the various Polaris*Manager interfaces - still
> relying
> > > on the current concrete PolarisMetaStoreManager, but we can easily
> extend
> > > the HK2 service-locator file to support different manager
> implementations
> > > in future PRs. (I prefer smaller incremental moves before we go with
> > > fully-swappable implementations).
> > >
> > > Mike
> > >
> > > On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <
> collado.mike@gmail.com>
> > > wrote:
> > >
> > > > Hey thanks for calling that out. It’s strictly a convenience change
> and
> > > > actually isn’t necessary. I added that in because, at the end of the
> > day,
> > > > most components need a PolarisMetaStoreManager, not a
> > > > MetaStoreManagerFactory. Registering it as a factory meant that
> > > components
> > > > could declare a dependency on the PolarisMetaStoreManager interface
> and
> > > it
> > > > would use the factory to create the right instance. Because the
> factory
> > > > method is annotated @RealmScope, the instance it creates is specific
> to
> > > the
> > > > realm. I could have done it in the PolarisApplication, as I did the
> > > > PolarisGrantManagerFactory at
> > > >
> > >
> >
> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > > > . I was just lazy when I did that one :) We can keep all the
> > hk2-specific
> > > > code out of the core module.
> > > >
> > > > I do want to point out where the @RealmScope differs from
> > @RequestScope -
> > > > the beans defined in @RequestScope are cleaned up at the end of the
> > > > request. The ones defined in the @RealmScope are reused across
> > requests.
> > > > This makes a difference with classes like the EntityCache that
> depends
> > on
> > > > being able to reuse, e.g., catalogs, but ensure the cache is specific
> > to
> > > > the current realm. We can, of course, pass around a bunch of
> > > > @ApplicationScope factories that return the realm-specific beans, but
> > > that
> > > > means we can't do things like add the PolarisGrantManager interface
> > > without
> > > > also adding a new realm-specific factory, even if the
> > PolarisGrantManager
> > > > implementation is really just the PolarisMetaStoreManager itself. I
> > > think,
> > > > ideally, we avoid declaring dependencies on factories and have
> > components
> > > > declare dependencies on the specific realm-scoped beans they need and
> > let
> > > > the CDI framework work out where the beans come from. This is
> basically
> > > > what I was doing with the MetaStoreManagerFactory interface - as I
> > said,
> > > I
> > > > was just being lazy by extending Factory directly.
> > > >
> > > > Mike
> > > >
> > > >
> > > >
> > > > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > > > <dm...@dremio.com.invalid> wrote:
> > > >
> > > >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
> > > missing
> > > >> the bigger picture.
> > > >>
> > > >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> > > >> Discoverable, but it extends the HK2 Factory class now. So we're
> > > basically
> > > >> trading one framework for another as a code-level runtime
> dependency.
> > I
> > > >> would really like it if we could avoid that.
> > > >>
> > > >> I think one of the key benefits of Quarkus is abstracting from
> > framework
> > > >> code in our class hierarchies. Granted, Quarkus requires certain Web
> > App
> > > >> frameworks, but that's at another level. As for CDI, our code under
> > > >> Quarkus
> > > >> would only have to have certain annotations, without having to
> > > >> extend/implement framework interfaces.
> > > >>
> > > >> Cheers,
> > > >> Dmitri.
> > > >>
> > > >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <
> > collado.mike@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a
> > CDI
> > > >> impl.
> > > >> > I only spent yesterday and today on this, so it's not complete,
> but
> > it
> > > >> is
> > > >> > functional and the tests pass :)
> > > >> >
> > > >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
> > > deleted
> > > >> all
> > > >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml
> config.
> > I
> > > >> > figured out how to use the Dropwizard Yaml to specify the
> > > >> > implementation of, e.g. the Authenticator and the
> > > >> MetaStoreManagerFactory,
> > > >> > but have the instances managed and injectable by HK2. The goal
> there
> > > >> was to
> > > >> > just keep the existing configuration format, but change the impl
> > under
> > > >> the
> > > >> > hood. I'm not married to the idea and I'm interested to see if the
> > > >> > jakarta.enterprise.inject.* annotations/interfaces that are used
> in
> > > the
> > > >> > Quarkus branch can make this simpler. However, I do think it would
> > > >> ideal if
> > > >> > we can get it working with the existing Yaml configuration, at
> least
> > > in
> > > >> the
> > > >> > short term.
> > > >> >
> > > >> > I did add support for a @RealmScope annotation to support
> > restricting
> > > >> items
> > > >> > to a given realm, such as the EntityCache and the
> > PolarisGrantManager.
> > > >> This
> > > >> > allowed me to do things like hide the grant lookups from the
> > > >> EntityCache so
> > > >> > that the Resolver doesn't have to pass around the
> > > ResolvedPolarisEntity,
> > > >> > but instead the grants are found from the cache without making it
> > > overt
> > > >> in
> > > >> > the PolarisAuthorizer API. This was one of my original goals with
> > > >> breaking
> > > >> > up the PolarisMetaStoreManager API into multiple interfaces. Right
> > > now,
> > > >> > everything still ties back to the configured
> PolarisMetaStoreManager
> > > >> > implementation, but eventually we can get to where the
> GrantManager,
> > > >> > CredentialVendor, etc. can all be swapped out for different
> > > >> > implementations.
> > > >> >
> > > >> > Please take a look at the changes at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > >> > and let me know what you think.
> > > >> >
> > > >> > Mike
> > > >> >
> > > >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> > > collado.mike@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > I added a table to the README with the differences that were
> > called
> > > >> out.
> > > >> > I
> > > >> > > added some details that I think are worth understanding better.
> > > E.g.,
> > > >> the
> > > >> > > Json layout we added has specific custom functionality we wanted
> > for
> > > >> > > supporting key/value pairs and the micrometer annotation was
> added
> > > for
> > > >> > some
> > > >> > > custom support we wanted aside from what is supported with the
> > > default
> > > >> > > dropwizard metrics support.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > >> > >
> > > >> > > I think it would be great if others could add to the table with
> > > >> features
> > > >> > > that would be impacted by the migration and to call out the
> level
> > of
> > > >> > effort
> > > >> > > in both dropwizard and quarkus.
> > > >> > >
> > > >> > > Another possible consideration would be upgrading our Dropwizard
> > > >> > > dependency to the latest development version. It may be the case
> > > that
> > > >> > doing
> > > >> > > so would address some of the targeted features with less effort
> in
> > > >> > > migrating.
> > > >> > >
> > > >> > > Mike
> > > >> > >
> > > >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> > > jb@nanthrax.net
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Hi folks,
> > > >> > >>
> > > >> > >> Following the last community sync meeting, we create a branch
> > > >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > >> > >>
> > > >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > >> > >>
> > > >> > >> We still have work to do, but you can already take a look and
> > > >> > >> experiment (in the polaris-service-quarkus module).
> > > >> > >>
> > > >> > >> We added a README.md file to:
> > > >> > >> 1. Highlight the main differences (in terms of code) between
> > > >> > >> Dropwizard and Quarkus
> > > >> > >> 2. To build and run Polaris powered by Quarkus
> > > >> > >> 3. The list of TODO items
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> >
> > > >>
> > >
> >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > >> > >>
> > > >> > >> If anyone wants to contribute on the branch before creating the
> > PR,
> > > >> > >> please let me know, I will add you as contributor on the
> branch.
> > > >> > >>
> > > >> > >> Any comments or questions are welcome !
> > > >> > >>
> > > >> > >> Thanks,
> > > >> > >> Regards
> > > >> > >> JB
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Michael Collado <co...@gmail.com>.
Thanks Dmitri. The jakarta.inject api doesn't include the @Produces
annotation - is it one that's included in the enterprise package used by
Quarkus? I think that would eliminate some of the extra code and complexity
in the current branch (e.g., a class can only implement one Factory<T>, but
I could use @Produces on multiple methods in the same class).
I made some (I think) substantial improvements in the application
bootstrapping - the previous work doesn't work for the bootstrap or purge
commands. I've posted a PR at https://github.com/apache/polaris/pull/493
with the latest. I stripped out the cache stuff, as that's already
implemented in the other PR I opened the other day. Take a look at this one
and let me know what you think.
On Tue, Nov 26, 2024 at 11:50 AM Dmitri Bourlatchkov
<dm...@dremio.com.invalid> wrote:
> Hi Mike,
>
> I had a brief, but end-to-end look at your latest HK2 branch. Overall, I
> think it is a good refactoring.
>
> I see that it uses standard jakarta annotations in most places, which
> should work fine with Quarkus too (as Alex already mentioned).
>
> I wonder whether implementing the HK2 Factory interface is actually
> necessary. Would it be able to find `@Produces` annotations automatically?
>
> Let's proceed with the migration path doc (as discussed in other emails in
> this thread).
>
> Cheers,
> Dmitri.
>
>
> On Tue, Nov 26, 2024 at 12:07 AM Michael Collado <co...@gmail.com>
> wrote:
>
> > FYI, I updated the branch to exclude HK2 dependencies from the core
> module
> > -
> >
> >
> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> > . The Factory implementations are defined in the polaris-service module.
> >
> > I also created another branch at
> >
> >
> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> > that merged the PR I have at https://github.com/apache/polaris/pull/465
> > using the CDI annotations rather than the extra factories I had created
> in
> > the initial PR. This is still moving us toward supporting different
> > implementations of the various Polaris*Manager interfaces - still relying
> > on the current concrete PolarisMetaStoreManager, but we can easily extend
> > the HK2 service-locator file to support different manager implementations
> > in future PRs. (I prefer smaller incremental moves before we go with
> > fully-swappable implementations).
> >
> > Mike
> >
> > On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> > wrote:
> >
> > > Hey thanks for calling that out. It’s strictly a convenience change and
> > > actually isn’t necessary. I added that in because, at the end of the
> day,
> > > most components need a PolarisMetaStoreManager, not a
> > > MetaStoreManagerFactory. Registering it as a factory meant that
> > components
> > > could declare a dependency on the PolarisMetaStoreManager interface and
> > it
> > > would use the factory to create the right instance. Because the factory
> > > method is annotated @RealmScope, the instance it creates is specific to
> > the
> > > realm. I could have done it in the PolarisApplication, as I did the
> > > PolarisGrantManagerFactory at
> > >
> >
> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > > . I was just lazy when I did that one :) We can keep all the
> hk2-specific
> > > code out of the core module.
> > >
> > > I do want to point out where the @RealmScope differs from
> @RequestScope -
> > > the beans defined in @RequestScope are cleaned up at the end of the
> > > request. The ones defined in the @RealmScope are reused across
> requests.
> > > This makes a difference with classes like the EntityCache that depends
> on
> > > being able to reuse, e.g., catalogs, but ensure the cache is specific
> to
> > > the current realm. We can, of course, pass around a bunch of
> > > @ApplicationScope factories that return the realm-specific beans, but
> > that
> > > means we can't do things like add the PolarisGrantManager interface
> > without
> > > also adding a new realm-specific factory, even if the
> PolarisGrantManager
> > > implementation is really just the PolarisMetaStoreManager itself. I
> > think,
> > > ideally, we avoid declaring dependencies on factories and have
> components
> > > declare dependencies on the specific realm-scoped beans they need and
> let
> > > the CDI framework work out where the beans come from. This is basically
> > > what I was doing with the MetaStoreManagerFactory interface - as I
> said,
> > I
> > > was just being lazy by extending Factory directly.
> > >
> > > Mike
> > >
> > >
> > >
> > > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > > <dm...@dremio.com.invalid> wrote:
> > >
> > >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
> > missing
> > >> the bigger picture.
> > >>
> > >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> > >> Discoverable, but it extends the HK2 Factory class now. So we're
> > basically
> > >> trading one framework for another as a code-level runtime dependency.
> I
> > >> would really like it if we could avoid that.
> > >>
> > >> I think one of the key benefits of Quarkus is abstracting from
> framework
> > >> code in our class hierarchies. Granted, Quarkus requires certain Web
> App
> > >> frameworks, but that's at another level. As for CDI, our code under
> > >> Quarkus
> > >> would only have to have certain annotations, without having to
> > >> extend/implement framework interfaces.
> > >>
> > >> Cheers,
> > >> Dmitri.
> > >>
> > >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <
> collado.mike@gmail.com>
> > >> wrote:
> > >>
> > >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a
> CDI
> > >> impl.
> > >> > I only spent yesterday and today on this, so it's not complete, but
> it
> > >> is
> > >> > functional and the tests pass :)
> > >> >
> > >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
> > deleted
> > >> all
> > >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config.
> I
> > >> > figured out how to use the Dropwizard Yaml to specify the
> > >> > implementation of, e.g. the Authenticator and the
> > >> MetaStoreManagerFactory,
> > >> > but have the instances managed and injectable by HK2. The goal there
> > >> was to
> > >> > just keep the existing configuration format, but change the impl
> under
> > >> the
> > >> > hood. I'm not married to the idea and I'm interested to see if the
> > >> > jakarta.enterprise.inject.* annotations/interfaces that are used in
> > the
> > >> > Quarkus branch can make this simpler. However, I do think it would
> > >> ideal if
> > >> > we can get it working with the existing Yaml configuration, at least
> > in
> > >> the
> > >> > short term.
> > >> >
> > >> > I did add support for a @RealmScope annotation to support
> restricting
> > >> items
> > >> > to a given realm, such as the EntityCache and the
> PolarisGrantManager.
> > >> This
> > >> > allowed me to do things like hide the grant lookups from the
> > >> EntityCache so
> > >> > that the Resolver doesn't have to pass around the
> > ResolvedPolarisEntity,
> > >> > but instead the grants are found from the cache without making it
> > overt
> > >> in
> > >> > the PolarisAuthorizer API. This was one of my original goals with
> > >> breaking
> > >> > up the PolarisMetaStoreManager API into multiple interfaces. Right
> > now,
> > >> > everything still ties back to the configured PolarisMetaStoreManager
> > >> > implementation, but eventually we can get to where the GrantManager,
> > >> > CredentialVendor, etc. can all be swapped out for different
> > >> > implementations.
> > >> >
> > >> > Please take a look at the changes at
> > >> >
> > >> >
> > >>
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > >> > and let me know what you think.
> > >> >
> > >> > Mike
> > >> >
> > >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> > collado.mike@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > I added a table to the README with the differences that were
> called
> > >> out.
> > >> > I
> > >> > > added some details that I think are worth understanding better.
> > E.g.,
> > >> the
> > >> > > Json layout we added has specific custom functionality we wanted
> for
> > >> > > supporting key/value pairs and the micrometer annotation was added
> > for
> > >> > some
> > >> > > custom support we wanted aside from what is supported with the
> > default
> > >> > > dropwizard metrics support.
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > >> > >
> > >> > > I think it would be great if others could add to the table with
> > >> features
> > >> > > that would be impacted by the migration and to call out the level
> of
> > >> > effort
> > >> > > in both dropwizard and quarkus.
> > >> > >
> > >> > > Another possible consideration would be upgrading our Dropwizard
> > >> > > dependency to the latest development version. It may be the case
> > that
> > >> > doing
> > >> > > so would address some of the targeted features with less effort in
> > >> > > migrating.
> > >> > >
> > >> > > Mike
> > >> > >
> > >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> Hi folks,
> > >> > >>
> > >> > >> Following the last community sync meeting, we create a branch
> > >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > >> > >>
> > >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > >> > >>
> > >> > >> We still have work to do, but you can already take a look and
> > >> > >> experiment (in the polaris-service-quarkus module).
> > >> > >>
> > >> > >> We added a README.md file to:
> > >> > >> 1. Highlight the main differences (in terms of code) between
> > >> > >> Dropwizard and Quarkus
> > >> > >> 2. To build and run Polaris powered by Quarkus
> > >> > >> 3. The list of TODO items
> > >> > >>
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > >> > >>
> > >> > >> If anyone wants to contribute on the branch before creating the
> PR,
> > >> > >> please let me know, I will add you as contributor on the branch.
> > >> > >>
> > >> > >> Any comments or questions are welcome !
> > >> > >>
> > >> > >> Thanks,
> > >> > >> Regards
> > >> > >> JB
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
Hi Mike,
I had a brief, but end-to-end look at your latest HK2 branch. Overall, I
think it is a good refactoring.
I see that it uses standard jakarta annotations in most places, which
should work fine with Quarkus too (as Alex already mentioned).
I wonder whether implementing the HK2 Factory interface is actually
necessary. Would it be able to find `@Produces` annotations automatically?
Let's proceed with the migration path doc (as discussed in other emails in
this thread).
Cheers,
Dmitri.
On Tue, Nov 26, 2024 at 12:07 AM Michael Collado <co...@gmail.com>
wrote:
> FYI, I updated the branch to exclude HK2 dependencies from the core module
> -
>
> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> . The Factory implementations are defined in the polaris-service module.
>
> I also created another branch at
>
> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> that merged the PR I have at https://github.com/apache/polaris/pull/465
> using the CDI annotations rather than the extra factories I had created in
> the initial PR. This is still moving us toward supporting different
> implementations of the various Polaris*Manager interfaces - still relying
> on the current concrete PolarisMetaStoreManager, but we can easily extend
> the HK2 service-locator file to support different manager implementations
> in future PRs. (I prefer smaller incremental moves before we go with
> fully-swappable implementations).
>
> Mike
>
> On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> wrote:
>
> > Hey thanks for calling that out. It’s strictly a convenience change and
> > actually isn’t necessary. I added that in because, at the end of the day,
> > most components need a PolarisMetaStoreManager, not a
> > MetaStoreManagerFactory. Registering it as a factory meant that
> components
> > could declare a dependency on the PolarisMetaStoreManager interface and
> it
> > would use the factory to create the right instance. Because the factory
> > method is annotated @RealmScope, the instance it creates is specific to
> the
> > realm. I could have done it in the PolarisApplication, as I did the
> > PolarisGrantManagerFactory at
> >
> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > . I was just lazy when I did that one :) We can keep all the hk2-specific
> > code out of the core module.
> >
> > I do want to point out where the @RealmScope differs from @RequestScope -
> > the beans defined in @RequestScope are cleaned up at the end of the
> > request. The ones defined in the @RealmScope are reused across requests.
> > This makes a difference with classes like the EntityCache that depends on
> > being able to reuse, e.g., catalogs, but ensure the cache is specific to
> > the current realm. We can, of course, pass around a bunch of
> > @ApplicationScope factories that return the realm-specific beans, but
> that
> > means we can't do things like add the PolarisGrantManager interface
> without
> > also adding a new realm-specific factory, even if the PolarisGrantManager
> > implementation is really just the PolarisMetaStoreManager itself. I
> think,
> > ideally, we avoid declaring dependencies on factories and have components
> > declare dependencies on the specific realm-scoped beans they need and let
> > the CDI framework work out where the beans come from. This is basically
> > what I was doing with the MetaStoreManagerFactory interface - as I said,
> I
> > was just being lazy by extending Factory directly.
> >
> > Mike
> >
> >
> >
> > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > <dm...@dremio.com.invalid> wrote:
> >
> >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
> missing
> >> the bigger picture.
> >>
> >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> >> Discoverable, but it extends the HK2 Factory class now. So we're
> basically
> >> trading one framework for another as a code-level runtime dependency. I
> >> would really like it if we could avoid that.
> >>
> >> I think one of the key benefits of Quarkus is abstracting from framework
> >> code in our class hierarchies. Granted, Quarkus requires certain Web App
> >> frameworks, but that's at another level. As for CDI, our code under
> >> Quarkus
> >> would only have to have certain annotations, without having to
> >> extend/implement framework interfaces.
> >>
> >> Cheers,
> >> Dmitri.
> >>
> >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> >> wrote:
> >>
> >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> >> impl.
> >> > I only spent yesterday and today on this, so it's not complete, but it
> >> is
> >> > functional and the tests pass :)
> >> >
> >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
> deleted
> >> all
> >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> >> > figured out how to use the Dropwizard Yaml to specify the
> >> > implementation of, e.g. the Authenticator and the
> >> MetaStoreManagerFactory,
> >> > but have the instances managed and injectable by HK2. The goal there
> >> was to
> >> > just keep the existing configuration format, but change the impl under
> >> the
> >> > hood. I'm not married to the idea and I'm interested to see if the
> >> > jakarta.enterprise.inject.* annotations/interfaces that are used in
> the
> >> > Quarkus branch can make this simpler. However, I do think it would
> >> ideal if
> >> > we can get it working with the existing Yaml configuration, at least
> in
> >> the
> >> > short term.
> >> >
> >> > I did add support for a @RealmScope annotation to support restricting
> >> items
> >> > to a given realm, such as the EntityCache and the PolarisGrantManager.
> >> This
> >> > allowed me to do things like hide the grant lookups from the
> >> EntityCache so
> >> > that the Resolver doesn't have to pass around the
> ResolvedPolarisEntity,
> >> > but instead the grants are found from the cache without making it
> overt
> >> in
> >> > the PolarisAuthorizer API. This was one of my original goals with
> >> breaking
> >> > up the PolarisMetaStoreManager API into multiple interfaces. Right
> now,
> >> > everything still ties back to the configured PolarisMetaStoreManager
> >> > implementation, but eventually we can get to where the GrantManager,
> >> > CredentialVendor, etc. can all be swapped out for different
> >> > implementations.
> >> >
> >> > Please take a look at the changes at
> >> >
> >> >
> >>
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> >> > and let me know what you think.
> >> >
> >> > Mike
> >> >
> >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> collado.mike@gmail.com>
> >> > wrote:
> >> >
> >> > > I added a table to the README with the differences that were called
> >> out.
> >> > I
> >> > > added some details that I think are worth understanding better.
> E.g.,
> >> the
> >> > > Json layout we added has specific custom functionality we wanted for
> >> > > supporting key/value pairs and the micrometer annotation was added
> for
> >> > some
> >> > > custom support we wanted aside from what is supported with the
> default
> >> > > dropwizard metrics support.
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> >> > >
> >> > > I think it would be great if others could add to the table with
> >> features
> >> > > that would be impacted by the migration and to call out the level of
> >> > effort
> >> > > in both dropwizard and quarkus.
> >> > >
> >> > > Another possible consideration would be upgrading our Dropwizard
> >> > > dependency to the latest development version. It may be the case
> that
> >> > doing
> >> > > so would address some of the targeted features with less effort in
> >> > > migrating.
> >> > >
> >> > > Mike
> >> > >
> >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> jb@nanthrax.net
> >> >
> >> > > wrote:
> >> > >
> >> > >> Hi folks,
> >> > >>
> >> > >> Following the last community sync meeting, we create a branch
> >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> >> > >>
> >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> >> > >>
> >> > >> We still have work to do, but you can already take a look and
> >> > >> experiment (in the polaris-service-quarkus module).
> >> > >>
> >> > >> We added a README.md file to:
> >> > >> 1. Highlight the main differences (in terms of code) between
> >> > >> Dropwizard and Quarkus
> >> > >> 2. To build and run Polaris powered by Quarkus
> >> > >> 3. The list of TODO items
> >> > >>
> >> > >>
> >> > >>
> >> >
> >>
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> >> > >>
> >> > >> If anyone wants to contribute on the branch before creating the PR,
> >> > >> please let me know, I will add you as contributor on the branch.
> >> > >>
> >> > >> Any comments or questions are welcome !
> >> > >>
> >> > >> Thanks,
> >> > >> Regards
> >> > >> JB
> >> > >>
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Michael Collado <co...@gmail.com>.
This sounds great. I think we can agree on some strategies so that work can
keep moving forward even as we're settling on the framework question:
1. Let's continue to rely on the interfaces we already have defined - e.g.,
the PolarisConfigurationStore is going to continue to be the mechanism for
querying service/entity configuration even if we have a framework-specific
underlying method for loading the configuration.
2. The core code is going to need to be framework-agnostic, so we can
continue to define core-types and core-behavior without assuming the source
of construction of those types (e.g., I think we can move forward with
federated-identities without implementing any OIDC-specific implementation
code).
3. Let's start incorporating the CDI annotations in a way that makes
migrating to another framework seamless. This makes the code more usable
now and also helps makes any migration easier
For the first two points, I think it'll be important for us to be clear on
what things are Polaris-specific rather than generic and possibly supplied
by framework plugins. We should be clear about the behavior of
Polaris-specific entities/operations (I'll update my federated-identities
doc to reflect this). But if a change or introduction of a core type is
necessary only because of a lack of framework support, it's a good thing to
call out in the PR and to add to the list of pros/cons we have for the
Quarkus evaluation.
Do these points sound reasonable?
Mike
On Tue, Nov 26, 2024 at 1:46 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Mike
>
> Fully agree, I just wanted to remind that CDI is a good think, but
> considering ecosystem/plugins is also key (and indeed decoupled from
> CDI).
>
> I think we already have a kind of consensus about CDI, and agree to
> move forward here with improvement of the current codebase.
>
> The reason why I mention runtime framework ecosystem/plugins is
> because we already have some PRs/discussions related to that (and
> somehow overlapping with runtime framework support): dynamic config,
> metadata storage, OIDC, ...
> By leveraging runtime framework capabilities, we will focus our effort
> on the Polaris new/core features.
>
> +1 about populating the table and migration path.
>
> I will rebase the Quarkus PR and share a document with the tables you
> created (let's use the Polaris proposal process ;)).
>
> Thanks !
> Regards
> JB
>
> On Tue, Nov 26, 2024 at 8:24 AM Michael Collado <co...@gmail.com>
> wrote:
> >
> > I’m happy to consider the rest of the ecosystem plugins, though I would
> > like to decouple that from the CDI stuff. As I’ve mentioned before, I’m
> all
> > for improving the customizability of the codebase and I love the support
> > for more flexible implementations of the various components. That’s
> > something I think we can agree on and move forward with in a way that’s
> > compatible with any future framework choice.
> >
> > I still think the framework migration is a sizable task and I think the
> > pros and cons of moving forward with such a migration should be written
> > down and considered carefully. I did start that table a while back with
> > some of the details. Maybe we can make some progress on filling it out
> with
> > more details? I’m really interested in knowing specifics about which
> > framework plugins will work, how much unnecessary code we can get rid of
> > and what new capabilities we’d be able to take advantage of. I know there
> > are a lot of vague promises that sound enticing, but I’ve also been
> bitten
> > by those in the past, so I’m a bit more wary than I once was ;)
> >
> > Some of the prospects, especially OIDC support, are super interesting to
> > me, but I just don’t know the full set of details well enough to weigh
> the
> > pros and cons.
> >
> > Mike
> >
> > On Mon, Nov 25, 2024 at 10:41 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi Michael
> > >
> > > Thanks for the update. It's great !
> > > Using CDI annotation is way neater than yaml bean verbose definition
> > > :) We can always improve step by step (about the
> > > PolarisMetaStoreManager and the CallContext).
> > >
> > > NB: the Quarkus PR is also there to have side/side comparison.
> > >
> > > My comment is that we should also consider the framework ecosystem: I
> > > see new Polaris PRs (like the one about dynamic config) that
> > > reimplement from scratch what already provided by Quarkus (thanks to
> > > the extensions). I think it's urgent to have a consensus here to know
> > > in which direction we are going.
> > >
> > > Let's chat together about that.
> > >
> > > Thanks !
> > > Regards
> > > JB
> > >
> > > On Tue, Nov 26, 2024 at 6:06 AM Michael Collado <
> collado.mike@gmail.com>
> > > wrote:
> > > >
> > > > FYI, I updated the branch to exclude HK2 dependencies from the core
> > > module
> > > > -
> > > >
> > >
> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> > > > . The Factory implementations are defined in the polaris-service
> module.
> > > >
> > > > I also created another branch at
> > > >
> > >
> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> > > > that merged the PR I have at
> https://github.com/apache/polaris/pull/465
> > > > using the CDI annotations rather than the extra factories I had
> created
> > > in
> > > > the initial PR. This is still moving us toward supporting different
> > > > implementations of the various Polaris*Manager interfaces - still
> relying
> > > > on the current concrete PolarisMetaStoreManager, but we can easily
> extend
> > > > the HK2 service-locator file to support different manager
> implementations
> > > > in future PRs. (I prefer smaller incremental moves before we go with
> > > > fully-swappable implementations).
> > > >
> > > > Mike
> > > >
> > > > On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <
> collado.mike@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey thanks for calling that out. It’s strictly a convenience
> change and
> > > > > actually isn’t necessary. I added that in because, at the end of
> the
> > > day,
> > > > > most components need a PolarisMetaStoreManager, not a
> > > > > MetaStoreManagerFactory. Registering it as a factory meant that
> > > components
> > > > > could declare a dependency on the PolarisMetaStoreManager interface
> > > and it
> > > > > would use the factory to create the right instance. Because the
> factory
> > > > > method is annotated @RealmScope, the instance it creates is
> specific
> > > to the
> > > > > realm. I could have done it in the PolarisApplication, as I did the
> > > > > PolarisGrantManagerFactory at
> > > > >
> > >
> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > > > > . I was just lazy when I did that one :) We can keep all the
> > > hk2-specific
> > > > > code out of the core module.
> > > > >
> > > > > I do want to point out where the @RealmScope differs from
> > > @RequestScope -
> > > > > the beans defined in @RequestScope are cleaned up at the end of the
> > > > > request. The ones defined in the @RealmScope are reused across
> > > requests.
> > > > > This makes a difference with classes like the EntityCache that
> depends
> > > on
> > > > > being able to reuse, e.g., catalogs, but ensure the cache is
> specific
> > > to
> > > > > the current realm. We can, of course, pass around a bunch of
> > > > > @ApplicationScope factories that return the realm-specific beans,
> but
> > > that
> > > > > means we can't do things like add the PolarisGrantManager interface
> > > without
> > > > > also adding a new realm-specific factory, even if the
> > > PolarisGrantManager
> > > > > implementation is really just the PolarisMetaStoreManager itself. I
> > > think,
> > > > > ideally, we avoid declaring dependencies on factories and have
> > > components
> > > > > declare dependencies on the specific realm-scoped beans they need
> and
> > > let
> > > > > the CDI framework work out where the beans come from. This is
> basically
> > > > > what I was doing with the MetaStoreManagerFactory interface - as I
> > > said, I
> > > > > was just being lazy by extending Factory directly.
> > > > >
> > > > > Mike
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > > > > <dm...@dremio.com.invalid> wrote:
> > > > >
> > > > >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
> > > missing
> > > > >> the bigger picture.
> > > > >>
> > > > >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> > > > >> Discoverable, but it extends the HK2 Factory class now. So we're
> > > basically
> > > > >> trading one framework for another as a code-level runtime
> dependency.
> > > I
> > > > >> would really like it if we could avoid that.
> > > > >>
> > > > >> I think one of the key benefits of Quarkus is abstracting from
> > > framework
> > > > >> code in our class hierarchies. Granted, Quarkus requires certain
> Web
> > > App
> > > > >> frameworks, but that's at another level. As for CDI, our code
> under
> > > > >> Quarkus
> > > > >> would only have to have certain annotations, without having to
> > > > >> extend/implement framework interfaces.
> > > > >>
> > > > >> Cheers,
> > > > >> Dmitri.
> > > > >>
> > > > >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <
> > > collado.mike@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > FYI, I took a stab at seeing how Polaris would work with HK2 as
> a
> > > CDI
> > > > >> impl.
> > > > >> > I only spent yesterday and today on this, so it's not complete,
> but
> > > it
> > > > >> is
> > > > >> > functional and the tests pass :)
> > > > >> >
> > > > >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
> > > deleted
> > > > >> all
> > > > >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml
> config.
> > > I
> > > > >> > figured out how to use the Dropwizard Yaml to specify the
> > > > >> > implementation of, e.g. the Authenticator and the
> > > > >> MetaStoreManagerFactory,
> > > > >> > but have the instances managed and injectable by HK2. The goal
> there
> > > > >> was to
> > > > >> > just keep the existing configuration format, but change the impl
> > > under
> > > > >> the
> > > > >> > hood. I'm not married to the idea and I'm interested to see if
> the
> > > > >> > jakarta.enterprise.inject.* annotations/interfaces that are
> used in
> > > the
> > > > >> > Quarkus branch can make this simpler. However, I do think it
> would
> > > > >> ideal if
> > > > >> > we can get it working with the existing Yaml configuration, at
> > > least in
> > > > >> the
> > > > >> > short term.
> > > > >> >
> > > > >> > I did add support for a @RealmScope annotation to support
> > > restricting
> > > > >> items
> > > > >> > to a given realm, such as the EntityCache and the
> > > PolarisGrantManager.
> > > > >> This
> > > > >> > allowed me to do things like hide the grant lookups from the
> > > > >> EntityCache so
> > > > >> > that the Resolver doesn't have to pass around the
> > > ResolvedPolarisEntity,
> > > > >> > but instead the grants are found from the cache without making
> it
> > > overt
> > > > >> in
> > > > >> > the PolarisAuthorizer API. This was one of my original goals
> with
> > > > >> breaking
> > > > >> > up the PolarisMetaStoreManager API into multiple interfaces.
> Right
> > > now,
> > > > >> > everything still ties back to the configured
> PolarisMetaStoreManager
> > > > >> > implementation, but eventually we can get to where the
> GrantManager,
> > > > >> > CredentialVendor, etc. can all be swapped out for different
> > > > >> > implementations.
> > > > >> >
> > > > >> > Please take a look at the changes at
> > > > >> >
> > > > >> >
> > > > >>
> > >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > > >> > and let me know what you think.
> > > > >> >
> > > > >> > Mike
> > > > >> >
> > > > >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> > > collado.mike@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > I added a table to the README with the differences that were
> > > called
> > > > >> out.
> > > > >> > I
> > > > >> > > added some details that I think are worth understanding
> better.
> > > E.g.,
> > > > >> the
> > > > >> > > Json layout we added has specific custom functionality we
> wanted
> > > for
> > > > >> > > supporting key/value pairs and the micrometer annotation was
> > > added for
> > > > >> > some
> > > > >> > > custom support we wanted aside from what is supported with the
> > > default
> > > > >> > > dropwizard metrics support.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > > >> > >
> > > > >> > > I think it would be great if others could add to the table
> with
> > > > >> features
> > > > >> > > that would be impacted by the migration and to call out the
> level
> > > of
> > > > >> > effort
> > > > >> > > in both dropwizard and quarkus.
> > > > >> > >
> > > > >> > > Another possible consideration would be upgrading our
> Dropwizard
> > > > >> > > dependency to the latest development version. It may be the
> case
> > > that
> > > > >> > doing
> > > > >> > > so would address some of the targeted features with less
> effort in
> > > > >> > > migrating.
> > > > >> > >
> > > > >> > > Mike
> > > > >> > >
> > > > >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> > > jb@nanthrax.net
> > > > >> >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> Hi folks,
> > > > >> > >>
> > > > >> > >> Following the last community sync meeting, we create a branch
> > > > >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > > >> > >>
> > > > >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > > >> > >>
> > > > >> > >> We still have work to do, but you can already take a look and
> > > > >> > >> experiment (in the polaris-service-quarkus module).
> > > > >> > >>
> > > > >> > >> We added a README.md file to:
> > > > >> > >> 1. Highlight the main differences (in terms of code) between
> > > > >> > >> Dropwizard and Quarkus
> > > > >> > >> 2. To build and run Polaris powered by Quarkus
> > > > >> > >> 3. The list of TODO items
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> >
> > > > >>
> > >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > > >> > >>
> > > > >> > >> If anyone wants to contribute on the branch before creating
> the
> > > PR,
> > > > >> > >> please let me know, I will add you as contributor on the
> branch.
> > > > >> > >>
> > > > >> > >> Any comments or questions are welcome !
> > > > >> > >>
> > > > >> > >> Thanks,
> > > > >> > >> Regards
> > > > >> > >> JB
> > > > >> > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Mike
Fully agree, I just wanted to remind that CDI is a good think, but
considering ecosystem/plugins is also key (and indeed decoupled from
CDI).
I think we already have a kind of consensus about CDI, and agree to
move forward here with improvement of the current codebase.
The reason why I mention runtime framework ecosystem/plugins is
because we already have some PRs/discussions related to that (and
somehow overlapping with runtime framework support): dynamic config,
metadata storage, OIDC, ...
By leveraging runtime framework capabilities, we will focus our effort
on the Polaris new/core features.
+1 about populating the table and migration path.
I will rebase the Quarkus PR and share a document with the tables you
created (let's use the Polaris proposal process ;)).
Thanks !
Regards
JB
On Tue, Nov 26, 2024 at 8:24 AM Michael Collado <co...@gmail.com> wrote:
>
> I’m happy to consider the rest of the ecosystem plugins, though I would
> like to decouple that from the CDI stuff. As I’ve mentioned before, I’m all
> for improving the customizability of the codebase and I love the support
> for more flexible implementations of the various components. That’s
> something I think we can agree on and move forward with in a way that’s
> compatible with any future framework choice.
>
> I still think the framework migration is a sizable task and I think the
> pros and cons of moving forward with such a migration should be written
> down and considered carefully. I did start that table a while back with
> some of the details. Maybe we can make some progress on filling it out with
> more details? I’m really interested in knowing specifics about which
> framework plugins will work, how much unnecessary code we can get rid of
> and what new capabilities we’d be able to take advantage of. I know there
> are a lot of vague promises that sound enticing, but I’ve also been bitten
> by those in the past, so I’m a bit more wary than I once was ;)
>
> Some of the prospects, especially OIDC support, are super interesting to
> me, but I just don’t know the full set of details well enough to weigh the
> pros and cons.
>
> Mike
>
> On Mon, Nov 25, 2024 at 10:41 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi Michael
> >
> > Thanks for the update. It's great !
> > Using CDI annotation is way neater than yaml bean verbose definition
> > :) We can always improve step by step (about the
> > PolarisMetaStoreManager and the CallContext).
> >
> > NB: the Quarkus PR is also there to have side/side comparison.
> >
> > My comment is that we should also consider the framework ecosystem: I
> > see new Polaris PRs (like the one about dynamic config) that
> > reimplement from scratch what already provided by Quarkus (thanks to
> > the extensions). I think it's urgent to have a consensus here to know
> > in which direction we are going.
> >
> > Let's chat together about that.
> >
> > Thanks !
> > Regards
> > JB
> >
> > On Tue, Nov 26, 2024 at 6:06 AM Michael Collado <co...@gmail.com>
> > wrote:
> > >
> > > FYI, I updated the branch to exclude HK2 dependencies from the core
> > module
> > > -
> > >
> > https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> > > . The Factory implementations are defined in the polaris-service module.
> > >
> > > I also created another branch at
> > >
> > https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> > > that merged the PR I have at https://github.com/apache/polaris/pull/465
> > > using the CDI annotations rather than the extra factories I had created
> > in
> > > the initial PR. This is still moving us toward supporting different
> > > implementations of the various Polaris*Manager interfaces - still relying
> > > on the current concrete PolarisMetaStoreManager, but we can easily extend
> > > the HK2 service-locator file to support different manager implementations
> > > in future PRs. (I prefer smaller incremental moves before we go with
> > > fully-swappable implementations).
> > >
> > > Mike
> > >
> > > On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> > > wrote:
> > >
> > > > Hey thanks for calling that out. It’s strictly a convenience change and
> > > > actually isn’t necessary. I added that in because, at the end of the
> > day,
> > > > most components need a PolarisMetaStoreManager, not a
> > > > MetaStoreManagerFactory. Registering it as a factory meant that
> > components
> > > > could declare a dependency on the PolarisMetaStoreManager interface
> > and it
> > > > would use the factory to create the right instance. Because the factory
> > > > method is annotated @RealmScope, the instance it creates is specific
> > to the
> > > > realm. I could have done it in the PolarisApplication, as I did the
> > > > PolarisGrantManagerFactory at
> > > >
> > https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > > > . I was just lazy when I did that one :) We can keep all the
> > hk2-specific
> > > > code out of the core module.
> > > >
> > > > I do want to point out where the @RealmScope differs from
> > @RequestScope -
> > > > the beans defined in @RequestScope are cleaned up at the end of the
> > > > request. The ones defined in the @RealmScope are reused across
> > requests.
> > > > This makes a difference with classes like the EntityCache that depends
> > on
> > > > being able to reuse, e.g., catalogs, but ensure the cache is specific
> > to
> > > > the current realm. We can, of course, pass around a bunch of
> > > > @ApplicationScope factories that return the realm-specific beans, but
> > that
> > > > means we can't do things like add the PolarisGrantManager interface
> > without
> > > > also adding a new realm-specific factory, even if the
> > PolarisGrantManager
> > > > implementation is really just the PolarisMetaStoreManager itself. I
> > think,
> > > > ideally, we avoid declaring dependencies on factories and have
> > components
> > > > declare dependencies on the specific realm-scoped beans they need and
> > let
> > > > the CDI framework work out where the beans come from. This is basically
> > > > what I was doing with the MetaStoreManagerFactory interface - as I
> > said, I
> > > > was just being lazy by extending Factory directly.
> > > >
> > > > Mike
> > > >
> > > >
> > > >
> > > > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > > > <dm...@dremio.com.invalid> wrote:
> > > >
> > > >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
> > missing
> > > >> the bigger picture.
> > > >>
> > > >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> > > >> Discoverable, but it extends the HK2 Factory class now. So we're
> > basically
> > > >> trading one framework for another as a code-level runtime dependency.
> > I
> > > >> would really like it if we could avoid that.
> > > >>
> > > >> I think one of the key benefits of Quarkus is abstracting from
> > framework
> > > >> code in our class hierarchies. Granted, Quarkus requires certain Web
> > App
> > > >> frameworks, but that's at another level. As for CDI, our code under
> > > >> Quarkus
> > > >> would only have to have certain annotations, without having to
> > > >> extend/implement framework interfaces.
> > > >>
> > > >> Cheers,
> > > >> Dmitri.
> > > >>
> > > >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <
> > collado.mike@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a
> > CDI
> > > >> impl.
> > > >> > I only spent yesterday and today on this, so it's not complete, but
> > it
> > > >> is
> > > >> > functional and the tests pass :)
> > > >> >
> > > >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
> > deleted
> > > >> all
> > > >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config.
> > I
> > > >> > figured out how to use the Dropwizard Yaml to specify the
> > > >> > implementation of, e.g. the Authenticator and the
> > > >> MetaStoreManagerFactory,
> > > >> > but have the instances managed and injectable by HK2. The goal there
> > > >> was to
> > > >> > just keep the existing configuration format, but change the impl
> > under
> > > >> the
> > > >> > hood. I'm not married to the idea and I'm interested to see if the
> > > >> > jakarta.enterprise.inject.* annotations/interfaces that are used in
> > the
> > > >> > Quarkus branch can make this simpler. However, I do think it would
> > > >> ideal if
> > > >> > we can get it working with the existing Yaml configuration, at
> > least in
> > > >> the
> > > >> > short term.
> > > >> >
> > > >> > I did add support for a @RealmScope annotation to support
> > restricting
> > > >> items
> > > >> > to a given realm, such as the EntityCache and the
> > PolarisGrantManager.
> > > >> This
> > > >> > allowed me to do things like hide the grant lookups from the
> > > >> EntityCache so
> > > >> > that the Resolver doesn't have to pass around the
> > ResolvedPolarisEntity,
> > > >> > but instead the grants are found from the cache without making it
> > overt
> > > >> in
> > > >> > the PolarisAuthorizer API. This was one of my original goals with
> > > >> breaking
> > > >> > up the PolarisMetaStoreManager API into multiple interfaces. Right
> > now,
> > > >> > everything still ties back to the configured PolarisMetaStoreManager
> > > >> > implementation, but eventually we can get to where the GrantManager,
> > > >> > CredentialVendor, etc. can all be swapped out for different
> > > >> > implementations.
> > > >> >
> > > >> > Please take a look at the changes at
> > > >> >
> > > >> >
> > > >>
> > https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > >> > and let me know what you think.
> > > >> >
> > > >> > Mike
> > > >> >
> > > >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> > collado.mike@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > I added a table to the README with the differences that were
> > called
> > > >> out.
> > > >> > I
> > > >> > > added some details that I think are worth understanding better.
> > E.g.,
> > > >> the
> > > >> > > Json layout we added has specific custom functionality we wanted
> > for
> > > >> > > supporting key/value pairs and the micrometer annotation was
> > added for
> > > >> > some
> > > >> > > custom support we wanted aside from what is supported with the
> > default
> > > >> > > dropwizard metrics support.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > >> > >
> > > >> > > I think it would be great if others could add to the table with
> > > >> features
> > > >> > > that would be impacted by the migration and to call out the level
> > of
> > > >> > effort
> > > >> > > in both dropwizard and quarkus.
> > > >> > >
> > > >> > > Another possible consideration would be upgrading our Dropwizard
> > > >> > > dependency to the latest development version. It may be the case
> > that
> > > >> > doing
> > > >> > > so would address some of the targeted features with less effort in
> > > >> > > migrating.
> > > >> > >
> > > >> > > Mike
> > > >> > >
> > > >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Hi folks,
> > > >> > >>
> > > >> > >> Following the last community sync meeting, we create a branch
> > > >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > >> > >>
> > > >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > >> > >>
> > > >> > >> We still have work to do, but you can already take a look and
> > > >> > >> experiment (in the polaris-service-quarkus module).
> > > >> > >>
> > > >> > >> We added a README.md file to:
> > > >> > >> 1. Highlight the main differences (in terms of code) between
> > > >> > >> Dropwizard and Quarkus
> > > >> > >> 2. To build and run Polaris powered by Quarkus
> > > >> > >> 3. The list of TODO items
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> >
> > > >>
> > https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > >> > >>
> > > >> > >> If anyone wants to contribute on the branch before creating the
> > PR,
> > > >> > >> please let me know, I will add you as contributor on the branch.
> > > >> > >>
> > > >> > >> Any comments or questions are welcome !
> > > >> > >>
> > > >> > >> Thanks,
> > > >> > >> Regards
> > > >> > >> JB
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >
> >
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Michael Collado <co...@gmail.com>.
I’m happy to consider the rest of the ecosystem plugins, though I would
like to decouple that from the CDI stuff. As I’ve mentioned before, I’m all
for improving the customizability of the codebase and I love the support
for more flexible implementations of the various components. That’s
something I think we can agree on and move forward with in a way that’s
compatible with any future framework choice.
I still think the framework migration is a sizable task and I think the
pros and cons of moving forward with such a migration should be written
down and considered carefully. I did start that table a while back with
some of the details. Maybe we can make some progress on filling it out with
more details? I’m really interested in knowing specifics about which
framework plugins will work, how much unnecessary code we can get rid of
and what new capabilities we’d be able to take advantage of. I know there
are a lot of vague promises that sound enticing, but I’ve also been bitten
by those in the past, so I’m a bit more wary than I once was ;)
Some of the prospects, especially OIDC support, are super interesting to
me, but I just don’t know the full set of details well enough to weigh the
pros and cons.
Mike
On Mon, Nov 25, 2024 at 10:41 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Michael
>
> Thanks for the update. It's great !
> Using CDI annotation is way neater than yaml bean verbose definition
> :) We can always improve step by step (about the
> PolarisMetaStoreManager and the CallContext).
>
> NB: the Quarkus PR is also there to have side/side comparison.
>
> My comment is that we should also consider the framework ecosystem: I
> see new Polaris PRs (like the one about dynamic config) that
> reimplement from scratch what already provided by Quarkus (thanks to
> the extensions). I think it's urgent to have a consensus here to know
> in which direction we are going.
>
> Let's chat together about that.
>
> Thanks !
> Regards
> JB
>
> On Tue, Nov 26, 2024 at 6:06 AM Michael Collado <co...@gmail.com>
> wrote:
> >
> > FYI, I updated the branch to exclude HK2 dependencies from the core
> module
> > -
> >
> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> > . The Factory implementations are defined in the polaris-service module.
> >
> > I also created another branch at
> >
> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> > that merged the PR I have at https://github.com/apache/polaris/pull/465
> > using the CDI annotations rather than the extra factories I had created
> in
> > the initial PR. This is still moving us toward supporting different
> > implementations of the various Polaris*Manager interfaces - still relying
> > on the current concrete PolarisMetaStoreManager, but we can easily extend
> > the HK2 service-locator file to support different manager implementations
> > in future PRs. (I prefer smaller incremental moves before we go with
> > fully-swappable implementations).
> >
> > Mike
> >
> > On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> > wrote:
> >
> > > Hey thanks for calling that out. It’s strictly a convenience change and
> > > actually isn’t necessary. I added that in because, at the end of the
> day,
> > > most components need a PolarisMetaStoreManager, not a
> > > MetaStoreManagerFactory. Registering it as a factory meant that
> components
> > > could declare a dependency on the PolarisMetaStoreManager interface
> and it
> > > would use the factory to create the right instance. Because the factory
> > > method is annotated @RealmScope, the instance it creates is specific
> to the
> > > realm. I could have done it in the PolarisApplication, as I did the
> > > PolarisGrantManagerFactory at
> > >
> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > > . I was just lazy when I did that one :) We can keep all the
> hk2-specific
> > > code out of the core module.
> > >
> > > I do want to point out where the @RealmScope differs from
> @RequestScope -
> > > the beans defined in @RequestScope are cleaned up at the end of the
> > > request. The ones defined in the @RealmScope are reused across
> requests.
> > > This makes a difference with classes like the EntityCache that depends
> on
> > > being able to reuse, e.g., catalogs, but ensure the cache is specific
> to
> > > the current realm. We can, of course, pass around a bunch of
> > > @ApplicationScope factories that return the realm-specific beans, but
> that
> > > means we can't do things like add the PolarisGrantManager interface
> without
> > > also adding a new realm-specific factory, even if the
> PolarisGrantManager
> > > implementation is really just the PolarisMetaStoreManager itself. I
> think,
> > > ideally, we avoid declaring dependencies on factories and have
> components
> > > declare dependencies on the specific realm-scoped beans they need and
> let
> > > the CDI framework work out where the beans come from. This is basically
> > > what I was doing with the MetaStoreManagerFactory interface - as I
> said, I
> > > was just being lazy by extending Factory directly.
> > >
> > > Mike
> > >
> > >
> > >
> > > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > > <dm...@dremio.com.invalid> wrote:
> > >
> > >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm
> missing
> > >> the bigger picture.
> > >>
> > >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> > >> Discoverable, but it extends the HK2 Factory class now. So we're
> basically
> > >> trading one framework for another as a code-level runtime dependency.
> I
> > >> would really like it if we could avoid that.
> > >>
> > >> I think one of the key benefits of Quarkus is abstracting from
> framework
> > >> code in our class hierarchies. Granted, Quarkus requires certain Web
> App
> > >> frameworks, but that's at another level. As for CDI, our code under
> > >> Quarkus
> > >> would only have to have certain annotations, without having to
> > >> extend/implement framework interfaces.
> > >>
> > >> Cheers,
> > >> Dmitri.
> > >>
> > >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <
> collado.mike@gmail.com>
> > >> wrote:
> > >>
> > >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a
> CDI
> > >> impl.
> > >> > I only spent yesterday and today on this, so it's not complete, but
> it
> > >> is
> > >> > functional and the tests pass :)
> > >> >
> > >> > I took a lot of the common ideas from the Quarkus branch (e.g.,
> deleted
> > >> all
> > >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config.
> I
> > >> > figured out how to use the Dropwizard Yaml to specify the
> > >> > implementation of, e.g. the Authenticator and the
> > >> MetaStoreManagerFactory,
> > >> > but have the instances managed and injectable by HK2. The goal there
> > >> was to
> > >> > just keep the existing configuration format, but change the impl
> under
> > >> the
> > >> > hood. I'm not married to the idea and I'm interested to see if the
> > >> > jakarta.enterprise.inject.* annotations/interfaces that are used in
> the
> > >> > Quarkus branch can make this simpler. However, I do think it would
> > >> ideal if
> > >> > we can get it working with the existing Yaml configuration, at
> least in
> > >> the
> > >> > short term.
> > >> >
> > >> > I did add support for a @RealmScope annotation to support
> restricting
> > >> items
> > >> > to a given realm, such as the EntityCache and the
> PolarisGrantManager.
> > >> This
> > >> > allowed me to do things like hide the grant lookups from the
> > >> EntityCache so
> > >> > that the Resolver doesn't have to pass around the
> ResolvedPolarisEntity,
> > >> > but instead the grants are found from the cache without making it
> overt
> > >> in
> > >> > the PolarisAuthorizer API. This was one of my original goals with
> > >> breaking
> > >> > up the PolarisMetaStoreManager API into multiple interfaces. Right
> now,
> > >> > everything still ties back to the configured PolarisMetaStoreManager
> > >> > implementation, but eventually we can get to where the GrantManager,
> > >> > CredentialVendor, etc. can all be swapped out for different
> > >> > implementations.
> > >> >
> > >> > Please take a look at the changes at
> > >> >
> > >> >
> > >>
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > >> > and let me know what you think.
> > >> >
> > >> > Mike
> > >> >
> > >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> collado.mike@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > I added a table to the README with the differences that were
> called
> > >> out.
> > >> > I
> > >> > > added some details that I think are worth understanding better.
> E.g.,
> > >> the
> > >> > > Json layout we added has specific custom functionality we wanted
> for
> > >> > > supporting key/value pairs and the micrometer annotation was
> added for
> > >> > some
> > >> > > custom support we wanted aside from what is supported with the
> default
> > >> > > dropwizard metrics support.
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > >> > >
> > >> > > I think it would be great if others could add to the table with
> > >> features
> > >> > > that would be impacted by the migration and to call out the level
> of
> > >> > effort
> > >> > > in both dropwizard and quarkus.
> > >> > >
> > >> > > Another possible consideration would be upgrading our Dropwizard
> > >> > > dependency to the latest development version. It may be the case
> that
> > >> > doing
> > >> > > so would address some of the targeted features with less effort in
> > >> > > migrating.
> > >> > >
> > >> > > Mike
> > >> > >
> > >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> Hi folks,
> > >> > >>
> > >> > >> Following the last community sync meeting, we create a branch
> > >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > >> > >>
> > >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > >> > >>
> > >> > >> We still have work to do, but you can already take a look and
> > >> > >> experiment (in the polaris-service-quarkus module).
> > >> > >>
> > >> > >> We added a README.md file to:
> > >> > >> 1. Highlight the main differences (in terms of code) between
> > >> > >> Dropwizard and Quarkus
> > >> > >> 2. To build and run Polaris powered by Quarkus
> > >> > >> 3. The list of TODO items
> > >> > >>
> > >> > >>
> > >> > >>
> > >> >
> > >>
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > >> > >>
> > >> > >> If anyone wants to contribute on the branch before creating the
> PR,
> > >> > >> please let me know, I will add you as contributor on the branch.
> > >> > >>
> > >> > >> Any comments or questions are welcome !
> > >> > >>
> > >> > >> Thanks,
> > >> > >> Regards
> > >> > >> JB
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Michael
Thanks for the update. It's great !
Using CDI annotation is way neater than yaml bean verbose definition
:) We can always improve step by step (about the
PolarisMetaStoreManager and the CallContext).
NB: the Quarkus PR is also there to have side/side comparison.
My comment is that we should also consider the framework ecosystem: I
see new Polaris PRs (like the one about dynamic config) that
reimplement from scratch what already provided by Quarkus (thanks to
the extensions). I think it's urgent to have a consensus here to know
in which direction we are going.
Let's chat together about that.
Thanks !
Regards
JB
On Tue, Nov 26, 2024 at 6:06 AM Michael Collado <co...@gmail.com> wrote:
>
> FYI, I updated the branch to exclude HK2 dependencies from the core module
> -
> https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
> . The Factory implementations are defined in the polaris-service module.
>
> I also created another branch at
> https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
> that merged the PR I have at https://github.com/apache/polaris/pull/465
> using the CDI annotations rather than the extra factories I had created in
> the initial PR. This is still moving us toward supporting different
> implementations of the various Polaris*Manager interfaces - still relying
> on the current concrete PolarisMetaStoreManager, but we can easily extend
> the HK2 service-locator file to support different manager implementations
> in future PRs. (I prefer smaller incremental moves before we go with
> fully-swappable implementations).
>
> Mike
>
> On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> wrote:
>
> > Hey thanks for calling that out. It’s strictly a convenience change and
> > actually isn’t necessary. I added that in because, at the end of the day,
> > most components need a PolarisMetaStoreManager, not a
> > MetaStoreManagerFactory. Registering it as a factory meant that components
> > could declare a dependency on the PolarisMetaStoreManager interface and it
> > would use the factory to create the right instance. Because the factory
> > method is annotated @RealmScope, the instance it creates is specific to the
> > realm. I could have done it in the PolarisApplication, as I did the
> > PolarisGrantManagerFactory at
> > https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> > . I was just lazy when I did that one :) We can keep all the hk2-specific
> > code out of the core module.
> >
> > I do want to point out where the @RealmScope differs from @RequestScope -
> > the beans defined in @RequestScope are cleaned up at the end of the
> > request. The ones defined in the @RealmScope are reused across requests.
> > This makes a difference with classes like the EntityCache that depends on
> > being able to reuse, e.g., catalogs, but ensure the cache is specific to
> > the current realm. We can, of course, pass around a bunch of
> > @ApplicationScope factories that return the realm-specific beans, but that
> > means we can't do things like add the PolarisGrantManager interface without
> > also adding a new realm-specific factory, even if the PolarisGrantManager
> > implementation is really just the PolarisMetaStoreManager itself. I think,
> > ideally, we avoid declaring dependencies on factories and have components
> > declare dependencies on the specific realm-scoped beans they need and let
> > the CDI framework work out where the beans come from. This is basically
> > what I was doing with the MetaStoreManagerFactory interface - as I said, I
> > was just being lazy by extending Factory directly.
> >
> > Mike
> >
> >
> >
> > On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> > <dm...@dremio.com.invalid> wrote:
> >
> >> I had a quite brief look at Mike's HK2 branch, so apologies if I'm missing
> >> the bigger picture.
> >>
> >> Still, I see that MetaStoreManagerFactory no longer extends the DW
> >> Discoverable, but it extends the HK2 Factory class now. So we're basically
> >> trading one framework for another as a code-level runtime dependency. I
> >> would really like it if we could avoid that.
> >>
> >> I think one of the key benefits of Quarkus is abstracting from framework
> >> code in our class hierarchies. Granted, Quarkus requires certain Web App
> >> frameworks, but that's at another level. As for CDI, our code under
> >> Quarkus
> >> would only have to have certain annotations, without having to
> >> extend/implement framework interfaces.
> >>
> >> Cheers,
> >> Dmitri.
> >>
> >> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> >> wrote:
> >>
> >> > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> >> impl.
> >> > I only spent yesterday and today on this, so it's not complete, but it
> >> is
> >> > functional and the tests pass :)
> >> >
> >> > I took a lot of the common ideas from the Quarkus branch (e.g., deleted
> >> all
> >> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> >> > figured out how to use the Dropwizard Yaml to specify the
> >> > implementation of, e.g. the Authenticator and the
> >> MetaStoreManagerFactory,
> >> > but have the instances managed and injectable by HK2. The goal there
> >> was to
> >> > just keep the existing configuration format, but change the impl under
> >> the
> >> > hood. I'm not married to the idea and I'm interested to see if the
> >> > jakarta.enterprise.inject.* annotations/interfaces that are used in the
> >> > Quarkus branch can make this simpler. However, I do think it would
> >> ideal if
> >> > we can get it working with the existing Yaml configuration, at least in
> >> the
> >> > short term.
> >> >
> >> > I did add support for a @RealmScope annotation to support restricting
> >> items
> >> > to a given realm, such as the EntityCache and the PolarisGrantManager.
> >> This
> >> > allowed me to do things like hide the grant lookups from the
> >> EntityCache so
> >> > that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
> >> > but instead the grants are found from the cache without making it overt
> >> in
> >> > the PolarisAuthorizer API. This was one of my original goals with
> >> breaking
> >> > up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> >> > everything still ties back to the configured PolarisMetaStoreManager
> >> > implementation, but eventually we can get to where the GrantManager,
> >> > CredentialVendor, etc. can all be swapped out for different
> >> > implementations.
> >> >
> >> > Please take a look at the changes at
> >> >
> >> >
> >> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> >> > and let me know what you think.
> >> >
> >> > Mike
> >> >
> >> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
> >> > wrote:
> >> >
> >> > > I added a table to the README with the differences that were called
> >> out.
> >> > I
> >> > > added some details that I think are worth understanding better. E.g.,
> >> the
> >> > > Json layout we added has specific custom functionality we wanted for
> >> > > supporting key/value pairs and the micrometer annotation was added for
> >> > some
> >> > > custom support we wanted aside from what is supported with the default
> >> > > dropwizard metrics support.
> >> > >
> >> > >
> >> > >
> >> >
> >> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> >> > >
> >> > > I think it would be great if others could add to the table with
> >> features
> >> > > that would be impacted by the migration and to call out the level of
> >> > effort
> >> > > in both dropwizard and quarkus.
> >> > >
> >> > > Another possible consideration would be upgrading our Dropwizard
> >> > > dependency to the latest development version. It may be the case that
> >> > doing
> >> > > so would address some of the targeted features with less effort in
> >> > > migrating.
> >> > >
> >> > > Mike
> >> > >
> >> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb@nanthrax.net
> >> >
> >> > > wrote:
> >> > >
> >> > >> Hi folks,
> >> > >>
> >> > >> Following the last community sync meeting, we create a branch
> >> > >> demonstrating use of Quarkus to powered Apache Polaris:
> >> > >>
> >> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> >> > >>
> >> > >> We still have work to do, but you can already take a look and
> >> > >> experiment (in the polaris-service-quarkus module).
> >> > >>
> >> > >> We added a README.md file to:
> >> > >> 1. Highlight the main differences (in terms of code) between
> >> > >> Dropwizard and Quarkus
> >> > >> 2. To build and run Polaris powered by Quarkus
> >> > >> 3. The list of TODO items
> >> > >>
> >> > >>
> >> > >>
> >> >
> >> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> >> > >>
> >> > >> If anyone wants to contribute on the branch before creating the PR,
> >> > >> please let me know, I will add you as contributor on the branch.
> >> > >>
> >> > >> Any comments or questions are welcome !
> >> > >>
> >> > >> Thanks,
> >> > >> Regards
> >> > >> JB
> >> > >>
> >> > >
> >> >
> >>
> >
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Michael Collado <co...@gmail.com>.
FYI, I updated the branch to exclude HK2 dependencies from the core module
-
https://github.com/apache/polaris/compare/main...collado-mike:polaris:mcollado-hk2-di
. The Factory implementations are defined in the polaris-service module.
I also created another branch at
https://github.com/collado-mike/polaris/compare/mcollado-hk2-di...mcollado-hk2-di-grantmanager
that merged the PR I have at https://github.com/apache/polaris/pull/465
using the CDI annotations rather than the extra factories I had created in
the initial PR. This is still moving us toward supporting different
implementations of the various Polaris*Manager interfaces - still relying
on the current concrete PolarisMetaStoreManager, but we can easily extend
the HK2 service-locator file to support different manager implementations
in future PRs. (I prefer smaller incremental moves before we go with
fully-swappable implementations).
Mike
On Tue, Nov 19, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
wrote:
> Hey thanks for calling that out. It’s strictly a convenience change and
> actually isn’t necessary. I added that in because, at the end of the day,
> most components need a PolarisMetaStoreManager, not a
> MetaStoreManagerFactory. Registering it as a factory meant that components
> could declare a dependency on the PolarisMetaStoreManager interface and it
> would use the factory to create the right instance. Because the factory
> method is annotated @RealmScope, the instance it creates is specific to the
> realm. I could have done it in the PolarisApplication, as I did the
> PolarisGrantManagerFactory at
> https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
> . I was just lazy when I did that one :) We can keep all the hk2-specific
> code out of the core module.
>
> I do want to point out where the @RealmScope differs from @RequestScope -
> the beans defined in @RequestScope are cleaned up at the end of the
> request. The ones defined in the @RealmScope are reused across requests.
> This makes a difference with classes like the EntityCache that depends on
> being able to reuse, e.g., catalogs, but ensure the cache is specific to
> the current realm. We can, of course, pass around a bunch of
> @ApplicationScope factories that return the realm-specific beans, but that
> means we can't do things like add the PolarisGrantManager interface without
> also adding a new realm-specific factory, even if the PolarisGrantManager
> implementation is really just the PolarisMetaStoreManager itself. I think,
> ideally, we avoid declaring dependencies on factories and have components
> declare dependencies on the specific realm-scoped beans they need and let
> the CDI framework work out where the beans come from. This is basically
> what I was doing with the MetaStoreManagerFactory interface - as I said, I
> was just being lazy by extending Factory directly.
>
> Mike
>
>
>
> On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
> <dm...@dremio.com.invalid> wrote:
>
>> I had a quite brief look at Mike's HK2 branch, so apologies if I'm missing
>> the bigger picture.
>>
>> Still, I see that MetaStoreManagerFactory no longer extends the DW
>> Discoverable, but it extends the HK2 Factory class now. So we're basically
>> trading one framework for another as a code-level runtime dependency. I
>> would really like it if we could avoid that.
>>
>> I think one of the key benefits of Quarkus is abstracting from framework
>> code in our class hierarchies. Granted, Quarkus requires certain Web App
>> frameworks, but that's at another level. As for CDI, our code under
>> Quarkus
>> would only have to have certain annotations, without having to
>> extend/implement framework interfaces.
>>
>> Cheers,
>> Dmitri.
>>
>> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
>> wrote:
>>
>> > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
>> impl.
>> > I only spent yesterday and today on this, so it's not complete, but it
>> is
>> > functional and the tests pass :)
>> >
>> > I took a lot of the common ideas from the Quarkus branch (e.g., deleted
>> all
>> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
>> > figured out how to use the Dropwizard Yaml to specify the
>> > implementation of, e.g. the Authenticator and the
>> MetaStoreManagerFactory,
>> > but have the instances managed and injectable by HK2. The goal there
>> was to
>> > just keep the existing configuration format, but change the impl under
>> the
>> > hood. I'm not married to the idea and I'm interested to see if the
>> > jakarta.enterprise.inject.* annotations/interfaces that are used in the
>> > Quarkus branch can make this simpler. However, I do think it would
>> ideal if
>> > we can get it working with the existing Yaml configuration, at least in
>> the
>> > short term.
>> >
>> > I did add support for a @RealmScope annotation to support restricting
>> items
>> > to a given realm, such as the EntityCache and the PolarisGrantManager.
>> This
>> > allowed me to do things like hide the grant lookups from the
>> EntityCache so
>> > that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
>> > but instead the grants are found from the cache without making it overt
>> in
>> > the PolarisAuthorizer API. This was one of my original goals with
>> breaking
>> > up the PolarisMetaStoreManager API into multiple interfaces. Right now,
>> > everything still ties back to the configured PolarisMetaStoreManager
>> > implementation, but eventually we can get to where the GrantManager,
>> > CredentialVendor, etc. can all be swapped out for different
>> > implementations.
>> >
>> > Please take a look at the changes at
>> >
>> >
>> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
>> > and let me know what you think.
>> >
>> > Mike
>> >
>> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
>> > wrote:
>> >
>> > > I added a table to the README with the differences that were called
>> out.
>> > I
>> > > added some details that I think are worth understanding better. E.g.,
>> the
>> > > Json layout we added has specific custom functionality we wanted for
>> > > supporting key/value pairs and the micrometer annotation was added for
>> > some
>> > > custom support we wanted aside from what is supported with the default
>> > > dropwizard metrics support.
>> > >
>> > >
>> > >
>> >
>> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
>> > >
>> > > I think it would be great if others could add to the table with
>> features
>> > > that would be impacted by the migration and to call out the level of
>> > effort
>> > > in both dropwizard and quarkus.
>> > >
>> > > Another possible consideration would be upgrading our Dropwizard
>> > > dependency to the latest development version. It may be the case that
>> > doing
>> > > so would address some of the targeted features with less effort in
>> > > migrating.
>> > >
>> > > Mike
>> > >
>> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb@nanthrax.net
>> >
>> > > wrote:
>> > >
>> > >> Hi folks,
>> > >>
>> > >> Following the last community sync meeting, we create a branch
>> > >> demonstrating use of Quarkus to powered Apache Polaris:
>> > >>
>> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
>> > >>
>> > >> We still have work to do, but you can already take a look and
>> > >> experiment (in the polaris-service-quarkus module).
>> > >>
>> > >> We added a README.md file to:
>> > >> 1. Highlight the main differences (in terms of code) between
>> > >> Dropwizard and Quarkus
>> > >> 2. To build and run Polaris powered by Quarkus
>> > >> 3. The list of TODO items
>> > >>
>> > >>
>> > >>
>> >
>> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
>> > >>
>> > >> If anyone wants to contribute on the branch before creating the PR,
>> > >> please let me know, I will add you as contributor on the branch.
>> > >>
>> > >> Any comments or questions are welcome !
>> > >>
>> > >> Thanks,
>> > >> Regards
>> > >> JB
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Michael Collado <co...@gmail.com>.
Hey thanks for calling that out. It’s strictly a convenience change and
actually isn’t necessary. I added that in because, at the end of the day,
most components need a PolarisMetaStoreManager, not a
MetaStoreManagerFactory. Registering it as a factory meant that components
could declare a dependency on the PolarisMetaStoreManager interface and it
would use the factory to create the right instance. Because the factory
method is annotated @RealmScope, the instance it creates is specific to the
realm. I could have done it in the PolarisApplication, as I did the
PolarisGrantManagerFactory at
https://github.com/collado-mike/polaris/blob/mcollado-hk2-di/polaris-service/src/main/java/org/apache/polaris/service/PolarisApplication.java#L518-L531
. I was just lazy when I did that one :) We can keep all the hk2-specific
code out of the core module.
I do want to point out where the @RealmScope differs from @RequestScope -
the beans defined in @RequestScope are cleaned up at the end of the
request. The ones defined in the @RealmScope are reused across requests.
This makes a difference with classes like the EntityCache that depends on
being able to reuse, e.g., catalogs, but ensure the cache is specific to
the current realm. We can, of course, pass around a bunch of
@ApplicationScope factories that return the realm-specific beans, but that
means we can't do things like add the PolarisGrantManager interface without
also adding a new realm-specific factory, even if the PolarisGrantManager
implementation is really just the PolarisMetaStoreManager itself. I think,
ideally, we avoid declaring dependencies on factories and have components
declare dependencies on the specific realm-scoped beans they need and let
the CDI framework work out where the beans come from. This is basically
what I was doing with the MetaStoreManagerFactory interface - as I said, I
was just being lazy by extending Factory directly.
Mike
On Tue, Nov 19, 2024 at 5:08 PM Dmitri Bourlatchkov
<dm...@dremio.com.invalid> wrote:
> I had a quite brief look at Mike's HK2 branch, so apologies if I'm missing
> the bigger picture.
>
> Still, I see that MetaStoreManagerFactory no longer extends the DW
> Discoverable, but it extends the HK2 Factory class now. So we're basically
> trading one framework for another as a code-level runtime dependency. I
> would really like it if we could avoid that.
>
> I think one of the key benefits of Quarkus is abstracting from framework
> code in our class hierarchies. Granted, Quarkus requires certain Web App
> frameworks, but that's at another level. As for CDI, our code under Quarkus
> would only have to have certain annotations, without having to
> extend/implement framework interfaces.
>
> Cheers,
> Dmitri.
>
> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
> wrote:
>
> > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> impl.
> > I only spent yesterday and today on this, so it's not complete, but it is
> > functional and the tests pass :)
> >
> > I took a lot of the common ideas from the Quarkus branch (e.g., deleted
> all
> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> > figured out how to use the Dropwizard Yaml to specify the
> > implementation of, e.g. the Authenticator and the
> MetaStoreManagerFactory,
> > but have the instances managed and injectable by HK2. The goal there was
> to
> > just keep the existing configuration format, but change the impl under
> the
> > hood. I'm not married to the idea and I'm interested to see if the
> > jakarta.enterprise.inject.* annotations/interfaces that are used in the
> > Quarkus branch can make this simpler. However, I do think it would ideal
> if
> > we can get it working with the existing Yaml configuration, at least in
> the
> > short term.
> >
> > I did add support for a @RealmScope annotation to support restricting
> items
> > to a given realm, such as the EntityCache and the PolarisGrantManager.
> This
> > allowed me to do things like hide the grant lookups from the EntityCache
> so
> > that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
> > but instead the grants are found from the cache without making it overt
> in
> > the PolarisAuthorizer API. This was one of my original goals with
> breaking
> > up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> > everything still ties back to the configured PolarisMetaStoreManager
> > implementation, but eventually we can get to where the GrantManager,
> > CredentialVendor, etc. can all be swapped out for different
> > implementations.
> >
> > Please take a look at the changes at
> >
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > and let me know what you think.
> >
> > Mike
> >
> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
> > wrote:
> >
> > > I added a table to the README with the differences that were called
> out.
> > I
> > > added some details that I think are worth understanding better. E.g.,
> the
> > > Json layout we added has specific custom functionality we wanted for
> > > supporting key/value pairs and the micrometer annotation was added for
> > some
> > > custom support we wanted aside from what is supported with the default
> > > dropwizard metrics support.
> > >
> > >
> > >
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > >
> > > I think it would be great if others could add to the table with
> features
> > > that would be impacted by the migration and to call out the level of
> > effort
> > > in both dropwizard and quarkus.
> > >
> > > Another possible consideration would be upgrading our Dropwizard
> > > dependency to the latest development version. It may be the case that
> > doing
> > > so would address some of the targeted features with less effort in
> > > migrating.
> > >
> > > Mike
> > >
> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> Following the last community sync meeting, we create a branch
> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > >>
> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > >>
> > >> We still have work to do, but you can already take a look and
> > >> experiment (in the polaris-service-quarkus module).
> > >>
> > >> We added a README.md file to:
> > >> 1. Highlight the main differences (in terms of code) between
> > >> Dropwizard and Quarkus
> > >> 2. To build and run Polaris powered by Quarkus
> > >> 3. The list of TODO items
> > >>
> > >>
> > >>
> >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > >>
> > >> If anyone wants to contribute on the branch before creating the PR,
> > >> please let me know, I will add you as contributor on the branch.
> > >>
> > >> Any comments or questions are welcome !
> > >>
> > >> Thanks,
> > >> Regards
> > >> JB
> > >>
> > >
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
I had a quite brief look at Mike's HK2 branch, so apologies if I'm missing
the bigger picture.
Still, I see that MetaStoreManagerFactory no longer extends the DW
Discoverable, but it extends the HK2 Factory class now. So we're basically
trading one framework for another as a code-level runtime dependency. I
would really like it if we could avoid that.
I think one of the key benefits of Quarkus is abstracting from framework
code in our class hierarchies. Granted, Quarkus requires certain Web App
frameworks, but that's at another level. As for CDI, our code under Quarkus
would only have to have certain annotations, without having to
extend/implement framework interfaces.
Cheers,
Dmitri.
On Thu, Nov 7, 2024 at 8:37 PM Michael Collado <co...@gmail.com>
wrote:
> FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI impl.
> I only spent yesterday and today on this, so it's not complete, but it is
> functional and the tests pass :)
>
> I took a lot of the common ideas from the Quarkus branch (e.g., deleted all
> the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> figured out how to use the Dropwizard Yaml to specify the
> implementation of, e.g. the Authenticator and the MetaStoreManagerFactory,
> but have the instances managed and injectable by HK2. The goal there was to
> just keep the existing configuration format, but change the impl under the
> hood. I'm not married to the idea and I'm interested to see if the
> jakarta.enterprise.inject.* annotations/interfaces that are used in the
> Quarkus branch can make this simpler. However, I do think it would ideal if
> we can get it working with the existing Yaml configuration, at least in the
> short term.
>
> I did add support for a @RealmScope annotation to support restricting items
> to a given realm, such as the EntityCache and the PolarisGrantManager. This
> allowed me to do things like hide the grant lookups from the EntityCache so
> that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
> but instead the grants are found from the cache without making it overt in
> the PolarisAuthorizer API. This was one of my original goals with breaking
> up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> everything still ties back to the configured PolarisMetaStoreManager
> implementation, but eventually we can get to where the GrantManager,
> CredentialVendor, etc. can all be swapped out for different
> implementations.
>
> Please take a look at the changes at
>
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> and let me know what you think.
>
> Mike
>
> On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
> wrote:
>
> > I added a table to the README with the differences that were called out.
> I
> > added some details that I think are worth understanding better. E.g., the
> > Json layout we added has specific custom functionality we wanted for
> > supporting key/value pairs and the micrometer annotation was added for
> some
> > custom support we wanted aside from what is supported with the default
> > dropwizard metrics support.
> >
> >
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> >
> > I think it would be great if others could add to the table with features
> > that would be impacted by the migration and to call out the level of
> effort
> > in both dropwizard and quarkus.
> >
> > Another possible consideration would be upgrading our Dropwizard
> > dependency to the latest development version. It may be the case that
> doing
> > so would address some of the targeted features with less effort in
> > migrating.
> >
> > Mike
> >
> > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> >> Hi folks,
> >>
> >> Following the last community sync meeting, we create a branch
> >> demonstrating use of Quarkus to powered Apache Polaris:
> >>
> >> https://github.com/jbonofre/polaris/tree/QUARKUS
> >>
> >> We still have work to do, but you can already take a look and
> >> experiment (in the polaris-service-quarkus module).
> >>
> >> We added a README.md file to:
> >> 1. Highlight the main differences (in terms of code) between
> >> Dropwizard and Quarkus
> >> 2. To build and run Polaris powered by Quarkus
> >> 3. The list of TODO items
> >>
> >>
> >>
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> >>
> >> If anyone wants to contribute on the branch before creating the PR,
> >> please let me know, I will add you as contributor on the branch.
> >>
> >> Any comments or questions are welcome !
> >>
> >> Thanks,
> >> Regards
> >> JB
> >>
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Alex Dutra <al...@dremio.com.INVALID>.
Hi,
> @RealmContext is to support beans that are scoped to the
> realm, not the request. E.g., the MetaStoreManagerFactory maintains a map
> of realm-specific PolarisMetaStoreManagers that are reused for every
> request for a realm. The scope is intended to be larger
> than @RequestScoped, but smaller than @ApplicationScoped.
As long as this solution works, I'm fine with it. I'd observe though that
beans like MetaStoreManagerFactory, which indeed are capable of managing
many realms, are natural good fits for the application scope. In the
Quarkus branch, this bean is annotated with @ApplicationScoped, while all
beans that use it and operate on a specific realm, are all annotated with
@RequestScoped.
I would _really_ prefer to avoid that if possible. The whole argument for
> using standardized annotations is that they can be used by multiple
> frameworks without being tied to any one.
To clarify my statement, I think that most consensual annotations would go
in the common module, and most notably, the @Inject annotation, as well as
@Context, @RequestScoped, @ApplicationScoped @Produces, @Vetoed, etc. When
it comes to selecting beans at runtime though, we either need to agree on a
common technique (knowing that Quarkus offers interesting possibilities,
but those are outside of CDI specs), or resort to specific code in runtime
modules – which I don't think would be that bad.
Using subclasses, it could look like the below example:
// common service module
@ApplicationScoped
public class DefaultFooFactory implements FooFactory { /* all business
logic here */ }
// dropwizard runtime module
@SomeBeanQualifier
public class DropwizardFooFactory extends DefaultFooFactory { /* empty
class */ }
// quarkus runtime module
@SomeOtherBeanQualifier
public class QuarkusFooFactory extends DefaultFooFactory { /* empty class
*/ }
Another option that avoids subclasses is to use a producer method instead,
e.g.:
// quarkus runtime module
@Produces @ApplicationScoped @SomeOtherBeanQualifier
public FooFactory fooFactory() {
return new DefaultFooFactory();
}
I definitely want
> to avoid duplicating test classes, as that's pretty much guaranteed to
> ensure that something gets broken and is uncaught because we lacked test
> coverage.
Agreed, it's certainly possible to commonize the tests with a bit of
effort. Andrew's PR is merged already; I think it's rather orthogonal to
the issue around duplicating tests. The new TestEnvironmentExtension is not
currently used in the Quarkus port, but it certainly could.
Thanks,
Alex
On Thu, Nov 14, 2024 at 6:53 PM Michael Collado <co...@gmail.com>
wrote:
> > Although, I wonder if this custom
> context isn't identical to the standard request context? IOW, is it
> possible for the Polaris context to change during an HTTP call to Polaris
> API? I don't think so, which leads me to think that, at least in Quarkus,
> we'd use the standard @RequestScoped annotation for passing things like
> this.
>
> TBH, I'm a bit fuzzy on the details of the custom scopes in HK2, but the
> intention of the @RealmContext is to support beans that are scoped to the
> realm, not the request. E.g., the MetaStoreManagerFactory maintains a map
> of realm-specific PolarisMetaStoreManagers that are reused for every
> request for a realm. The scope is intended to be larger
> than @RequestScoped, but smaller than @ApplicationScoped. This is
> especially important for the EntityCache, which maintains a cache of the
> versioned entities and allows reuse of common entities (e.g., Catalogs and
> PrincipalRoles) without having to fetch them for every request. I _think_
> what I did with @RealmContext is to maintain a map of reusable beans that
> are not recreated for each request (the context itself is a singleton).
>
> > And to circle back to DI, whether this common module would contain
> classes
> annotated with Jakarta CDI annotations is imho up for debate. This would
> simplify things for sure, but on the flip side, would require both runtimes
> to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
> beans). I think that this is going to be too restrictive. An alternative
> would be to keep the common classes clean of CDI annotations and let the
> runtime modules create subclasses annotated with their own annotations.
>
> I would _really_ prefer to avoid that if possible. The whole argument for
> using standardized annotations is that they can be used by multiple
> frameworks without being tied to any one. Personally, I'm not married to
> the @Named annotation or any of the specifics I used to get my prototype to
> work. If there is a more generalized annotation or alternative to
> specifying named instances of contracts, I'm all for it.
>
> Andrew Guterman <https://github.com/andrew4699> has been working on making
> the target test environment more pluggable (see
> https://github.com/apache/polaris/pull/391 ). Maybe we can get his input
> on
> how to structure the tests in a way that they can hit the target runtime
> without making them framework specific. This is super useful for, e.g.,
> testing in a CD pipeline where you might hit a docker image or k8s
> deployment to validate behavior before deploying to prod. I definitely want
> to avoid duplicating test classes, as that's pretty much guaranteed to
> ensure that something gets broken and is uncaught because we lacked test
> coverage.
>
> Mike
>
>
> On Thu, Nov 14, 2024 at 7:21 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi
> >
> > Using @Named is fine with Quarkus. We can also create custom qualifier
> > (with Literal): named is actually a qualifier :)
> >
> > I think we discussed about the plan, and to my understanding:
> > 1. I open the Quarkus PR (as draft) to keep what we did
> > 2. We improve the current structure:
> > 2.1. polaris-core should not depend to dropwizard (remove discoverable
> > here)
> > 2.2. we create polaris-service-common containing all the beans,
> > without dependency to Dropwizard (or Quarkus). Related to that, itests
> > should go into polaris-service-itest.
> > 2.2. we create polaris-service-dropwizard where we take
> > polaris-service-common and do the beans injection with Dropwizard.
> > (2.3). we can create polaris-service-quarkus where we take
> > polaris-service-common and do the beans injection with Quarkus
> >
> > My standpoint is still the same: if the "clean refactoring" is a good
> > thing, I think it's not possible to have both
> > polaris-service-dropwizard and polaris-service-quarkus, for three
> > reasons:
> > - it doesn't bring any benefits for the users, only confusion (which
> > service should I use)
> > - it will be almost impossible to maintain (in sync)
> > - it will prevent to leverage extensions provided (especially by Quarkus)
> >
> > So, my proposal is to move forward on the refactoring (we have a
> > consensus here).
> > I will work with Michael to identify exactly the gaps between
> > polaris-service-dropwizard and polaris-service-quarkus (configuration,
> > extensions, ...).
> >
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On Wed, Nov 13, 2024 at 8:13 PM Alex Dutra
> > <al...@dremio.com.invalid> wrote:
> > >
> > > Hi Michael, hi all,
> > >
> > > I had a look at your branch and am very pleased to see that it was
> > > relatively easy to get DI working with DW + HK2. Good news!
> > >
> > > Digging a bit into the details, I see that most beans are annotated
> with
> > > @Named now. This is probably OK for Quarkus too, although there are
> other
> > > techniques that allow to select a specific implementation at runtime.
> > >
> > > I also think that the @RealmContext annotation is very smart, and
> > resonates
> > > with how I handled that in Quarkus. Although, I wonder if this custom
> > > context isn't identical to the standard request context? IOW, is it
> > > possible for the Polaris context to change during an HTTP call to
> Polaris
> > > API? I don't think so, which leads me to think that, at least in
> Quarkus,
> > > we'd use the standard @RequestScoped annotation for passing things like
> > > this.
> > >
> > > But now taking a step back and looking at the big picture, I think that
> > > what we need now is a plan forward. Given that many of us voiced
> concerns
> > > around an all-or-nothing switch from DW to Quarkus (quite
> understandably
> > > so), and assuming that there is agreement to maintain both runtimes (DW
> > and
> > > Quarkus) for some time, I'd like to suggest the phased approach below:
> > >
> > > 1. Remove the dependency to DW from polaris-core – there is a PR for
> > > that already: https://github.com/apache/polaris/pull/435
> > > 2. Refactor polaris-service into 2 modules: e.g.
> > "polaris-service-common"
> > > and "polaris-service-dropwizard" – I believe Dmitri is already
> working
> > > on this
> > > 3. Introduce DI + HK2 in polaris-service-dropwizard – that would
> come
> > > from your branch basically
> > > 4. Introduce polaris-service-quarkus – that would come mostly from
> > JB's
> > > branch
> > >
> > > Regarding step #2, it seems we can easily put 80 to 90% of the actual
> > > polaris-service production classes into the common module. The
> dropwizard
> > > runtime module would contain just a handful of DW-specific classes.
> > >
> > > The biggest issue to solve with step #2 is imho test classes. Quarkus
> and
> > > DW have diverse requirements for tests, but we may still be able to
> > > refactor things using abstract classes or extensions, and reuse most of
> > > them. In any case, I think that if, at least at the beginning, test
> > classes
> > > are duplicated in each runtime module, that's acceptable until we
> figure
> > > out a better way.
> > >
> > > And to circle back to DI, whether this common module would contain
> > classes
> > > annotated with Jakarta CDI annotations is imho up for debate. This
> would
> > > simplify things for sure, but on the flip side, would require both
> > runtimes
> > > to use the exact same CDI mechanisms (e.g. both runtimes would use
> @Named
> > > beans). I think that this is going to be too restrictive. An
> alternative
> > > would be to keep the common classes clean of CDI annotations and let
> the
> > > runtime modules create subclasses annotated with their own annotations.
> > >
> > > And back to the @RealmContext topic, that's the other big aspect that
> we
> > > will need to solve (maybe step #5?): CallContext is heavily reliant on
> > > ThreadLocal, and that is not very CDI-friendly, as you've seen. For
> now,
> > I
> > > did a patchy workaround for Quarkus. But we'd need to refactor that
> code
> > > cleanly, either by creating a custom context as you did, or by making
> > > CallContext a request-scoped bean that is evaluated for each HTTP
> > request.
> > > Either way, it then needs to be propagated to other threads (e.g. in
> > > TaskExecutorImpl); for Quarkus, this will require using proper context
> > > propagation <https://quarkus.io/guides/context-propagation> (for DW,
> > maybe
> > > ThreadLocal is fine here). And since CallContext is in the core module,
> > we
> > > will likely need to remove the static methods getCurrentContext() and
> > > setCurrentContext() and retrieve the context only by scope resolution
> > > (which, in turn, may use ThreadLocal under the hood if that's
> appropriate
> > > for DW).
> > >
> > > If everybody is on board with the plan above, how about we create GH
> > issues
> > > for each step, and maybe even a GH Project to track progress on this
> > front
> > > and split the effort?
> > >
> > > Thanks,
> > >
> > > Alex
> > >
> > >
> > > On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi Michael
> > > >
> > > > Thanks for the update !
> > > >
> > > > As discussed during the last community meeting:
> > > > 1. I will open the draft PR with Quarkus in the meantime
> > > > 2. I will take a look on your branch powered by Glassfish HK2 CDI
> > > > (give me a few days as I was travelling this week :))
> > > >
> > > > Thanks !
> > > > Regards
> > > >
> > > > On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <
> collado.mike@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > FYI, I took a stab at seeing how Polaris would work with HK2 as a
> CDI
> > > > impl.
> > > > > I only spent yesterday and today on this, so it's not complete, but
> > it is
> > > > > functional and the tests pass :)
> > > > >
> > > > > I took a lot of the common ideas from the Quarkus branch (e.g.,
> > deleted
> > > > all
> > > > > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml
> config. I
> > > > > figured out how to use the Dropwizard Yaml to specify the
> > > > > implementation of, e.g. the Authenticator and the
> > > > MetaStoreManagerFactory,
> > > > > but have the instances managed and injectable by HK2. The goal
> there
> > was
> > > > to
> > > > > just keep the existing configuration format, but change the impl
> > under
> > > > the
> > > > > hood. I'm not married to the idea and I'm interested to see if the
> > > > > jakarta.enterprise.inject.* annotations/interfaces that are used in
> > the
> > > > > Quarkus branch can make this simpler. However, I do think it would
> > ideal
> > > > if
> > > > > we can get it working with the existing Yaml configuration, at
> least
> > in
> > > > the
> > > > > short term.
> > > > >
> > > > > I did add support for a @RealmScope annotation to support
> restricting
> > > > items
> > > > > to a given realm, such as the EntityCache and the
> > PolarisGrantManager.
> > > > This
> > > > > allowed me to do things like hide the grant lookups from the
> > EntityCache
> > > > so
> > > > > that the Resolver doesn't have to pass around the
> > ResolvedPolarisEntity,
> > > > > but instead the grants are found from the cache without making it
> > overt
> > > > in
> > > > > the PolarisAuthorizer API. This was one of my original goals with
> > > > breaking
> > > > > up the PolarisMetaStoreManager API into multiple interfaces. Right
> > now,
> > > > > everything still ties back to the configured
> PolarisMetaStoreManager
> > > > > implementation, but eventually we can get to where the
> GrantManager,
> > > > > CredentialVendor, etc. can all be swapped out for different
> > > > implementations.
> > > > >
> > > > > Please take a look at the changes at
> > > > >
> > > >
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > > > and let me know what you think.
> > > > >
> > > > > Mike
> > > > >
> > > > > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> > collado.mike@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I added a table to the README with the differences that were
> called
> > > > out. I
> > > > > > added some details that I think are worth understanding better.
> > E.g.,
> > > > the
> > > > > > Json layout we added has specific custom functionality we wanted
> > for
> > > > > > supporting key/value pairs and the micrometer annotation was
> added
> > for
> > > > some
> > > > > > custom support we wanted aside from what is supported with the
> > default
> > > > > > dropwizard metrics support.
> > > > > >
> > > > > >
> > > > > >
> > > >
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > > > >
> > > > > > I think it would be great if others could add to the table with
> > > > features
> > > > > > that would be impacted by the migration and to call out the level
> > of
> > > > effort
> > > > > > in both dropwizard and quarkus.
> > > > > >
> > > > > > Another possible consideration would be upgrading our Dropwizard
> > > > > > dependency to the latest development version. It may be the case
> > that
> > > > doing
> > > > > > so would address some of the targeted features with less effort
> in
> > > > > > migrating.
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> > jb@nanthrax.net>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> Following the last community sync meeting, we create a branch
> > > > > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > > > >>
> > > > > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > > > >>
> > > > > >> We still have work to do, but you can already take a look and
> > > > > >> experiment (in the polaris-service-quarkus module).
> > > > > >>
> > > > > >> We added a README.md file to:
> > > > > >> 1. Highlight the main differences (in terms of code) between
> > > > > >> Dropwizard and Quarkus
> > > > > >> 2. To build and run Polaris powered by Quarkus
> > > > > >> 3. The list of TODO items
> > > > > >>
> > > > > >>
> > > > > >>
> > > >
> >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > > > >>
> > > > > >> If anyone wants to contribute on the branch before creating the
> > PR,
> > > > > >> please let me know, I will add you as contributor on the branch.
> > > > > >>
> > > > > >> Any comments or questions are welcome !
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Regards
> > > > > >> JB
> > > > > >>
> > > > > >
> > > >
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Michael Collado <co...@gmail.com>.
> Although, I wonder if this custom
context isn't identical to the standard request context? IOW, is it
possible for the Polaris context to change during an HTTP call to Polaris
API? I don't think so, which leads me to think that, at least in Quarkus,
we'd use the standard @RequestScoped annotation for passing things like
this.
TBH, I'm a bit fuzzy on the details of the custom scopes in HK2, but the
intention of the @RealmContext is to support beans that are scoped to the
realm, not the request. E.g., the MetaStoreManagerFactory maintains a map
of realm-specific PolarisMetaStoreManagers that are reused for every
request for a realm. The scope is intended to be larger
than @RequestScoped, but smaller than @ApplicationScoped. This is
especially important for the EntityCache, which maintains a cache of the
versioned entities and allows reuse of common entities (e.g., Catalogs and
PrincipalRoles) without having to fetch them for every request. I _think_
what I did with @RealmContext is to maintain a map of reusable beans that
are not recreated for each request (the context itself is a singleton).
> And to circle back to DI, whether this common module would contain classes
annotated with Jakarta CDI annotations is imho up for debate. This would
simplify things for sure, but on the flip side, would require both runtimes
to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
beans). I think that this is going to be too restrictive. An alternative
would be to keep the common classes clean of CDI annotations and let the
runtime modules create subclasses annotated with their own annotations.
I would _really_ prefer to avoid that if possible. The whole argument for
using standardized annotations is that they can be used by multiple
frameworks without being tied to any one. Personally, I'm not married to
the @Named annotation or any of the specifics I used to get my prototype to
work. If there is a more generalized annotation or alternative to
specifying named instances of contracts, I'm all for it.
Andrew Guterman <https://github.com/andrew4699> has been working on making
the target test environment more pluggable (see
https://github.com/apache/polaris/pull/391 ). Maybe we can get his input on
how to structure the tests in a way that they can hit the target runtime
without making them framework specific. This is super useful for, e.g.,
testing in a CD pipeline where you might hit a docker image or k8s
deployment to validate behavior before deploying to prod. I definitely want
to avoid duplicating test classes, as that's pretty much guaranteed to
ensure that something gets broken and is uncaught because we lacked test
coverage.
Mike
On Thu, Nov 14, 2024 at 7:21 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi
>
> Using @Named is fine with Quarkus. We can also create custom qualifier
> (with Literal): named is actually a qualifier :)
>
> I think we discussed about the plan, and to my understanding:
> 1. I open the Quarkus PR (as draft) to keep what we did
> 2. We improve the current structure:
> 2.1. polaris-core should not depend to dropwizard (remove discoverable
> here)
> 2.2. we create polaris-service-common containing all the beans,
> without dependency to Dropwizard (or Quarkus). Related to that, itests
> should go into polaris-service-itest.
> 2.2. we create polaris-service-dropwizard where we take
> polaris-service-common and do the beans injection with Dropwizard.
> (2.3). we can create polaris-service-quarkus where we take
> polaris-service-common and do the beans injection with Quarkus
>
> My standpoint is still the same: if the "clean refactoring" is a good
> thing, I think it's not possible to have both
> polaris-service-dropwizard and polaris-service-quarkus, for three
> reasons:
> - it doesn't bring any benefits for the users, only confusion (which
> service should I use)
> - it will be almost impossible to maintain (in sync)
> - it will prevent to leverage extensions provided (especially by Quarkus)
>
> So, my proposal is to move forward on the refactoring (we have a
> consensus here).
> I will work with Michael to identify exactly the gaps between
> polaris-service-dropwizard and polaris-service-quarkus (configuration,
> extensions, ...).
>
> Thoughts ?
>
> Regards
> JB
>
> On Wed, Nov 13, 2024 at 8:13 PM Alex Dutra
> <al...@dremio.com.invalid> wrote:
> >
> > Hi Michael, hi all,
> >
> > I had a look at your branch and am very pleased to see that it was
> > relatively easy to get DI working with DW + HK2. Good news!
> >
> > Digging a bit into the details, I see that most beans are annotated with
> > @Named now. This is probably OK for Quarkus too, although there are other
> > techniques that allow to select a specific implementation at runtime.
> >
> > I also think that the @RealmContext annotation is very smart, and
> resonates
> > with how I handled that in Quarkus. Although, I wonder if this custom
> > context isn't identical to the standard request context? IOW, is it
> > possible for the Polaris context to change during an HTTP call to Polaris
> > API? I don't think so, which leads me to think that, at least in Quarkus,
> > we'd use the standard @RequestScoped annotation for passing things like
> > this.
> >
> > But now taking a step back and looking at the big picture, I think that
> > what we need now is a plan forward. Given that many of us voiced concerns
> > around an all-or-nothing switch from DW to Quarkus (quite understandably
> > so), and assuming that there is agreement to maintain both runtimes (DW
> and
> > Quarkus) for some time, I'd like to suggest the phased approach below:
> >
> > 1. Remove the dependency to DW from polaris-core – there is a PR for
> > that already: https://github.com/apache/polaris/pull/435
> > 2. Refactor polaris-service into 2 modules: e.g.
> "polaris-service-common"
> > and "polaris-service-dropwizard" – I believe Dmitri is already working
> > on this
> > 3. Introduce DI + HK2 in polaris-service-dropwizard – that would come
> > from your branch basically
> > 4. Introduce polaris-service-quarkus – that would come mostly from
> JB's
> > branch
> >
> > Regarding step #2, it seems we can easily put 80 to 90% of the actual
> > polaris-service production classes into the common module. The dropwizard
> > runtime module would contain just a handful of DW-specific classes.
> >
> > The biggest issue to solve with step #2 is imho test classes. Quarkus and
> > DW have diverse requirements for tests, but we may still be able to
> > refactor things using abstract classes or extensions, and reuse most of
> > them. In any case, I think that if, at least at the beginning, test
> classes
> > are duplicated in each runtime module, that's acceptable until we figure
> > out a better way.
> >
> > And to circle back to DI, whether this common module would contain
> classes
> > annotated with Jakarta CDI annotations is imho up for debate. This would
> > simplify things for sure, but on the flip side, would require both
> runtimes
> > to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
> > beans). I think that this is going to be too restrictive. An alternative
> > would be to keep the common classes clean of CDI annotations and let the
> > runtime modules create subclasses annotated with their own annotations.
> >
> > And back to the @RealmContext topic, that's the other big aspect that we
> > will need to solve (maybe step #5?): CallContext is heavily reliant on
> > ThreadLocal, and that is not very CDI-friendly, as you've seen. For now,
> I
> > did a patchy workaround for Quarkus. But we'd need to refactor that code
> > cleanly, either by creating a custom context as you did, or by making
> > CallContext a request-scoped bean that is evaluated for each HTTP
> request.
> > Either way, it then needs to be propagated to other threads (e.g. in
> > TaskExecutorImpl); for Quarkus, this will require using proper context
> > propagation <https://quarkus.io/guides/context-propagation> (for DW,
> maybe
> > ThreadLocal is fine here). And since CallContext is in the core module,
> we
> > will likely need to remove the static methods getCurrentContext() and
> > setCurrentContext() and retrieve the context only by scope resolution
> > (which, in turn, may use ThreadLocal under the hood if that's appropriate
> > for DW).
> >
> > If everybody is on board with the plan above, how about we create GH
> issues
> > for each step, and maybe even a GH Project to track progress on this
> front
> > and split the effort?
> >
> > Thanks,
> >
> > Alex
> >
> >
> > On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi Michael
> > >
> > > Thanks for the update !
> > >
> > > As discussed during the last community meeting:
> > > 1. I will open the draft PR with Quarkus in the meantime
> > > 2. I will take a look on your branch powered by Glassfish HK2 CDI
> > > (give me a few days as I was travelling this week :))
> > >
> > > Thanks !
> > > Regards
> > >
> > > On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <collado.mike@gmail.com
> >
> > > wrote:
> > > >
> > > > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> > > impl.
> > > > I only spent yesterday and today on this, so it's not complete, but
> it is
> > > > functional and the tests pass :)
> > > >
> > > > I took a lot of the common ideas from the Quarkus branch (e.g.,
> deleted
> > > all
> > > > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> > > > figured out how to use the Dropwizard Yaml to specify the
> > > > implementation of, e.g. the Authenticator and the
> > > MetaStoreManagerFactory,
> > > > but have the instances managed and injectable by HK2. The goal there
> was
> > > to
> > > > just keep the existing configuration format, but change the impl
> under
> > > the
> > > > hood. I'm not married to the idea and I'm interested to see if the
> > > > jakarta.enterprise.inject.* annotations/interfaces that are used in
> the
> > > > Quarkus branch can make this simpler. However, I do think it would
> ideal
> > > if
> > > > we can get it working with the existing Yaml configuration, at least
> in
> > > the
> > > > short term.
> > > >
> > > > I did add support for a @RealmScope annotation to support restricting
> > > items
> > > > to a given realm, such as the EntityCache and the
> PolarisGrantManager.
> > > This
> > > > allowed me to do things like hide the grant lookups from the
> EntityCache
> > > so
> > > > that the Resolver doesn't have to pass around the
> ResolvedPolarisEntity,
> > > > but instead the grants are found from the cache without making it
> overt
> > > in
> > > > the PolarisAuthorizer API. This was one of my original goals with
> > > breaking
> > > > up the PolarisMetaStoreManager API into multiple interfaces. Right
> now,
> > > > everything still ties back to the configured PolarisMetaStoreManager
> > > > implementation, but eventually we can get to where the GrantManager,
> > > > CredentialVendor, etc. can all be swapped out for different
> > > implementations.
> > > >
> > > > Please take a look at the changes at
> > > >
> > >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > > and let me know what you think.
> > > >
> > > > Mike
> > > >
> > > > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
> collado.mike@gmail.com>
> > > > wrote:
> > > >
> > > > > I added a table to the README with the differences that were called
> > > out. I
> > > > > added some details that I think are worth understanding better.
> E.g.,
> > > the
> > > > > Json layout we added has specific custom functionality we wanted
> for
> > > > > supporting key/value pairs and the micrometer annotation was added
> for
> > > some
> > > > > custom support we wanted aside from what is supported with the
> default
> > > > > dropwizard metrics support.
> > > > >
> > > > >
> > > > >
> > >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > > >
> > > > > I think it would be great if others could add to the table with
> > > features
> > > > > that would be impacted by the migration and to call out the level
> of
> > > effort
> > > > > in both dropwizard and quarkus.
> > > > >
> > > > > Another possible consideration would be upgrading our Dropwizard
> > > > > dependency to the latest development version. It may be the case
> that
> > > doing
> > > > > so would address some of the targeted features with less effort in
> > > > > migrating.
> > > > >
> > > > > Mike
> > > > >
> > > > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> Following the last community sync meeting, we create a branch
> > > > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > > >>
> > > > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > > >>
> > > > >> We still have work to do, but you can already take a look and
> > > > >> experiment (in the polaris-service-quarkus module).
> > > > >>
> > > > >> We added a README.md file to:
> > > > >> 1. Highlight the main differences (in terms of code) between
> > > > >> Dropwizard and Quarkus
> > > > >> 2. To build and run Polaris powered by Quarkus
> > > > >> 3. The list of TODO items
> > > > >>
> > > > >>
> > > > >>
> > >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > > >>
> > > > >> If anyone wants to contribute on the branch before creating the
> PR,
> > > > >> please let me know, I will add you as contributor on the branch.
> > > > >>
> > > > >> Any comments or questions are welcome !
> > > > >>
> > > > >> Thanks,
> > > > >> Regards
> > > > >> JB
> > > > >>
> > > > >
> > >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
Using @Named is fine with Quarkus. We can also create custom qualifier
(with Literal): named is actually a qualifier :)
I think we discussed about the plan, and to my understanding:
1. I open the Quarkus PR (as draft) to keep what we did
2. We improve the current structure:
2.1. polaris-core should not depend to dropwizard (remove discoverable here)
2.2. we create polaris-service-common containing all the beans,
without dependency to Dropwizard (or Quarkus). Related to that, itests
should go into polaris-service-itest.
2.2. we create polaris-service-dropwizard where we take
polaris-service-common and do the beans injection with Dropwizard.
(2.3). we can create polaris-service-quarkus where we take
polaris-service-common and do the beans injection with Quarkus
My standpoint is still the same: if the "clean refactoring" is a good
thing, I think it's not possible to have both
polaris-service-dropwizard and polaris-service-quarkus, for three
reasons:
- it doesn't bring any benefits for the users, only confusion (which
service should I use)
- it will be almost impossible to maintain (in sync)
- it will prevent to leverage extensions provided (especially by Quarkus)
So, my proposal is to move forward on the refactoring (we have a
consensus here).
I will work with Michael to identify exactly the gaps between
polaris-service-dropwizard and polaris-service-quarkus (configuration,
extensions, ...).
Thoughts ?
Regards
JB
On Wed, Nov 13, 2024 at 8:13 PM Alex Dutra
<al...@dremio.com.invalid> wrote:
>
> Hi Michael, hi all,
>
> I had a look at your branch and am very pleased to see that it was
> relatively easy to get DI working with DW + HK2. Good news!
>
> Digging a bit into the details, I see that most beans are annotated with
> @Named now. This is probably OK for Quarkus too, although there are other
> techniques that allow to select a specific implementation at runtime.
>
> I also think that the @RealmContext annotation is very smart, and resonates
> with how I handled that in Quarkus. Although, I wonder if this custom
> context isn't identical to the standard request context? IOW, is it
> possible for the Polaris context to change during an HTTP call to Polaris
> API? I don't think so, which leads me to think that, at least in Quarkus,
> we'd use the standard @RequestScoped annotation for passing things like
> this.
>
> But now taking a step back and looking at the big picture, I think that
> what we need now is a plan forward. Given that many of us voiced concerns
> around an all-or-nothing switch from DW to Quarkus (quite understandably
> so), and assuming that there is agreement to maintain both runtimes (DW and
> Quarkus) for some time, I'd like to suggest the phased approach below:
>
> 1. Remove the dependency to DW from polaris-core – there is a PR for
> that already: https://github.com/apache/polaris/pull/435
> 2. Refactor polaris-service into 2 modules: e.g. "polaris-service-common"
> and "polaris-service-dropwizard" – I believe Dmitri is already working
> on this
> 3. Introduce DI + HK2 in polaris-service-dropwizard – that would come
> from your branch basically
> 4. Introduce polaris-service-quarkus – that would come mostly from JB's
> branch
>
> Regarding step #2, it seems we can easily put 80 to 90% of the actual
> polaris-service production classes into the common module. The dropwizard
> runtime module would contain just a handful of DW-specific classes.
>
> The biggest issue to solve with step #2 is imho test classes. Quarkus and
> DW have diverse requirements for tests, but we may still be able to
> refactor things using abstract classes or extensions, and reuse most of
> them. In any case, I think that if, at least at the beginning, test classes
> are duplicated in each runtime module, that's acceptable until we figure
> out a better way.
>
> And to circle back to DI, whether this common module would contain classes
> annotated with Jakarta CDI annotations is imho up for debate. This would
> simplify things for sure, but on the flip side, would require both runtimes
> to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
> beans). I think that this is going to be too restrictive. An alternative
> would be to keep the common classes clean of CDI annotations and let the
> runtime modules create subclasses annotated with their own annotations.
>
> And back to the @RealmContext topic, that's the other big aspect that we
> will need to solve (maybe step #5?): CallContext is heavily reliant on
> ThreadLocal, and that is not very CDI-friendly, as you've seen. For now, I
> did a patchy workaround for Quarkus. But we'd need to refactor that code
> cleanly, either by creating a custom context as you did, or by making
> CallContext a request-scoped bean that is evaluated for each HTTP request.
> Either way, it then needs to be propagated to other threads (e.g. in
> TaskExecutorImpl); for Quarkus, this will require using proper context
> propagation <https://quarkus.io/guides/context-propagation> (for DW, maybe
> ThreadLocal is fine here). And since CallContext is in the core module, we
> will likely need to remove the static methods getCurrentContext() and
> setCurrentContext() and retrieve the context only by scope resolution
> (which, in turn, may use ThreadLocal under the hood if that's appropriate
> for DW).
>
> If everybody is on board with the plan above, how about we create GH issues
> for each step, and maybe even a GH Project to track progress on this front
> and split the effort?
>
> Thanks,
>
> Alex
>
>
> On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi Michael
> >
> > Thanks for the update !
> >
> > As discussed during the last community meeting:
> > 1. I will open the draft PR with Quarkus in the meantime
> > 2. I will take a look on your branch powered by Glassfish HK2 CDI
> > (give me a few days as I was travelling this week :))
> >
> > Thanks !
> > Regards
> >
> > On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <co...@gmail.com>
> > wrote:
> > >
> > > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> > impl.
> > > I only spent yesterday and today on this, so it's not complete, but it is
> > > functional and the tests pass :)
> > >
> > > I took a lot of the common ideas from the Quarkus branch (e.g., deleted
> > all
> > > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> > > figured out how to use the Dropwizard Yaml to specify the
> > > implementation of, e.g. the Authenticator and the
> > MetaStoreManagerFactory,
> > > but have the instances managed and injectable by HK2. The goal there was
> > to
> > > just keep the existing configuration format, but change the impl under
> > the
> > > hood. I'm not married to the idea and I'm interested to see if the
> > > jakarta.enterprise.inject.* annotations/interfaces that are used in the
> > > Quarkus branch can make this simpler. However, I do think it would ideal
> > if
> > > we can get it working with the existing Yaml configuration, at least in
> > the
> > > short term.
> > >
> > > I did add support for a @RealmScope annotation to support restricting
> > items
> > > to a given realm, such as the EntityCache and the PolarisGrantManager.
> > This
> > > allowed me to do things like hide the grant lookups from the EntityCache
> > so
> > > that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
> > > but instead the grants are found from the cache without making it overt
> > in
> > > the PolarisAuthorizer API. This was one of my original goals with
> > breaking
> > > up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> > > everything still ties back to the configured PolarisMetaStoreManager
> > > implementation, but eventually we can get to where the GrantManager,
> > > CredentialVendor, etc. can all be swapped out for different
> > implementations.
> > >
> > > Please take a look at the changes at
> > >
> > https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > and let me know what you think.
> > >
> > > Mike
> > >
> > > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
> > > wrote:
> > >
> > > > I added a table to the README with the differences that were called
> > out. I
> > > > added some details that I think are worth understanding better. E.g.,
> > the
> > > > Json layout we added has specific custom functionality we wanted for
> > > > supporting key/value pairs and the micrometer annotation was added for
> > some
> > > > custom support we wanted aside from what is supported with the default
> > > > dropwizard metrics support.
> > > >
> > > >
> > > >
> > https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > >
> > > > I think it would be great if others could add to the table with
> > features
> > > > that would be impacted by the migration and to call out the level of
> > effort
> > > > in both dropwizard and quarkus.
> > > >
> > > > Another possible consideration would be upgrading our Dropwizard
> > > > dependency to the latest development version. It may be the case that
> > doing
> > > > so would address some of the targeted features with less effort in
> > > > migrating.
> > > >
> > > > Mike
> > > >
> > > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> Following the last community sync meeting, we create a branch
> > > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > >>
> > > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > >>
> > > >> We still have work to do, but you can already take a look and
> > > >> experiment (in the polaris-service-quarkus module).
> > > >>
> > > >> We added a README.md file to:
> > > >> 1. Highlight the main differences (in terms of code) between
> > > >> Dropwizard and Quarkus
> > > >> 2. To build and run Polaris powered by Quarkus
> > > >> 3. The list of TODO items
> > > >>
> > > >>
> > > >>
> > https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > >>
> > > >> If anyone wants to contribute on the branch before creating the PR,
> > > >> please let me know, I will add you as contributor on the branch.
> > > >>
> > > >> Any comments or questions are welcome !
> > > >>
> > > >> Thanks,
> > > >> Regards
> > > >> JB
> > > >>
> > > >
> >
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
For that matter, it would be nice to reach a consensus on how to handle
root credentials during bootstrapping, as it will be critical to running
Polaris in isolated JVMs during integration tests.
So please review my latest changes in
https://github.com/apache/polaris/pull/422
Thanks,
Dmitri.
On Wed, Nov 13, 2024 at 4:28 PM Dmitri Bourlatchkov <
dmitri.bourlatchkov@dremio.com> wrote:
> To avoid confusion: I mean `polaris-test` is for the integration kind of
> tests that require some support from the environment, like starting a
> Polaris server.
>
> On Wed, Nov 13, 2024 at 4:27 PM Dmitri Bourlatchkov <
> dmitri.bourlatchkov@dremio.com> wrote:
>
>> Re: test classes - I'd propose to make a new module, e.g. `polaris-test`
>> to contain abstract (actually or in spirit) test code, which would then be
>> picked up by DW and Quarkus-specific actual test classes.
>>
>> I'm sure we can isolate env. details in JUnit5 extensions and inject
>> relevant access objects (e.g. REST clients / URLs / etc) in runtime.
>>
>> Cheers,
>> Dmitri.
>>
>> On Wed, Nov 13, 2024 at 2:13 PM Alex Dutra <al...@dremio.com.invalid>
>> wrote:
>>
>>> Hi Michael, hi all,
>>>
>>> I had a look at your branch and am very pleased to see that it was
>>> relatively easy to get DI working with DW + HK2. Good news!
>>>
>>> Digging a bit into the details, I see that most beans are annotated with
>>> @Named now. This is probably OK for Quarkus too, although there are other
>>> techniques that allow to select a specific implementation at runtime.
>>>
>>> I also think that the @RealmContext annotation is very smart, and
>>> resonates
>>> with how I handled that in Quarkus. Although, I wonder if this custom
>>> context isn't identical to the standard request context? IOW, is it
>>> possible for the Polaris context to change during an HTTP call to Polaris
>>> API? I don't think so, which leads me to think that, at least in Quarkus,
>>> we'd use the standard @RequestScoped annotation for passing things like
>>> this.
>>>
>>> But now taking a step back and looking at the big picture, I think that
>>> what we need now is a plan forward. Given that many of us voiced concerns
>>> around an all-or-nothing switch from DW to Quarkus (quite understandably
>>> so), and assuming that there is agreement to maintain both runtimes (DW
>>> and
>>> Quarkus) for some time, I'd like to suggest the phased approach below:
>>>
>>> 1. Remove the dependency to DW from polaris-core – there is a PR for
>>> that already: https://github.com/apache/polaris/pull/435
>>> 2. Refactor polaris-service into 2 modules: e.g.
>>> "polaris-service-common"
>>> and "polaris-service-dropwizard" – I believe Dmitri is already working
>>> on this
>>> 3. Introduce DI + HK2 in polaris-service-dropwizard – that would come
>>> from your branch basically
>>> 4. Introduce polaris-service-quarkus – that would come mostly from
>>> JB's
>>> branch
>>>
>>> Regarding step #2, it seems we can easily put 80 to 90% of the actual
>>> polaris-service production classes into the common module. The dropwizard
>>> runtime module would contain just a handful of DW-specific classes.
>>>
>>> The biggest issue to solve with step #2 is imho test classes. Quarkus and
>>> DW have diverse requirements for tests, but we may still be able to
>>> refactor things using abstract classes or extensions, and reuse most of
>>> them. In any case, I think that if, at least at the beginning, test
>>> classes
>>> are duplicated in each runtime module, that's acceptable until we figure
>>> out a better way.
>>>
>>> And to circle back to DI, whether this common module would contain
>>> classes
>>> annotated with Jakarta CDI annotations is imho up for debate. This would
>>> simplify things for sure, but on the flip side, would require both
>>> runtimes
>>> to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
>>> beans). I think that this is going to be too restrictive. An alternative
>>> would be to keep the common classes clean of CDI annotations and let the
>>> runtime modules create subclasses annotated with their own annotations.
>>>
>>> And back to the @RealmContext topic, that's the other big aspect that we
>>> will need to solve (maybe step #5?): CallContext is heavily reliant on
>>> ThreadLocal, and that is not very CDI-friendly, as you've seen. For now,
>>> I
>>> did a patchy workaround for Quarkus. But we'd need to refactor that code
>>> cleanly, either by creating a custom context as you did, or by making
>>> CallContext a request-scoped bean that is evaluated for each HTTP
>>> request.
>>> Either way, it then needs to be propagated to other threads (e.g. in
>>> TaskExecutorImpl); for Quarkus, this will require using proper context
>>> propagation <https://quarkus.io/guides/context-propagation> (for DW,
>>> maybe
>>> ThreadLocal is fine here). And since CallContext is in the core module,
>>> we
>>> will likely need to remove the static methods getCurrentContext() and
>>> setCurrentContext() and retrieve the context only by scope resolution
>>> (which, in turn, may use ThreadLocal under the hood if that's appropriate
>>> for DW).
>>>
>>> If everybody is on board with the plan above, how about we create GH
>>> issues
>>> for each step, and maybe even a GH Project to track progress on this
>>> front
>>> and split the effort?
>>>
>>> Thanks,
>>>
>>> Alex
>>>
>>>
>>> On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>> > Hi Michael
>>> >
>>> > Thanks for the update !
>>> >
>>> > As discussed during the last community meeting:
>>> > 1. I will open the draft PR with Quarkus in the meantime
>>> > 2. I will take a look on your branch powered by Glassfish HK2 CDI
>>> > (give me a few days as I was travelling this week :))
>>> >
>>> > Thanks !
>>> > Regards
>>> >
>>> > On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <collado.mike@gmail.com
>>> >
>>> > wrote:
>>> > >
>>> > > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
>>> > impl.
>>> > > I only spent yesterday and today on this, so it's not complete, but
>>> it is
>>> > > functional and the tests pass :)
>>> > >
>>> > > I took a lot of the common ideas from the Quarkus branch (e.g.,
>>> deleted
>>> > all
>>> > > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
>>> > > figured out how to use the Dropwizard Yaml to specify the
>>> > > implementation of, e.g. the Authenticator and the
>>> > MetaStoreManagerFactory,
>>> > > but have the instances managed and injectable by HK2. The goal there
>>> was
>>> > to
>>> > > just keep the existing configuration format, but change the impl
>>> under
>>> > the
>>> > > hood. I'm not married to the idea and I'm interested to see if the
>>> > > jakarta.enterprise.inject.* annotations/interfaces that are used in
>>> the
>>> > > Quarkus branch can make this simpler. However, I do think it would
>>> ideal
>>> > if
>>> > > we can get it working with the existing Yaml configuration, at least
>>> in
>>> > the
>>> > > short term.
>>> > >
>>> > > I did add support for a @RealmScope annotation to support restricting
>>> > items
>>> > > to a given realm, such as the EntityCache and the
>>> PolarisGrantManager.
>>> > This
>>> > > allowed me to do things like hide the grant lookups from the
>>> EntityCache
>>> > so
>>> > > that the Resolver doesn't have to pass around the
>>> ResolvedPolarisEntity,
>>> > > but instead the grants are found from the cache without making it
>>> overt
>>> > in
>>> > > the PolarisAuthorizer API. This was one of my original goals with
>>> > breaking
>>> > > up the PolarisMetaStoreManager API into multiple interfaces. Right
>>> now,
>>> > > everything still ties back to the configured PolarisMetaStoreManager
>>> > > implementation, but eventually we can get to where the GrantManager,
>>> > > CredentialVendor, etc. can all be swapped out for different
>>> > implementations.
>>> > >
>>> > > Please take a look at the changes at
>>> > >
>>> >
>>> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
>>> > > and let me know what you think.
>>> > >
>>> > > Mike
>>> > >
>>> > > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
>>> collado.mike@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > I added a table to the README with the differences that were called
>>> > out. I
>>> > > > added some details that I think are worth understanding better.
>>> E.g.,
>>> > the
>>> > > > Json layout we added has specific custom functionality we wanted
>>> for
>>> > > > supporting key/value pairs and the micrometer annotation was added
>>> for
>>> > some
>>> > > > custom support we wanted aside from what is supported with the
>>> default
>>> > > > dropwizard metrics support.
>>> > > >
>>> > > >
>>> > > >
>>> >
>>> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
>>> > > >
>>> > > > I think it would be great if others could add to the table with
>>> > features
>>> > > > that would be impacted by the migration and to call out the level
>>> of
>>> > effort
>>> > > > in both dropwizard and quarkus.
>>> > > >
>>> > > > Another possible consideration would be upgrading our Dropwizard
>>> > > > dependency to the latest development version. It may be the case
>>> that
>>> > doing
>>> > > > so would address some of the targeted features with less effort in
>>> > > > migrating.
>>> > > >
>>> > > > Mike
>>> > > >
>>> > > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
>>> jb@nanthrax.net>
>>> > > > wrote:
>>> > > >
>>> > > >> Hi folks,
>>> > > >>
>>> > > >> Following the last community sync meeting, we create a branch
>>> > > >> demonstrating use of Quarkus to powered Apache Polaris:
>>> > > >>
>>> > > >> https://github.com/jbonofre/polaris/tree/QUARKUS
>>> > > >>
>>> > > >> We still have work to do, but you can already take a look and
>>> > > >> experiment (in the polaris-service-quarkus module).
>>> > > >>
>>> > > >> We added a README.md file to:
>>> > > >> 1. Highlight the main differences (in terms of code) between
>>> > > >> Dropwizard and Quarkus
>>> > > >> 2. To build and run Polaris powered by Quarkus
>>> > > >> 3. The list of TODO items
>>> > > >>
>>> > > >>
>>> > > >>
>>> >
>>> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
>>> > > >>
>>> > > >> If anyone wants to contribute on the branch before creating the
>>> PR,
>>> > > >> please let me know, I will add you as contributor on the branch.
>>> > > >>
>>> > > >> Any comments or questions are welcome !
>>> > > >>
>>> > > >> Thanks,
>>> > > >> Regards
>>> > > >> JB
>>> > > >>
>>> > > >
>>> >
>>>
>>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
To avoid confusion: I mean `polaris-test` is for the integration kind of
tests that require some support from the environment, like starting a
Polaris server.
On Wed, Nov 13, 2024 at 4:27 PM Dmitri Bourlatchkov <
dmitri.bourlatchkov@dremio.com> wrote:
> Re: test classes - I'd propose to make a new module, e.g. `polaris-test`
> to contain abstract (actually or in spirit) test code, which would then be
> picked up by DW and Quarkus-specific actual test classes.
>
> I'm sure we can isolate env. details in JUnit5 extensions and inject
> relevant access objects (e.g. REST clients / URLs / etc) in runtime.
>
> Cheers,
> Dmitri.
>
> On Wed, Nov 13, 2024 at 2:13 PM Alex Dutra <al...@dremio.com.invalid>
> wrote:
>
>> Hi Michael, hi all,
>>
>> I had a look at your branch and am very pleased to see that it was
>> relatively easy to get DI working with DW + HK2. Good news!
>>
>> Digging a bit into the details, I see that most beans are annotated with
>> @Named now. This is probably OK for Quarkus too, although there are other
>> techniques that allow to select a specific implementation at runtime.
>>
>> I also think that the @RealmContext annotation is very smart, and
>> resonates
>> with how I handled that in Quarkus. Although, I wonder if this custom
>> context isn't identical to the standard request context? IOW, is it
>> possible for the Polaris context to change during an HTTP call to Polaris
>> API? I don't think so, which leads me to think that, at least in Quarkus,
>> we'd use the standard @RequestScoped annotation for passing things like
>> this.
>>
>> But now taking a step back and looking at the big picture, I think that
>> what we need now is a plan forward. Given that many of us voiced concerns
>> around an all-or-nothing switch from DW to Quarkus (quite understandably
>> so), and assuming that there is agreement to maintain both runtimes (DW
>> and
>> Quarkus) for some time, I'd like to suggest the phased approach below:
>>
>> 1. Remove the dependency to DW from polaris-core – there is a PR for
>> that already: https://github.com/apache/polaris/pull/435
>> 2. Refactor polaris-service into 2 modules: e.g.
>> "polaris-service-common"
>> and "polaris-service-dropwizard" – I believe Dmitri is already working
>> on this
>> 3. Introduce DI + HK2 in polaris-service-dropwizard – that would come
>> from your branch basically
>> 4. Introduce polaris-service-quarkus – that would come mostly from JB's
>> branch
>>
>> Regarding step #2, it seems we can easily put 80 to 90% of the actual
>> polaris-service production classes into the common module. The dropwizard
>> runtime module would contain just a handful of DW-specific classes.
>>
>> The biggest issue to solve with step #2 is imho test classes. Quarkus and
>> DW have diverse requirements for tests, but we may still be able to
>> refactor things using abstract classes or extensions, and reuse most of
>> them. In any case, I think that if, at least at the beginning, test
>> classes
>> are duplicated in each runtime module, that's acceptable until we figure
>> out a better way.
>>
>> And to circle back to DI, whether this common module would contain classes
>> annotated with Jakarta CDI annotations is imho up for debate. This would
>> simplify things for sure, but on the flip side, would require both
>> runtimes
>> to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
>> beans). I think that this is going to be too restrictive. An alternative
>> would be to keep the common classes clean of CDI annotations and let the
>> runtime modules create subclasses annotated with their own annotations.
>>
>> And back to the @RealmContext topic, that's the other big aspect that we
>> will need to solve (maybe step #5?): CallContext is heavily reliant on
>> ThreadLocal, and that is not very CDI-friendly, as you've seen. For now, I
>> did a patchy workaround for Quarkus. But we'd need to refactor that code
>> cleanly, either by creating a custom context as you did, or by making
>> CallContext a request-scoped bean that is evaluated for each HTTP request.
>> Either way, it then needs to be propagated to other threads (e.g. in
>> TaskExecutorImpl); for Quarkus, this will require using proper context
>> propagation <https://quarkus.io/guides/context-propagation> (for DW,
>> maybe
>> ThreadLocal is fine here). And since CallContext is in the core module, we
>> will likely need to remove the static methods getCurrentContext() and
>> setCurrentContext() and retrieve the context only by scope resolution
>> (which, in turn, may use ThreadLocal under the hood if that's appropriate
>> for DW).
>>
>> If everybody is on board with the plan above, how about we create GH
>> issues
>> for each step, and maybe even a GH Project to track progress on this front
>> and split the effort?
>>
>> Thanks,
>>
>> Alex
>>
>>
>> On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>> > Hi Michael
>> >
>> > Thanks for the update !
>> >
>> > As discussed during the last community meeting:
>> > 1. I will open the draft PR with Quarkus in the meantime
>> > 2. I will take a look on your branch powered by Glassfish HK2 CDI
>> > (give me a few days as I was travelling this week :))
>> >
>> > Thanks !
>> > Regards
>> >
>> > On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <co...@gmail.com>
>> > wrote:
>> > >
>> > > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
>> > impl.
>> > > I only spent yesterday and today on this, so it's not complete, but
>> it is
>> > > functional and the tests pass :)
>> > >
>> > > I took a lot of the common ideas from the Quarkus branch (e.g.,
>> deleted
>> > all
>> > > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
>> > > figured out how to use the Dropwizard Yaml to specify the
>> > > implementation of, e.g. the Authenticator and the
>> > MetaStoreManagerFactory,
>> > > but have the instances managed and injectable by HK2. The goal there
>> was
>> > to
>> > > just keep the existing configuration format, but change the impl under
>> > the
>> > > hood. I'm not married to the idea and I'm interested to see if the
>> > > jakarta.enterprise.inject.* annotations/interfaces that are used in
>> the
>> > > Quarkus branch can make this simpler. However, I do think it would
>> ideal
>> > if
>> > > we can get it working with the existing Yaml configuration, at least
>> in
>> > the
>> > > short term.
>> > >
>> > > I did add support for a @RealmScope annotation to support restricting
>> > items
>> > > to a given realm, such as the EntityCache and the PolarisGrantManager.
>> > This
>> > > allowed me to do things like hide the grant lookups from the
>> EntityCache
>> > so
>> > > that the Resolver doesn't have to pass around the
>> ResolvedPolarisEntity,
>> > > but instead the grants are found from the cache without making it
>> overt
>> > in
>> > > the PolarisAuthorizer API. This was one of my original goals with
>> > breaking
>> > > up the PolarisMetaStoreManager API into multiple interfaces. Right
>> now,
>> > > everything still ties back to the configured PolarisMetaStoreManager
>> > > implementation, but eventually we can get to where the GrantManager,
>> > > CredentialVendor, etc. can all be swapped out for different
>> > implementations.
>> > >
>> > > Please take a look at the changes at
>> > >
>> >
>> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
>> > > and let me know what you think.
>> > >
>> > > Mike
>> > >
>> > > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <
>> collado.mike@gmail.com>
>> > > wrote:
>> > >
>> > > > I added a table to the README with the differences that were called
>> > out. I
>> > > > added some details that I think are worth understanding better.
>> E.g.,
>> > the
>> > > > Json layout we added has specific custom functionality we wanted for
>> > > > supporting key/value pairs and the micrometer annotation was added
>> for
>> > some
>> > > > custom support we wanted aside from what is supported with the
>> default
>> > > > dropwizard metrics support.
>> > > >
>> > > >
>> > > >
>> >
>> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
>> > > >
>> > > > I think it would be great if others could add to the table with
>> > features
>> > > > that would be impacted by the migration and to call out the level of
>> > effort
>> > > > in both dropwizard and quarkus.
>> > > >
>> > > > Another possible consideration would be upgrading our Dropwizard
>> > > > dependency to the latest development version. It may be the case
>> that
>> > doing
>> > > > so would address some of the targeted features with less effort in
>> > > > migrating.
>> > > >
>> > > > Mike
>> > > >
>> > > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
>> jb@nanthrax.net>
>> > > > wrote:
>> > > >
>> > > >> Hi folks,
>> > > >>
>> > > >> Following the last community sync meeting, we create a branch
>> > > >> demonstrating use of Quarkus to powered Apache Polaris:
>> > > >>
>> > > >> https://github.com/jbonofre/polaris/tree/QUARKUS
>> > > >>
>> > > >> We still have work to do, but you can already take a look and
>> > > >> experiment (in the polaris-service-quarkus module).
>> > > >>
>> > > >> We added a README.md file to:
>> > > >> 1. Highlight the main differences (in terms of code) between
>> > > >> Dropwizard and Quarkus
>> > > >> 2. To build and run Polaris powered by Quarkus
>> > > >> 3. The list of TODO items
>> > > >>
>> > > >>
>> > > >>
>> >
>> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
>> > > >>
>> > > >> If anyone wants to contribute on the branch before creating the PR,
>> > > >> please let me know, I will add you as contributor on the branch.
>> > > >>
>> > > >> Any comments or questions are welcome !
>> > > >>
>> > > >> Thanks,
>> > > >> Regards
>> > > >> JB
>> > > >>
>> > > >
>> >
>>
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
Re: test classes - I'd propose to make a new module, e.g. `polaris-test` to
contain abstract (actually or in spirit) test code, which would then be
picked up by DW and Quarkus-specific actual test classes.
I'm sure we can isolate env. details in JUnit5 extensions and inject
relevant access objects (e.g. REST clients / URLs / etc) in runtime.
Cheers,
Dmitri.
On Wed, Nov 13, 2024 at 2:13 PM Alex Dutra <al...@dremio.com.invalid>
wrote:
> Hi Michael, hi all,
>
> I had a look at your branch and am very pleased to see that it was
> relatively easy to get DI working with DW + HK2. Good news!
>
> Digging a bit into the details, I see that most beans are annotated with
> @Named now. This is probably OK for Quarkus too, although there are other
> techniques that allow to select a specific implementation at runtime.
>
> I also think that the @RealmContext annotation is very smart, and resonates
> with how I handled that in Quarkus. Although, I wonder if this custom
> context isn't identical to the standard request context? IOW, is it
> possible for the Polaris context to change during an HTTP call to Polaris
> API? I don't think so, which leads me to think that, at least in Quarkus,
> we'd use the standard @RequestScoped annotation for passing things like
> this.
>
> But now taking a step back and looking at the big picture, I think that
> what we need now is a plan forward. Given that many of us voiced concerns
> around an all-or-nothing switch from DW to Quarkus (quite understandably
> so), and assuming that there is agreement to maintain both runtimes (DW and
> Quarkus) for some time, I'd like to suggest the phased approach below:
>
> 1. Remove the dependency to DW from polaris-core – there is a PR for
> that already: https://github.com/apache/polaris/pull/435
> 2. Refactor polaris-service into 2 modules: e.g.
> "polaris-service-common"
> and "polaris-service-dropwizard" – I believe Dmitri is already working
> on this
> 3. Introduce DI + HK2 in polaris-service-dropwizard – that would come
> from your branch basically
> 4. Introduce polaris-service-quarkus – that would come mostly from JB's
> branch
>
> Regarding step #2, it seems we can easily put 80 to 90% of the actual
> polaris-service production classes into the common module. The dropwizard
> runtime module would contain just a handful of DW-specific classes.
>
> The biggest issue to solve with step #2 is imho test classes. Quarkus and
> DW have diverse requirements for tests, but we may still be able to
> refactor things using abstract classes or extensions, and reuse most of
> them. In any case, I think that if, at least at the beginning, test classes
> are duplicated in each runtime module, that's acceptable until we figure
> out a better way.
>
> And to circle back to DI, whether this common module would contain classes
> annotated with Jakarta CDI annotations is imho up for debate. This would
> simplify things for sure, but on the flip side, would require both runtimes
> to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
> beans). I think that this is going to be too restrictive. An alternative
> would be to keep the common classes clean of CDI annotations and let the
> runtime modules create subclasses annotated with their own annotations.
>
> And back to the @RealmContext topic, that's the other big aspect that we
> will need to solve (maybe step #5?): CallContext is heavily reliant on
> ThreadLocal, and that is not very CDI-friendly, as you've seen. For now, I
> did a patchy workaround for Quarkus. But we'd need to refactor that code
> cleanly, either by creating a custom context as you did, or by making
> CallContext a request-scoped bean that is evaluated for each HTTP request.
> Either way, it then needs to be propagated to other threads (e.g. in
> TaskExecutorImpl); for Quarkus, this will require using proper context
> propagation <https://quarkus.io/guides/context-propagation> (for DW, maybe
> ThreadLocal is fine here). And since CallContext is in the core module, we
> will likely need to remove the static methods getCurrentContext() and
> setCurrentContext() and retrieve the context only by scope resolution
> (which, in turn, may use ThreadLocal under the hood if that's appropriate
> for DW).
>
> If everybody is on board with the plan above, how about we create GH issues
> for each step, and maybe even a GH Project to track progress on this front
> and split the effort?
>
> Thanks,
>
> Alex
>
>
> On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi Michael
> >
> > Thanks for the update !
> >
> > As discussed during the last community meeting:
> > 1. I will open the draft PR with Quarkus in the meantime
> > 2. I will take a look on your branch powered by Glassfish HK2 CDI
> > (give me a few days as I was travelling this week :))
> >
> > Thanks !
> > Regards
> >
> > On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <co...@gmail.com>
> > wrote:
> > >
> > > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> > impl.
> > > I only spent yesterday and today on this, so it's not complete, but it
> is
> > > functional and the tests pass :)
> > >
> > > I took a lot of the common ideas from the Quarkus branch (e.g., deleted
> > all
> > > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> > > figured out how to use the Dropwizard Yaml to specify the
> > > implementation of, e.g. the Authenticator and the
> > MetaStoreManagerFactory,
> > > but have the instances managed and injectable by HK2. The goal there
> was
> > to
> > > just keep the existing configuration format, but change the impl under
> > the
> > > hood. I'm not married to the idea and I'm interested to see if the
> > > jakarta.enterprise.inject.* annotations/interfaces that are used in the
> > > Quarkus branch can make this simpler. However, I do think it would
> ideal
> > if
> > > we can get it working with the existing Yaml configuration, at least in
> > the
> > > short term.
> > >
> > > I did add support for a @RealmScope annotation to support restricting
> > items
> > > to a given realm, such as the EntityCache and the PolarisGrantManager.
> > This
> > > allowed me to do things like hide the grant lookups from the
> EntityCache
> > so
> > > that the Resolver doesn't have to pass around the
> ResolvedPolarisEntity,
> > > but instead the grants are found from the cache without making it overt
> > in
> > > the PolarisAuthorizer API. This was one of my original goals with
> > breaking
> > > up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> > > everything still ties back to the configured PolarisMetaStoreManager
> > > implementation, but eventually we can get to where the GrantManager,
> > > CredentialVendor, etc. can all be swapped out for different
> > implementations.
> > >
> > > Please take a look at the changes at
> > >
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > and let me know what you think.
> > >
> > > Mike
> > >
> > > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <collado.mike@gmail.com
> >
> > > wrote:
> > >
> > > > I added a table to the README with the differences that were called
> > out. I
> > > > added some details that I think are worth understanding better. E.g.,
> > the
> > > > Json layout we added has specific custom functionality we wanted for
> > > > supporting key/value pairs and the micrometer annotation was added
> for
> > some
> > > > custom support we wanted aside from what is supported with the
> default
> > > > dropwizard metrics support.
> > > >
> > > >
> > > >
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > >
> > > > I think it would be great if others could add to the table with
> > features
> > > > that would be impacted by the migration and to call out the level of
> > effort
> > > > in both dropwizard and quarkus.
> > > >
> > > > Another possible consideration would be upgrading our Dropwizard
> > > > dependency to the latest development version. It may be the case that
> > doing
> > > > so would address some of the targeted features with less effort in
> > > > migrating.
> > > >
> > > > Mike
> > > >
> > > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> Following the last community sync meeting, we create a branch
> > > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > >>
> > > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > >>
> > > >> We still have work to do, but you can already take a look and
> > > >> experiment (in the polaris-service-quarkus module).
> > > >>
> > > >> We added a README.md file to:
> > > >> 1. Highlight the main differences (in terms of code) between
> > > >> Dropwizard and Quarkus
> > > >> 2. To build and run Polaris powered by Quarkus
> > > >> 3. The list of TODO items
> > > >>
> > > >>
> > > >>
> >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > >>
> > > >> If anyone wants to contribute on the branch before creating the PR,
> > > >> please let me know, I will add you as contributor on the branch.
> > > >>
> > > >> Any comments or questions are welcome !
> > > >>
> > > >> Thanks,
> > > >> Regards
> > > >> JB
> > > >>
> > > >
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Dmitri Bourlatchkov <dm...@dremio.com.INVALID>.
2. Refactor polaris-service into 2 modules: e.g. "polaris-service-common"
and "polaris-service-dropwizard" – I believe Dmitri is already working
on this
Confirming: I do have some WIP for this.
On Wed, Nov 13, 2024 at 2:13 PM Alex Dutra <al...@dremio.com.invalid>
wrote:
> Hi Michael, hi all,
>
> I had a look at your branch and am very pleased to see that it was
> relatively easy to get DI working with DW + HK2. Good news!
>
> Digging a bit into the details, I see that most beans are annotated with
> @Named now. This is probably OK for Quarkus too, although there are other
> techniques that allow to select a specific implementation at runtime.
>
> I also think that the @RealmContext annotation is very smart, and resonates
> with how I handled that in Quarkus. Although, I wonder if this custom
> context isn't identical to the standard request context? IOW, is it
> possible for the Polaris context to change during an HTTP call to Polaris
> API? I don't think so, which leads me to think that, at least in Quarkus,
> we'd use the standard @RequestScoped annotation for passing things like
> this.
>
> But now taking a step back and looking at the big picture, I think that
> what we need now is a plan forward. Given that many of us voiced concerns
> around an all-or-nothing switch from DW to Quarkus (quite understandably
> so), and assuming that there is agreement to maintain both runtimes (DW and
> Quarkus) for some time, I'd like to suggest the phased approach below:
>
> 1. Remove the dependency to DW from polaris-core – there is a PR for
> that already: https://github.com/apache/polaris/pull/435
> 2. Refactor polaris-service into 2 modules: e.g.
> "polaris-service-common"
> and "polaris-service-dropwizard" – I believe Dmitri is already working
> on this
> 3. Introduce DI + HK2 in polaris-service-dropwizard – that would come
> from your branch basically
> 4. Introduce polaris-service-quarkus – that would come mostly from JB's
> branch
>
> Regarding step #2, it seems we can easily put 80 to 90% of the actual
> polaris-service production classes into the common module. The dropwizard
> runtime module would contain just a handful of DW-specific classes.
>
> The biggest issue to solve with step #2 is imho test classes. Quarkus and
> DW have diverse requirements for tests, but we may still be able to
> refactor things using abstract classes or extensions, and reuse most of
> them. In any case, I think that if, at least at the beginning, test classes
> are duplicated in each runtime module, that's acceptable until we figure
> out a better way.
>
> And to circle back to DI, whether this common module would contain classes
> annotated with Jakarta CDI annotations is imho up for debate. This would
> simplify things for sure, but on the flip side, would require both runtimes
> to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
> beans). I think that this is going to be too restrictive. An alternative
> would be to keep the common classes clean of CDI annotations and let the
> runtime modules create subclasses annotated with their own annotations.
>
> And back to the @RealmContext topic, that's the other big aspect that we
> will need to solve (maybe step #5?): CallContext is heavily reliant on
> ThreadLocal, and that is not very CDI-friendly, as you've seen. For now, I
> did a patchy workaround for Quarkus. But we'd need to refactor that code
> cleanly, either by creating a custom context as you did, or by making
> CallContext a request-scoped bean that is evaluated for each HTTP request.
> Either way, it then needs to be propagated to other threads (e.g. in
> TaskExecutorImpl); for Quarkus, this will require using proper context
> propagation <https://quarkus.io/guides/context-propagation> (for DW, maybe
> ThreadLocal is fine here). And since CallContext is in the core module, we
> will likely need to remove the static methods getCurrentContext() and
> setCurrentContext() and retrieve the context only by scope resolution
> (which, in turn, may use ThreadLocal under the hood if that's appropriate
> for DW).
>
> If everybody is on board with the plan above, how about we create GH issues
> for each step, and maybe even a GH Project to track progress on this front
> and split the effort?
>
> Thanks,
>
> Alex
>
>
> On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi Michael
> >
> > Thanks for the update !
> >
> > As discussed during the last community meeting:
> > 1. I will open the draft PR with Quarkus in the meantime
> > 2. I will take a look on your branch powered by Glassfish HK2 CDI
> > (give me a few days as I was travelling this week :))
> >
> > Thanks !
> > Regards
> >
> > On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <co...@gmail.com>
> > wrote:
> > >
> > > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> > impl.
> > > I only spent yesterday and today on this, so it's not complete, but it
> is
> > > functional and the tests pass :)
> > >
> > > I took a lot of the common ideas from the Quarkus branch (e.g., deleted
> > all
> > > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> > > figured out how to use the Dropwizard Yaml to specify the
> > > implementation of, e.g. the Authenticator and the
> > MetaStoreManagerFactory,
> > > but have the instances managed and injectable by HK2. The goal there
> was
> > to
> > > just keep the existing configuration format, but change the impl under
> > the
> > > hood. I'm not married to the idea and I'm interested to see if the
> > > jakarta.enterprise.inject.* annotations/interfaces that are used in the
> > > Quarkus branch can make this simpler. However, I do think it would
> ideal
> > if
> > > we can get it working with the existing Yaml configuration, at least in
> > the
> > > short term.
> > >
> > > I did add support for a @RealmScope annotation to support restricting
> > items
> > > to a given realm, such as the EntityCache and the PolarisGrantManager.
> > This
> > > allowed me to do things like hide the grant lookups from the
> EntityCache
> > so
> > > that the Resolver doesn't have to pass around the
> ResolvedPolarisEntity,
> > > but instead the grants are found from the cache without making it overt
> > in
> > > the PolarisAuthorizer API. This was one of my original goals with
> > breaking
> > > up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> > > everything still ties back to the configured PolarisMetaStoreManager
> > > implementation, but eventually we can get to where the GrantManager,
> > > CredentialVendor, etc. can all be swapped out for different
> > implementations.
> > >
> > > Please take a look at the changes at
> > >
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > > and let me know what you think.
> > >
> > > Mike
> > >
> > > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <collado.mike@gmail.com
> >
> > > wrote:
> > >
> > > > I added a table to the README with the differences that were called
> > out. I
> > > > added some details that I think are worth understanding better. E.g.,
> > the
> > > > Json layout we added has specific custom functionality we wanted for
> > > > supporting key/value pairs and the micrometer annotation was added
> for
> > some
> > > > custom support we wanted aside from what is supported with the
> default
> > > > dropwizard metrics support.
> > > >
> > > >
> > > >
> >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > > >
> > > > I think it would be great if others could add to the table with
> > features
> > > > that would be impacted by the migration and to call out the level of
> > effort
> > > > in both dropwizard and quarkus.
> > > >
> > > > Another possible consideration would be upgrading our Dropwizard
> > > > dependency to the latest development version. It may be the case that
> > doing
> > > > so would address some of the targeted features with less effort in
> > > > migrating.
> > > >
> > > > Mike
> > > >
> > > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> Following the last community sync meeting, we create a branch
> > > >> demonstrating use of Quarkus to powered Apache Polaris:
> > > >>
> > > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > > >>
> > > >> We still have work to do, but you can already take a look and
> > > >> experiment (in the polaris-service-quarkus module).
> > > >>
> > > >> We added a README.md file to:
> > > >> 1. Highlight the main differences (in terms of code) between
> > > >> Dropwizard and Quarkus
> > > >> 2. To build and run Polaris powered by Quarkus
> > > >> 3. The list of TODO items
> > > >>
> > > >>
> > > >>
> >
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > > >>
> > > >> If anyone wants to contribute on the branch before creating the PR,
> > > >> please let me know, I will add you as contributor on the branch.
> > > >>
> > > >> Any comments or questions are welcome !
> > > >>
> > > >> Thanks,
> > > >> Regards
> > > >> JB
> > > >>
> > > >
> >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Alex Dutra <al...@dremio.com.INVALID>.
Hi Michael, hi all,
I had a look at your branch and am very pleased to see that it was
relatively easy to get DI working with DW + HK2. Good news!
Digging a bit into the details, I see that most beans are annotated with
@Named now. This is probably OK for Quarkus too, although there are other
techniques that allow to select a specific implementation at runtime.
I also think that the @RealmContext annotation is very smart, and resonates
with how I handled that in Quarkus. Although, I wonder if this custom
context isn't identical to the standard request context? IOW, is it
possible for the Polaris context to change during an HTTP call to Polaris
API? I don't think so, which leads me to think that, at least in Quarkus,
we'd use the standard @RequestScoped annotation for passing things like
this.
But now taking a step back and looking at the big picture, I think that
what we need now is a plan forward. Given that many of us voiced concerns
around an all-or-nothing switch from DW to Quarkus (quite understandably
so), and assuming that there is agreement to maintain both runtimes (DW and
Quarkus) for some time, I'd like to suggest the phased approach below:
1. Remove the dependency to DW from polaris-core – there is a PR for
that already: https://github.com/apache/polaris/pull/435
2. Refactor polaris-service into 2 modules: e.g. "polaris-service-common"
and "polaris-service-dropwizard" – I believe Dmitri is already working
on this
3. Introduce DI + HK2 in polaris-service-dropwizard – that would come
from your branch basically
4. Introduce polaris-service-quarkus – that would come mostly from JB's
branch
Regarding step #2, it seems we can easily put 80 to 90% of the actual
polaris-service production classes into the common module. The dropwizard
runtime module would contain just a handful of DW-specific classes.
The biggest issue to solve with step #2 is imho test classes. Quarkus and
DW have diverse requirements for tests, but we may still be able to
refactor things using abstract classes or extensions, and reuse most of
them. In any case, I think that if, at least at the beginning, test classes
are duplicated in each runtime module, that's acceptable until we figure
out a better way.
And to circle back to DI, whether this common module would contain classes
annotated with Jakarta CDI annotations is imho up for debate. This would
simplify things for sure, but on the flip side, would require both runtimes
to use the exact same CDI mechanisms (e.g. both runtimes would use @Named
beans). I think that this is going to be too restrictive. An alternative
would be to keep the common classes clean of CDI annotations and let the
runtime modules create subclasses annotated with their own annotations.
And back to the @RealmContext topic, that's the other big aspect that we
will need to solve (maybe step #5?): CallContext is heavily reliant on
ThreadLocal, and that is not very CDI-friendly, as you've seen. For now, I
did a patchy workaround for Quarkus. But we'd need to refactor that code
cleanly, either by creating a custom context as you did, or by making
CallContext a request-scoped bean that is evaluated for each HTTP request.
Either way, it then needs to be propagated to other threads (e.g. in
TaskExecutorImpl); for Quarkus, this will require using proper context
propagation <https://quarkus.io/guides/context-propagation> (for DW, maybe
ThreadLocal is fine here). And since CallContext is in the core module, we
will likely need to remove the static methods getCurrentContext() and
setCurrentContext() and retrieve the context only by scope resolution
(which, in turn, may use ThreadLocal under the hood if that's appropriate
for DW).
If everybody is on board with the plan above, how about we create GH issues
for each step, and maybe even a GH Project to track progress on this front
and split the effort?
Thanks,
Alex
On Sat, Nov 9, 2024 at 10:15 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Michael
>
> Thanks for the update !
>
> As discussed during the last community meeting:
> 1. I will open the draft PR with Quarkus in the meantime
> 2. I will take a look on your branch powered by Glassfish HK2 CDI
> (give me a few days as I was travelling this week :))
>
> Thanks !
> Regards
>
> On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <co...@gmail.com>
> wrote:
> >
> > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> impl.
> > I only spent yesterday and today on this, so it's not complete, but it is
> > functional and the tests pass :)
> >
> > I took a lot of the common ideas from the Quarkus branch (e.g., deleted
> all
> > the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> > figured out how to use the Dropwizard Yaml to specify the
> > implementation of, e.g. the Authenticator and the
> MetaStoreManagerFactory,
> > but have the instances managed and injectable by HK2. The goal there was
> to
> > just keep the existing configuration format, but change the impl under
> the
> > hood. I'm not married to the idea and I'm interested to see if the
> > jakarta.enterprise.inject.* annotations/interfaces that are used in the
> > Quarkus branch can make this simpler. However, I do think it would ideal
> if
> > we can get it working with the existing Yaml configuration, at least in
> the
> > short term.
> >
> > I did add support for a @RealmScope annotation to support restricting
> items
> > to a given realm, such as the EntityCache and the PolarisGrantManager.
> This
> > allowed me to do things like hide the grant lookups from the EntityCache
> so
> > that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
> > but instead the grants are found from the cache without making it overt
> in
> > the PolarisAuthorizer API. This was one of my original goals with
> breaking
> > up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> > everything still ties back to the configured PolarisMetaStoreManager
> > implementation, but eventually we can get to where the GrantManager,
> > CredentialVendor, etc. can all be swapped out for different
> implementations.
> >
> > Please take a look at the changes at
> >
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> > and let me know what you think.
> >
> > Mike
> >
> > On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
> > wrote:
> >
> > > I added a table to the README with the differences that were called
> out. I
> > > added some details that I think are worth understanding better. E.g.,
> the
> > > Json layout we added has specific custom functionality we wanted for
> > > supporting key/value pairs and the micrometer annotation was added for
> some
> > > custom support we wanted aside from what is supported with the default
> > > dropwizard metrics support.
> > >
> > >
> > >
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> > >
> > > I think it would be great if others could add to the table with
> features
> > > that would be impacted by the migration and to call out the level of
> effort
> > > in both dropwizard and quarkus.
> > >
> > > Another possible consideration would be upgrading our Dropwizard
> > > dependency to the latest development version. It may be the case that
> doing
> > > so would address some of the targeted features with less effort in
> > > migrating.
> > >
> > > Mike
> > >
> > > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> Following the last community sync meeting, we create a branch
> > >> demonstrating use of Quarkus to powered Apache Polaris:
> > >>
> > >> https://github.com/jbonofre/polaris/tree/QUARKUS
> > >>
> > >> We still have work to do, but you can already take a look and
> > >> experiment (in the polaris-service-quarkus module).
> > >>
> > >> We added a README.md file to:
> > >> 1. Highlight the main differences (in terms of code) between
> > >> Dropwizard and Quarkus
> > >> 2. To build and run Polaris powered by Quarkus
> > >> 3. The list of TODO items
> > >>
> > >>
> > >>
> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> > >>
> > >> If anyone wants to contribute on the branch before creating the PR,
> > >> please let me know, I will add you as contributor on the branch.
> > >>
> > >> Any comments or questions are welcome !
> > >>
> > >> Thanks,
> > >> Regards
> > >> JB
> > >>
> > >
>
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Michael
Thanks for the update !
As discussed during the last community meeting:
1. I will open the draft PR with Quarkus in the meantime
2. I will take a look on your branch powered by Glassfish HK2 CDI
(give me a few days as I was travelling this week :))
Thanks !
Regards
On Fri, Nov 8, 2024 at 2:36 AM Michael Collado <co...@gmail.com> wrote:
>
> FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI impl.
> I only spent yesterday and today on this, so it's not complete, but it is
> functional and the tests pass :)
>
> I took a lot of the common ideas from the Quarkus branch (e.g., deleted all
> the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
> figured out how to use the Dropwizard Yaml to specify the
> implementation of, e.g. the Authenticator and the MetaStoreManagerFactory,
> but have the instances managed and injectable by HK2. The goal there was to
> just keep the existing configuration format, but change the impl under the
> hood. I'm not married to the idea and I'm interested to see if the
> jakarta.enterprise.inject.* annotations/interfaces that are used in the
> Quarkus branch can make this simpler. However, I do think it would ideal if
> we can get it working with the existing Yaml configuration, at least in the
> short term.
>
> I did add support for a @RealmScope annotation to support restricting items
> to a given realm, such as the EntityCache and the PolarisGrantManager. This
> allowed me to do things like hide the grant lookups from the EntityCache so
> that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
> but instead the grants are found from the cache without making it overt in
> the PolarisAuthorizer API. This was one of my original goals with breaking
> up the PolarisMetaStoreManager API into multiple interfaces. Right now,
> everything still ties back to the configured PolarisMetaStoreManager
> implementation, but eventually we can get to where the GrantManager,
> CredentialVendor, etc. can all be swapped out for different implementations.
>
> Please take a look at the changes at
> https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
> and let me know what you think.
>
> Mike
>
> On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
> wrote:
>
> > I added a table to the README with the differences that were called out. I
> > added some details that I think are worth understanding better. E.g., the
> > Json layout we added has specific custom functionality we wanted for
> > supporting key/value pairs and the micrometer annotation was added for some
> > custom support we wanted aside from what is supported with the default
> > dropwizard metrics support.
> >
> >
> > https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
> >
> > I think it would be great if others could add to the table with features
> > that would be impacted by the migration and to call out the level of effort
> > in both dropwizard and quarkus.
> >
> > Another possible consideration would be upgrading our Dropwizard
> > dependency to the latest development version. It may be the case that doing
> > so would address some of the targeted features with less effort in
> > migrating.
> >
> > Mike
> >
> > On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> >> Hi folks,
> >>
> >> Following the last community sync meeting, we create a branch
> >> demonstrating use of Quarkus to powered Apache Polaris:
> >>
> >> https://github.com/jbonofre/polaris/tree/QUARKUS
> >>
> >> We still have work to do, but you can already take a look and
> >> experiment (in the polaris-service-quarkus module).
> >>
> >> We added a README.md file to:
> >> 1. Highlight the main differences (in terms of code) between
> >> Dropwizard and Quarkus
> >> 2. To build and run Polaris powered by Quarkus
> >> 3. The list of TODO items
> >>
> >>
> >> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
> >>
> >> If anyone wants to contribute on the branch before creating the PR,
> >> please let me know, I will add you as contributor on the branch.
> >>
> >> Any comments or questions are welcome !
> >>
> >> Thanks,
> >> Regards
> >> JB
> >>
> >
Re: [DISCUSS] Apache Polaris powered by Quarkus
Posted by Michael Collado <co...@gmail.com>.
FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI impl.
I only spent yesterday and today on this, so it's not complete, but it is
functional and the tests pass :)
I took a lot of the common ideas from the Quarkus branch (e.g., deleted all
the HasXXX and XXXAware interfaces), but kept the JSON/Yaml config. I
figured out how to use the Dropwizard Yaml to specify the
implementation of, e.g. the Authenticator and the MetaStoreManagerFactory,
but have the instances managed and injectable by HK2. The goal there was to
just keep the existing configuration format, but change the impl under the
hood. I'm not married to the idea and I'm interested to see if the
jakarta.enterprise.inject.* annotations/interfaces that are used in the
Quarkus branch can make this simpler. However, I do think it would ideal if
we can get it working with the existing Yaml configuration, at least in the
short term.
I did add support for a @RealmScope annotation to support restricting items
to a given realm, such as the EntityCache and the PolarisGrantManager. This
allowed me to do things like hide the grant lookups from the EntityCache so
that the Resolver doesn't have to pass around the ResolvedPolarisEntity,
but instead the grants are found from the cache without making it overt in
the PolarisAuthorizer API. This was one of my original goals with breaking
up the PolarisMetaStoreManager API into multiple interfaces. Right now,
everything still ties back to the configured PolarisMetaStoreManager
implementation, but eventually we can get to where the GrantManager,
CredentialVendor, etc. can all be swapped out for different implementations.
Please take a look at the changes at
https://github.com/collado-mike/polaris/compare/c0e8dae5182d33e046216510e2b02b7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
and let me know what you think.
Mike
On Tue, Nov 5, 2024 at 9:42 AM Michael Collado <co...@gmail.com>
wrote:
> I added a table to the README with the differences that were called out. I
> added some details that I think are worth understanding better. E.g., the
> Json layout we added has specific custom functionality we wanted for
> supporting key/value pairs and the micrometer annotation was added for some
> custom support we wanted aside from what is supported with the default
> dropwizard metrics support.
>
>
> https://github.com/collado-mike/polaris/blob/14865a97ad8f790a9992432d79975c05ff5c36fa/polaris-service-quarkus/README.md
>
> I think it would be great if others could add to the table with features
> that would be impacted by the migration and to call out the level of effort
> in both dropwizard and quarkus.
>
> Another possible consideration would be upgrading our Dropwizard
> dependency to the latest development version. It may be the case that doing
> so would address some of the targeted features with less effort in
> migrating.
>
> Mike
>
> On Tue, Oct 29, 2024 at 5:50 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi folks,
>>
>> Following the last community sync meeting, we create a branch
>> demonstrating use of Quarkus to powered Apache Polaris:
>>
>> https://github.com/jbonofre/polaris/tree/QUARKUS
>>
>> We still have work to do, but you can already take a look and
>> experiment (in the polaris-service-quarkus module).
>>
>> We added a README.md file to:
>> 1. Highlight the main differences (in terms of code) between
>> Dropwizard and Quarkus
>> 2. To build and run Polaris powered by Quarkus
>> 3. The list of TODO items
>>
>>
>> https://github.com/jbonofre/polaris/blob/QUARKUS/polaris-service-quarkus/README.md
>>
>> If anyone wants to contribute on the branch before creating the PR,
>> please let me know, I will add you as contributor on the branch.
>>
>> Any comments or questions are welcome !
>>
>> Thanks,
>> Regards
>> JB
>>
>