You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by James Turton <dz...@apache.org> on 2022/08/16 14:01:31 UTC
[DISCUSS] the plugin architecture
Hi all
This topic has come up on-and-off for a while now and circumstances mean
we should probably consider it again. To save myself from having to
structure an essay I'm going let my writing deteriorate to short bullets
now.
* In a recent Phoenix plugin PR we learned that Drill users want to
use plugins based on both the "thin" and "thick" Phoenix JDBC
drivers. A change to move to the thick driver was recently expedited
because of an Avatica classpath conflict in the thin driver which
tanks Drill if the classloader dice fall the wrong way. We had
started getting reports of that happening in Drill master. Now we're
left with the question of how we could retain the thin driver.
* We want to support building plugins (contrib/) away from the main
source tree, shipping a leaner base Drill and the ability to install
plugins from Maven coordinates created by third parties.
* We recently started publishing the Drill SNAPSHOT artifacts that
plugin authors targeting the coming Drill release depend on from our CI.
* I propose a new drill-contrib repo to hold what is currently
contrib/ in the main repo. The drill-contrib repo would probably
have to be versioned and released in lockstep with the main repo,
making one question the point of it, but it would force us to eat
our own dogfood in that we'd be forced to build and ship plugins
away from the main source tree.
* Plugins becoming installed individually we could reintroduce the
thin Phoenix driver as, say, storage-pqs and not worry about
crashing the default Drill build.
* A second iteration could push modularity and isolation further. OSGi
looks complex and bigger than we need but JPMS module layers (see
the end of this article
<https://inside.java/2021/09/10/what-are-modules-about/>) look like
they might do the trick and offer classloader isolation. An upshot
would be that Drill 2.0 would then not run on the ageing Java 8 - I
guess we'd have to be prepared to accept that.
* Introducing JPMS would also allow us to limit how much of the core
Drill APIs get programmed against by plugin authors by using its
visibility features.
* If we got module layers working to provide classloader isolation
then the Phoenix thin driver and the Avatica shaded into it (the
thin driver is a little fat in this sense 🙂) could in principle
return to Drill unmodified. Otherwise we're looking at building and
distributing a new thin Phoenix driver just for Drill.
If you really let your imagination go, you can imagine all plugins
running as totally decoupled microservices with integration over a some
network protocol and a new API. I must admit that sounds slow and
painful to me. And it could also be achieved by a new
storage-external-microservice-adapter plugin for anyone who really wants it.
What do you think?
James
Re: [DISCUSS] the plugin architecture
Posted by James Turton <dz...@apache.org>.
After chewing a little more on this, I don't think it goes anywhere
interesting without classloader isolation. Probably just have to bite
the bullet and do that upfront.
On 2022/08/16 19:12, James Turton wrote:
> A little bit more information about the proposed first stage that
> moves a minimally modified contrib/ out to a new "drill-contrib"
> repository based on my experiments.
>
> Â * It is possible to build and test the plugins after doing this.
> Â * Interactive debugging of plugin unit tests still works but requires
> Â Â an "Attach source" step if you don't want to debug into decompiled
> Â Â Drill code.
> Â * Simultaneous changes to core Drill and a plugin would probably
> Â Â require a bit more juggling than in the past and would lead to two
> Â Â PRs, one to each repo. On the other hand this arrangement might make
> Â Â it easier to keep plugin-specific code out of Drill core, promoting
> Â Â better system design.
> Â * We can use git subtree to take the commit history relevant to
> Â Â plugins with.
> Â * There is a user installation question that comes up - how, after
> Â Â this first stage, should users install plugins now that the Drill
> Â Â they downloaded was no longer the "kitchen sink"?
>
>
> On 2022/08/16 16:01, James Turton wrote:
>> Hi all
>>
>> This topic has come up on-and-off for a while now and circumstances
>> mean we should probably consider it again. To save myself from having
>> to structure an essay I'm going let my writing deteriorate to short
>> bullets now.
>>
>> Â * In a recent Phoenix plugin PR we learned that Drill users want to
>> Â Â use plugins based on both the "thin" and "thick" Phoenix JDBC
>> Â Â drivers. A change to move to the thick driver was recently expedited
>> Â Â because of an Avatica classpath conflict in the thin driver which
>> Â Â tanks Drill if the classloader dice fall the wrong way. We had
>> Â Â started getting reports of that happening in Drill master. Now we're
>> Â Â left with the question of how we could retain the thin driver.
>> Â * We want to support building plugins (contrib/) away from the main
>> Â Â source tree, shipping a leaner base Drill and the ability to install
>> Â Â plugins from Maven coordinates created by third parties.
>> Â * We recently started publishing the Drill SNAPSHOT artifacts that
>> Â Â plugin authors targeting the coming Drill release depend on from
>> our CI.
>> Â * I propose a new drill-contrib repo to hold what is currently
>> Â Â contrib/ in the main repo. The drill-contrib repo would probably
>> Â Â have to be versioned and released in lockstep with the main repo,
>> Â Â making one question the point of it, but it would force us to eat
>> Â Â our own dogfood in that we'd be forced to build and ship plugins
>> Â Â away from the main source tree.
>> Â * Plugins becoming installed individually we could reintroduce the
>> Â Â thin Phoenix driver as, say, storage-pqs and not worry about
>> Â Â crashing the default Drill build.
>> Â * A second iteration could push modularity and isolation further. OSGi
>> Â Â looks complex and bigger than we need but JPMS module layers (see
>> Â Â the end of this article
>> <https://inside.java/2021/09/10/what-are-modules-about/>) look like
>> Â Â they might do the trick and offer classloader isolation. An upshot
>> Â Â would be that Drill 2.0 would then not run on the ageing Java 8 - I
>> Â Â guess we'd have to be prepared to accept that.
>> Â * Introducing JPMS would also allow us to limit how much of the core
>> Â Â Drill APIs get programmed against by plugin authors by using its
>> Â Â visibility features.
>> Â * If we got module layers working to provide classloader isolation
>> Â Â then the Phoenix thin driver and the Avatica shaded into it (the
>>   thin driver is a little fat in this sense 🙂) could in principle
>> Â Â return to Drill unmodified. Otherwise we're looking at building and
>> Â Â distributing a new thin Phoenix driver just for Drill.
>>
>> If you really let your imagination go, you can imagine all plugins
>> running as totally decoupled microservices with integration over a
>> some network protocol and a new API. I must admit that sounds slow
>> and painful to me. And it could also be achieved by a new
>> storage-external-microservice-adapter plugin for anyone who really
>> wants it.
>>
>> What do you think?
>>
>> James
>>
>
Re: [DISCUSS] the plugin architecture
Posted by James Turton <dz...@apache.org>.
A little bit more information about the proposed first stage that moves
a minimally modified contrib/ out to a new "drill-contrib" repository
based on my experiments.
* It is possible to build and test the plugins after doing this.
* Interactive debugging of plugin unit tests still works but requires
an "Attach source" step if you don't want to debug into decompiled
Drill code.
* Simultaneous changes to core Drill and a plugin would probably
require a bit more juggling than in the past and would lead to two
PRs, one to each repo. On the other hand this arrangement might make
it easier to keep plugin-specific code out of Drill core, promoting
better system design.
* We can use git subtree to take the commit history relevant to
plugins with.
* There is a user installation question that comes up - how, after
this first stage, should users install plugins now that the Drill
they downloaded was no longer the "kitchen sink"?
On 2022/08/16 16:01, James Turton wrote:
> Hi all
>
> This topic has come up on-and-off for a while now and circumstances
> mean we should probably consider it again. To save myself from having
> to structure an essay I'm going let my writing deteriorate to short
> bullets now.
>
> Â * In a recent Phoenix plugin PR we learned that Drill users want to
> Â Â use plugins based on both the "thin" and "thick" Phoenix JDBC
> Â Â drivers. A change to move to the thick driver was recently expedited
> Â Â because of an Avatica classpath conflict in the thin driver which
> Â Â tanks Drill if the classloader dice fall the wrong way. We had
> Â Â started getting reports of that happening in Drill master. Now we're
> Â Â left with the question of how we could retain the thin driver.
> Â * We want to support building plugins (contrib/) away from the main
> Â Â source tree, shipping a leaner base Drill and the ability to install
> Â Â plugins from Maven coordinates created by third parties.
> Â * We recently started publishing the Drill SNAPSHOT artifacts that
> Â Â plugin authors targeting the coming Drill release depend on from
> our CI.
> Â * I propose a new drill-contrib repo to hold what is currently
> Â Â contrib/ in the main repo. The drill-contrib repo would probably
> Â Â have to be versioned and released in lockstep with the main repo,
> Â Â making one question the point of it, but it would force us to eat
> Â Â our own dogfood in that we'd be forced to build and ship plugins
> Â Â away from the main source tree.
> Â * Plugins becoming installed individually we could reintroduce the
> Â Â thin Phoenix driver as, say, storage-pqs and not worry about
> Â Â crashing the default Drill build.
> Â * A second iteration could push modularity and isolation further. OSGi
> Â Â looks complex and bigger than we need but JPMS module layers (see
> Â Â the end of this article
> <https://inside.java/2021/09/10/what-are-modules-about/>) look like
> Â Â they might do the trick and offer classloader isolation. An upshot
> Â Â would be that Drill 2.0 would then not run on the ageing Java 8 - I
> Â Â guess we'd have to be prepared to accept that.
> Â * Introducing JPMS would also allow us to limit how much of the core
> Â Â Drill APIs get programmed against by plugin authors by using its
> Â Â visibility features.
> Â * If we got module layers working to provide classloader isolation
> Â Â then the Phoenix thin driver and the Avatica shaded into it (the
>   thin driver is a little fat in this sense 🙂) could in principle
> Â Â return to Drill unmodified. Otherwise we're looking at building and
> Â Â distributing a new thin Phoenix driver just for Drill.
>
> If you really let your imagination go, you can imagine all plugins
> running as totally decoupled microservices with integration over a
> some network protocol and a new API. I must admit that sounds slow and
> painful to me. And it could also be achieved by a new
> storage-external-microservice-adapter plugin for anyone who really
> wants it.
>
> What do you think?
>
> James
>