You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by James Turton <dz...@apache.org> on 2022/08/16 14:01:31 UTC

[DISCUSS] the plugin architecture

Hi all

This topic has come up on-and-off for a while now and circumstances mean 
we should probably consider it again. To save myself from having to 
structure an essay I'm going let my writing deteriorate to short bullets 
now.

  * In a recent Phoenix plugin PR we learned that Drill users want to
    use plugins based on both the "thin" and "thick" Phoenix JDBC
    drivers. A change to move to the thick driver was recently expedited
    because of an Avatica classpath conflict in the thin driver which
    tanks Drill if the classloader dice fall the wrong way. We had
    started getting reports of that happening in Drill master. Now we're
    left with the question of how we could retain the thin driver.
  * We want to support building plugins (contrib/) away from the main
    source tree, shipping a leaner base Drill and the ability to install
    plugins from Maven coordinates created by third parties.
  * We recently started publishing the Drill SNAPSHOT artifacts that
    plugin authors targeting the coming Drill release depend on from our CI.
  * I propose a new drill-contrib repo to hold what is currently
    contrib/ in the main repo. The drill-contrib repo would probably
    have to be versioned and released in lockstep with the main repo,
    making one question the point of it, but it would force us to eat
    our own dogfood in that we'd be forced to build and ship plugins
    away from the main source tree.
  * Plugins becoming installed individually we could reintroduce the
    thin Phoenix driver as, say, storage-pqs and not worry about
    crashing the default Drill build.
  * A second iteration could push modularity and isolation further. OSGi
    looks complex and bigger than we need but JPMS module layers (see
    the end of this article
    <https://inside.java/2021/09/10/what-are-modules-about/>) look like
    they might do the trick and offer classloader isolation. An upshot
    would be that Drill 2.0 would then not run on the ageing Java 8 - I
    guess we'd have to be prepared to accept that.
  * Introducing JPMS would also allow us to limit how much of the core
    Drill APIs get programmed against by plugin authors by using its
    visibility features.
  * If we got module layers working to provide classloader isolation
    then the Phoenix thin driver and the Avatica shaded into it (the
    thin driver is a little fat in this sense 🙂) could in principle
    return to Drill unmodified. Otherwise we're looking at building and
    distributing a new thin Phoenix driver just for Drill.

If you really let your imagination go, you can imagine all plugins 
running as totally decoupled microservices with integration over a some 
network protocol and a new API. I must admit that sounds slow and 
painful to me. And it could also be achieved by a new 
storage-external-microservice-adapter plugin for anyone who really wants it.

What do you think?

James

Re: [DISCUSS] the plugin architecture

Posted by James Turton <dz...@apache.org>.
After chewing a little more on this, I don't think it goes anywhere 
interesting without classloader isolation. Probably just have to bite 
the bullet and do that upfront.

On 2022/08/16 19:12, James Turton wrote:
> A little bit more information about the proposed first stage that 
> moves a minimally modified contrib/ out to a new "drill-contrib" 
> repository based on my experiments.
>
>  * It is possible to build and test the plugins after doing this.
>  * Interactive debugging of plugin unit tests still works but requires
>    an "Attach source" step if you don't want to debug into decompiled
>    Drill code.
>  * Simultaneous changes to core Drill and a plugin would probably
>    require a bit more juggling than in the past and would lead to two
>    PRs, one to each repo. On the other hand this arrangement might make
>    it easier to keep plugin-specific code out of Drill core, promoting
>    better system design.
>  * We can use git subtree to take the commit history relevant to
>    plugins with.
>  * There is a user installation question that comes up - how, after
>    this first stage, should users install plugins now that the Drill
>    they downloaded was no longer the "kitchen sink"?
>
>
> On 2022/08/16 16:01, James Turton wrote:
>> Hi all
>>
>> This topic has come up on-and-off for a while now and circumstances 
>> mean we should probably consider it again. To save myself from having 
>> to structure an essay I'm going let my writing deteriorate to short 
>> bullets now.
>>
>>  * In a recent Phoenix plugin PR we learned that Drill users want to
>>    use plugins based on both the "thin" and "thick" Phoenix JDBC
>>    drivers. A change to move to the thick driver was recently expedited
>>    because of an Avatica classpath conflict in the thin driver which
>>    tanks Drill if the classloader dice fall the wrong way. We had
>>    started getting reports of that happening in Drill master. Now we're
>>    left with the question of how we could retain the thin driver.
>>  * We want to support building plugins (contrib/) away from the main
>>    source tree, shipping a leaner base Drill and the ability to install
>>    plugins from Maven coordinates created by third parties.
>>  * We recently started publishing the Drill SNAPSHOT artifacts that
>>    plugin authors targeting the coming Drill release depend on from 
>> our CI.
>>  * I propose a new drill-contrib repo to hold what is currently
>>    contrib/ in the main repo. The drill-contrib repo would probably
>>    have to be versioned and released in lockstep with the main repo,
>>    making one question the point of it, but it would force us to eat
>>    our own dogfood in that we'd be forced to build and ship plugins
>>    away from the main source tree.
>>  * Plugins becoming installed individually we could reintroduce the
>>    thin Phoenix driver as, say, storage-pqs and not worry about
>>    crashing the default Drill build.
>>  * A second iteration could push modularity and isolation further. OSGi
>>    looks complex and bigger than we need but JPMS module layers (see
>>    the end of this article
>> <https://inside.java/2021/09/10/what-are-modules-about/>) look like
>>    they might do the trick and offer classloader isolation. An upshot
>>    would be that Drill 2.0 would then not run on the ageing Java 8 - I
>>    guess we'd have to be prepared to accept that.
>>  * Introducing JPMS would also allow us to limit how much of the core
>>    Drill APIs get programmed against by plugin authors by using its
>>    visibility features.
>>  * If we got module layers working to provide classloader isolation
>>    then the Phoenix thin driver and the Avatica shaded into it (the
>>    thin driver is a little fat in this sense 🙂) could in principle
>>    return to Drill unmodified. Otherwise we're looking at building and
>>    distributing a new thin Phoenix driver just for Drill.
>>
>> If you really let your imagination go, you can imagine all plugins 
>> running as totally decoupled microservices with integration over a 
>> some network protocol and a new API. I must admit that sounds slow 
>> and painful to me. And it could also be achieved by a new 
>> storage-external-microservice-adapter plugin for anyone who really 
>> wants it.
>>
>> What do you think?
>>
>> James
>>
>


Re: [DISCUSS] the plugin architecture

Posted by James Turton <dz...@apache.org>.
A little bit more information about the proposed first stage that moves 
a minimally modified contrib/ out to a new "drill-contrib" repository 
based on my experiments.

  * It is possible to build and test the plugins after doing this.
  * Interactive debugging of plugin unit tests still works but requires
    an "Attach source" step if you don't want to debug into decompiled
    Drill code.
  * Simultaneous changes to core Drill and a plugin would probably
    require a bit more juggling than in the past and would lead to two
    PRs, one to each repo. On the other hand this arrangement might make
    it easier to keep plugin-specific code out of Drill core, promoting
    better system design.
  * We can use git subtree to take the commit history relevant to
    plugins with.
  * There is a user installation question that comes up - how, after
    this first stage, should users install plugins now that the Drill
    they downloaded was no longer the "kitchen sink"?


On 2022/08/16 16:01, James Turton wrote:
> Hi all
>
> This topic has come up on-and-off for a while now and circumstances 
> mean we should probably consider it again. To save myself from having 
> to structure an essay I'm going let my writing deteriorate to short 
> bullets now.
>
>  * In a recent Phoenix plugin PR we learned that Drill users want to
>    use plugins based on both the "thin" and "thick" Phoenix JDBC
>    drivers. A change to move to the thick driver was recently expedited
>    because of an Avatica classpath conflict in the thin driver which
>    tanks Drill if the classloader dice fall the wrong way. We had
>    started getting reports of that happening in Drill master. Now we're
>    left with the question of how we could retain the thin driver.
>  * We want to support building plugins (contrib/) away from the main
>    source tree, shipping a leaner base Drill and the ability to install
>    plugins from Maven coordinates created by third parties.
>  * We recently started publishing the Drill SNAPSHOT artifacts that
>    plugin authors targeting the coming Drill release depend on from 
> our CI.
>  * I propose a new drill-contrib repo to hold what is currently
>    contrib/ in the main repo. The drill-contrib repo would probably
>    have to be versioned and released in lockstep with the main repo,
>    making one question the point of it, but it would force us to eat
>    our own dogfood in that we'd be forced to build and ship plugins
>    away from the main source tree.
>  * Plugins becoming installed individually we could reintroduce the
>    thin Phoenix driver as, say, storage-pqs and not worry about
>    crashing the default Drill build.
>  * A second iteration could push modularity and isolation further. OSGi
>    looks complex and bigger than we need but JPMS module layers (see
>    the end of this article
> <https://inside.java/2021/09/10/what-are-modules-about/>) look like
>    they might do the trick and offer classloader isolation. An upshot
>    would be that Drill 2.0 would then not run on the ageing Java 8 - I
>    guess we'd have to be prepared to accept that.
>  * Introducing JPMS would also allow us to limit how much of the core
>    Drill APIs get programmed against by plugin authors by using its
>    visibility features.
>  * If we got module layers working to provide classloader isolation
>    then the Phoenix thin driver and the Avatica shaded into it (the
>    thin driver is a little fat in this sense 🙂) could in principle
>    return to Drill unmodified. Otherwise we're looking at building and
>    distributing a new thin Phoenix driver just for Drill.
>
> If you really let your imagination go, you can imagine all plugins 
> running as totally decoupled microservices with integration over a 
> some network protocol and a new API. I must admit that sounds slow and 
> painful to me. And it could also be achieved by a new 
> storage-external-microservice-adapter plugin for anyone who really 
> wants it.
>
> What do you think?
>
> James
>