You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@shardingsphere.apache.org by "zhangliang@apache.org" <zh...@apache.org> on 2019/12/09 05:29:47 UTC

Discussion of pluggable platform of ShardingSphere

Hi, ShardingSphere community,

More and more features are added into ShardingSphere now, as you know, the
scope of ShardingSphere is no longer for sharding only.
There are more and more features related with sharding, such as distributed
transaction, distributed orchestration, observability and so on; and there
are couple of features did not relate with sharding obviously, for example:
encrypt, shadow data source, SQL audit and so on.

I'd like to discuss about establishing a pluggable platform of
ShardingSphere. The proposal of pluggable platform is decoupling all
features and technical implementations, the benefits are:

1. Flexible for add new feature.
2. Reduce the negative effects if problem occur on one feature.
3. Provide a platform to make more contributors work together without
interact on each other.

I plan split all features from sharding-core first, and then use SPI to
introduce features into the pluggable platform(same thing with sharding,
sharding can be remove from main process too).

The pluggable platform is a blank JDBC and database protocol finally,  and
provide assist technical features such as SQL parser and SQL rewrite.

Any advice?

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo

Re: Discussion of pluggable platform of ShardingSphere

Posted by Sheng Wu <wu...@gmail.com>.

+1 Love the idea. Just feeling the project name(Sharding) will be the
bottleneck :P

Sheng Wu 吴晟
Twitter, wusheng1108


zhangliang@apache.org <zh...@apache.org> 于2019年12月9日周一 下午1:30写道：

> Hi, ShardingSphere community,
>
> More and more features are added into ShardingSphere now, as you know, the
> scope of ShardingSphere is no longer for sharding only.
> There are more and more features related with sharding, such as distributed
> transaction, distributed orchestration, observability and so on; and there
> are couple of features did not relate with sharding obviously, for example:
> encrypt, shadow data source, SQL audit and so on.
>
> I'd like to discuss about establishing a pluggable platform of
> ShardingSphere. The proposal of pluggable platform is decoupling all
> features and technical implementations, the benefits are:
>
> 1. Flexible for add new feature.
> 2. Reduce the negative effects if problem occur on one feature.
> 3. Provide a platform to make more contributors work together without
> interact on each other.
>
> I plan split all features from sharding-core first, and then use SPI to
> introduce features into the pluggable platform(same thing with sharding,
> sharding can be remove from main process too).
>
> The pluggable platform is a blank JDBC and database protocol finally,  and
> provide assist technical features such as SQL parser and SQL rewrite.
>
> Any advice?
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by "zhangliang@apache.org" <zh...@apache.org>.

> +1 Love the idea. Just feeling the project name(Sharding) will be the
bottleneck :P

Sharding is a brand of previous name (Sharding-JDBC), because JDBC is not
suitable with our project, so we keep the first word as brand and just
change the last word to `Sphere`.

Maybe we can talk about change another name before graduated, hope to
discuss further :)

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo


zhangliang@apache.org <zh...@apache.org> 于2019年12月9日周一 下午1:44写道：

> It is big project, we should not include this changes on version 4.x.
> I plan finishing it at version 5.x. I can start as a branch to do some
> preparation works now[1].
>
> Hope after finish this project, the landscape of ShardingSphere[2] can be
> more powerful.
>
> [1] https://github.com/apache/incubator-shardingsphere/issues/3691
> [2] https://github.com/apache/incubator-shardingsphere/issues/2434
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
>
> zhangliang@apache.org <zh...@apache.org> 于2019年12月9日周一 下午1:29写道：
>
>> Hi, ShardingSphere community,
>>
>> More and more features are added into ShardingSphere now, as you know,
>> the scope of ShardingSphere is no longer for sharding only.
>> There are more and more features related with sharding, such as
>> distributed transaction, distributed orchestration, observability and so
>> on; and there are couple of features did not relate with sharding
>> obviously, for example: encrypt, shadow data source, SQL audit and so on.
>>
>> I'd like to discuss about establishing a pluggable platform of
>> ShardingSphere. The proposal of pluggable platform is decoupling all
>> features and technical implementations, the benefits are:
>>
>> 1. Flexible for add new feature.
>> 2. Reduce the negative effects if problem occur on one feature.
>> 3. Provide a platform to make more contributors work together without
>> interact on each other.
>>
>> I plan split all features from sharding-core first, and then use SPI to
>> introduce features into the pluggable platform(same thing with sharding,
>> sharding can be remove from main process too).
>>
>> The pluggable platform is a blank JDBC and database protocol finally,
>> and provide assist technical features such as SQL parser and SQL rewrite.
>>
>> Any advice?
>>
>> ------------------
>>
>> Liang Zhang (John)
>> Apache ShardingSphere & Dubbo
>>
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by "zhangliang@apache.org" <zh...@apache.org>.

It is big project, we should not include this changes on version 4.x.
I plan finishing it at version 5.x. I can start as a branch to do some
preparation works now[1].

Hope after finish this project, the landscape of ShardingSphere[2] can be
more powerful.

[1] https://github.com/apache/incubator-shardingsphere/issues/3691
[2] https://github.com/apache/incubator-shardingsphere/issues/2434

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo


zhangliang@apache.org <zh...@apache.org> 于2019年12月9日周一 下午1:29写道：

> Hi, ShardingSphere community,
>
> More and more features are added into ShardingSphere now, as you know, the
> scope of ShardingSphere is no longer for sharding only.
> There are more and more features related with sharding, such as
> distributed transaction, distributed orchestration, observability and so
> on; and there are couple of features did not relate with sharding
> obviously, for example: encrypt, shadow data source, SQL audit and so on.
>
> I'd like to discuss about establishing a pluggable platform of
> ShardingSphere. The proposal of pluggable platform is decoupling all
> features and technical implementations, the benefits are:
>
> 1. Flexible for add new feature.
> 2. Reduce the negative effects if problem occur on one feature.
> 3. Provide a platform to make more contributors work together without
> interact on each other.
>
> I plan split all features from sharding-core first, and then use SPI to
> introduce features into the pluggable platform(same thing with sharding,
> sharding can be remove from main process too).
>
> The pluggable platform is a blank JDBC and database protocol finally,  and
> provide assist technical features such as SQL parser and SQL rewrite.
>
> Any advice?
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by "zhangliang@apache.org" <zh...@apache.org>.

Hi,

> What features will be planned in pluggable platform?

There is no new feature of pluggable platform. We just need redesign the
current framework to make it more flexible.

> And how to assign features splits from sharding-core?

As the new design, sharding-core is not a mandatory feature, it
will downgrade as a regular one. The features should introduce as their jar
file and configuration (API should inside jar).

> Based on user concept，what does user benefit from this pluggable platform?

User do not need care about it(except API changes). The benefit for let
more contributors join the project without interact on each other.

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo


Nicholas <th...@gmail.com> 于2019年12月9日周一 下午2:13写道：

> Hi guys,
>     What features will be planned in pluggable platform? And how to assign
> features splits from sharding-core? Based on user concept，what does user
> benefit from this pluggable platform?
>
> Thanks,
> Nicholas Jiang
>
> On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
> wrote:
> > Hi, ShardingSphere community,
> >
> > More and more features are added into ShardingSphere now, as you know,
> the
> > scope of ShardingSphere is no longer for sharding only.
> > There are more and more features related with sharding, such as
> distributed
> > transaction, distributed orchestration, observability and so on; and
> there
> > are couple of features did not relate with sharding obviously, for
> example:
> > encrypt, shadow data source, SQL audit and so on.
> >
> > I'd like to discuss about establishing a pluggable platform of
> > ShardingSphere. The proposal of pluggable platform is decoupling all
> > features and technical implementations, the benefits are:
> >
> > 1. Flexible for add new feature.
> > 2. Reduce the negative effects if problem occur on one feature.
> > 3. Provide a platform to make more contributors work together without
> > interact on each other.
> >
> > I plan split all features from sharding-core first, and then use SPI to
> > introduce features into the pluggable platform(same thing with sharding,
> > sharding can be remove from main process too).
> >
> > The pluggable platform is a blank JDBC and database protocol finally,
> and
> > provide assist technical features such as SQL parser and SQL rewrite.
> >
> > Any advice?
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by Zheng Feng <zh...@gmail.com>.

Hi Liang,

The Plan B looks like similar to the netty pipeline architecture and will
be much more flexible.

Regards,
Zheng Feng


zhangliang@apache.org <zh...@apache.org> 于2020年1月3日周五 上午12:13写道：

> I have 2 ideas for the pluggable platform.
>
> Plan A:
>
> Design a unify DataSource to process all rules, it should be a transparent
> DataSource if no rule provided (but it should be valid). We can name it as
> `ShardingSphereDataSource`.
> There are lots of hooks in `ShardingSphereDataSource`, such as:
>   - route hook: for sharding, master-slave and shadow;
>   - SQL rewrite hook: for sharding, encrypt and shadow;
>   - execution hook: for sharding and transaction;
>   - merge hook: for sharding and encrypt;
>   - init hook: for orchestration;
>   - config refresh hook: for orchestration;
>
> The hook should load feature implementations via SPI and appended
> by pluggable platform automatically. (SPI should design the process order)
>
> Advantage:
>   - Only one API for `ShardingSphereDataSource` which should not confuse
> the users;
>   - Process logics internally, pretty high performance for this solution;
>   - Can be reused for Proxy.
>
> Disadvantage:
>   - Difficult for developers, they need to know every hooks.
>
> Plan B:
>
> Provide independent DataSources, every DataSources process oneself only.
> And we may have lots of DataSource, such as ShardingDataSource,
> MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
> TransactionDataSource, OrchestrationDataSource and so on.
> Every DataSources should composite by users themselves, ShardingSphere can
> privide a Facade class to simplify the usage.
>
> Advantage:
>   - Isolation by design level, every DataSource has clear and simple
> responsibility;
>   - Simple for developer, they need to know independent logic for one
> DataSource only, do not need expose internal implementations.
>
> Disadvantage:
>   - Can not reuse by Proxy;
>   - Difficult for user to find every possible combination of DataSource;
>   - Hide the internal implementation may cause low performance;
>
> By the way, I just discuss about how to design Proxy for Plan B. It should
> be every features have a independent proxy process (may consider about put
> them into one process in future).
> ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
> the performance should not enough.
>
> Conclusion:
> Even the Plan B has better design, but I worry about the performance and
> the difficulty for user. So I prefer Plan A for now.
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
>
> guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：
>
> > +1 It sounds great.
> > And is the platform suitable for JDTX? And Sharding may not be a
> necessary
> > function, maybe transaction will be more widely used. So, will JDTX will
> be
> > a plugin which is easily used in the platform?
> >
> > Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
> >
> > > +1, That sounds great!
> > >
> > >
> > > For a pluggable platform, contributors will be very convenient to
> > > contribute code, This will attract more people.
> > >
> > >
> > > Zonglei Dong
> > > Apache ShardingSphere
> > >
> > >
> > > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
> > > +1
> > >
> > >
> > > Good idea. It’s a good long-term planning for ShardingSphere.
> > >
> > >
> > > —————————
> > > Haisheng Sun (sunbufu)
> > > Apache ShardingSphere
> > >
> > >
> > > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
> > > Agree with Liang,
> > >
> > >
> > > Maybe there is no feature benefit apparently, however we make its
> > > architecture become…a structured plaza with many rooms. Consequently,
> > more
> > > contributors could fill in specific implements for those rooms with
> less
> > > harms to this plaza, i.e our project. Oh, i have a great imagination,
> > don’t
> > > you think so? :)
> > >
> > >
> > > Regards,
> > > Trista
> > >
> > >
> > > Juan Pan
> > >
> > >
> > > panjuan@apache.org
> > > Juan Pan(Trista), Apache ShardingSphere
> > >
> > >
> > > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
> > > Hi guys,
> > > What features will be planned in pluggable platform? And how to assign
> > > features splits from sharding-core? Based on user concept，what does
> user
> > > benefit from this pluggable platform?
> > >
> > > Thanks,
> > > Nicholas Jiang
> > >
> > > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zhangliang@apache.org
> >
> > > wrote:
> > > Hi, ShardingSphere community,
> > >
> > > More and more features are added into ShardingSphere now, as you know,
> > the
> > > scope of ShardingSphere is no longer for sharding only.
> > > There are more and more features related with sharding, such as
> > distributed
> > > transaction, distributed orchestration, observability and so on; and
> > there
> > > are couple of features did not relate with sharding obviously, for
> > example:
> > > encrypt, shadow data source, SQL audit and so on.
> > >
> > > I'd like to discuss about establishing a pluggable platform of
> > > ShardingSphere. The proposal of pluggable platform is decoupling all
> > > features and technical implementations, the benefits are:
> > >
> > > 1. Flexible for add new feature.
> > > 2. Reduce the negative effects if problem occur on one feature.
> > > 3. Provide a platform to make more contributors work together without
> > > interact on each other.
> > >
> > > I plan split all features from sharding-core first, and then use SPI to
> > > introduce features into the pluggable platform(same thing with
> sharding,
> > > sharding can be remove from main process too).
> > >
> > > The pluggable platform is a blank JDBC and database protocol finally,
> > and
> > > provide assist technical features such as SQL parser and SQL rewrite.
> > >
> > > Any advice?
> > >
> > > ------------------
> > >
> > > Liang Zhang (John)
> > > Apache ShardingSphere & Dubbo
> > >
> > >
> >
>

Re:Re: Discussion of pluggable platform of ShardingSphere

Posted by Sion Yang <sc...@163.com>.

I prefer Plan A too.








--

Yi Yang(Sion)
Apache ShardingSphere



At 2020-01-03 00:13:07, "zhangliang@apache.org" <zh...@apache.org> wrote:
>I have 2 ideas for the pluggable platform.
>
>Plan A:
>
>Design a unify DataSource to process all rules, it should be a transparent
>DataSource if no rule provided (but it should be valid). We can name it as
>`ShardingSphereDataSource`.
>There are lots of hooks in `ShardingSphereDataSource`, such as:
>  - route hook: for sharding, master-slave and shadow;
>  - SQL rewrite hook: for sharding, encrypt and shadow;
>  - execution hook: for sharding and transaction;
>  - merge hook: for sharding and encrypt;
>  - init hook: for orchestration;
>  - config refresh hook: for orchestration;
>
>The hook should load feature implementations via SPI and appended
>by pluggable platform automatically. (SPI should design the process order)
>
>Advantage:
>  - Only one API for `ShardingSphereDataSource` which should not confuse
>the users;
>  - Process logics internally, pretty high performance for this solution;
>  - Can be reused for Proxy.
>
>Disadvantage:
>  - Difficult for developers, they need to know every hooks.
>
>Plan B:
>
>Provide independent DataSources, every DataSources process oneself only.
>And we may have lots of DataSource, such as ShardingDataSource,
>MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
>TransactionDataSource, OrchestrationDataSource and so on.
>Every DataSources should composite by users themselves, ShardingSphere can
>privide a Facade class to simplify the usage.
>
>Advantage:
>  - Isolation by design level, every DataSource has clear and simple
>responsibility;
>  - Simple for developer, they need to know independent logic for one
>DataSource only, do not need expose internal implementations.
>
>Disadvantage:
>  - Can not reuse by Proxy;
>  - Difficult for user to find every possible combination of DataSource;
>  - Hide the internal implementation may cause low performance;
>
>By the way, I just discuss about how to design Proxy for Plan B. It should
>be every features have a independent proxy process (may consider about put
>them into one process in future).
>ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
>the performance should not enough.
>
>Conclusion:
>Even the Plan B has better design, but I worry about the performance and
>the difficulty for user. So I prefer Plan A for now.
>
>------------------
>
>Liang Zhang (John)
>Apache ShardingSphere & Dubbo
>
>
>guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：
>
>> +1 It sounds great.
>> And is the platform suitable for JDTX? And Sharding may not be a necessary
>> function, maybe transaction will be more widely used. So, will JDTX will be
>> a plugin which is easily used in the platform?
>>
>> Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
>>
>> > +1, That sounds great!
>> >
>> >
>> > For a pluggable platform, contributors will be very convenient to
>> > contribute code, This will attract more people.
>> >
>> >
>> > Zonglei Dong
>> > Apache ShardingSphere
>> >
>> >
>> > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
>> > +1
>> >
>> >
>> > Good idea. It’s a good long-term planning for ShardingSphere.
>> >
>> >
>> > —————————
>> > Haisheng Sun (sunbufu)
>> > Apache ShardingSphere
>> >
>> >
>> > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
>> > Agree with Liang,
>> >
>> >
>> > Maybe there is no feature benefit apparently, however we make its
>> > architecture become…a structured plaza with many rooms. Consequently,
>> more
>> > contributors could fill in specific implements for those rooms with less
>> > harms to this plaza, i.e our project. Oh, i have a great imagination,
>> don’t
>> > you think so? :)
>> >
>> >
>> > Regards,
>> > Trista
>> >
>> >
>> > Juan Pan
>> >
>> >
>> > panjuan@apache.org
>> > Juan Pan(Trista), Apache ShardingSphere
>> >
>> >
>> > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
>> > Hi guys,
>> > What features will be planned in pluggable platform? And how to assign
>> > features splits from sharding-core? Based on user concept，what does user
>> > benefit from this pluggable platform?
>> >
>> > Thanks,
>> > Nicholas Jiang
>> >
>> > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
>> > wrote:
>> > Hi, ShardingSphere community,
>> >
>> > More and more features are added into ShardingSphere now, as you know,
>> the
>> > scope of ShardingSphere is no longer for sharding only.
>> > There are more and more features related with sharding, such as
>> distributed
>> > transaction, distributed orchestration, observability and so on; and
>> there
>> > are couple of features did not relate with sharding obviously, for
>> example:
>> > encrypt, shadow data source, SQL audit and so on.
>> >
>> > I'd like to discuss about establishing a pluggable platform of
>> > ShardingSphere. The proposal of pluggable platform is decoupling all
>> > features and technical implementations, the benefits are:
>> >
>> > 1. Flexible for add new feature.
>> > 2. Reduce the negative effects if problem occur on one feature.
>> > 3. Provide a platform to make more contributors work together without
>> > interact on each other.
>> >
>> > I plan split all features from sharding-core first, and then use SPI to
>> > introduce features into the pluggable platform(same thing with sharding,
>> > sharding can be remove from main process too).
>> >
>> > The pluggable platform is a blank JDBC and database protocol finally,
>> and
>> > provide assist technical features such as SQL parser and SQL rewrite.
>> >
>> > Any advice?
>> >
>> > ------------------
>> >
>> > Liang Zhang (John)
>> > Apache ShardingSphere & Dubbo
>> >
>> >
>>

回复: Discussion of pluggable platform of ShardingSphere

Posted by Sun Kane <ka...@outlook.com>.

I personally recommend solution A, just like the current hardware system,although the hardware outside the mainboard is pluggable, users rarely plug hardware such like graphics cards or CPU.

reduce the difficulty for users to get started will the best
________________________________
发件人: zhangliang@apache.org <zh...@apache.org>
发送时间: 2020年1月3日 0:13
收件人: dev@shardingsphere.apache.org <de...@shardingsphere.apache.org>
主题: Re: Discussion of pluggable platform of ShardingSphere

I have 2 ideas for the pluggable platform.

Plan A:

Design a unify DataSource to process all rules, it should be a transparent
DataSource if no rule provided (but it should be valid). We can name it as
`ShardingSphereDataSource`.
There are lots of hooks in `ShardingSphereDataSource`, such as:
  - route hook: for sharding, master-slave and shadow;
  - SQL rewrite hook: for sharding, encrypt and shadow;
  - execution hook: for sharding and transaction;
  - merge hook: for sharding and encrypt;
  - init hook: for orchestration;
  - config refresh hook: for orchestration;

The hook should load feature implementations via SPI and appended
by pluggable platform automatically. (SPI should design the process order)

Advantage:
  - Only one API for `ShardingSphereDataSource` which should not confuse
the users;
  - Process logics internally, pretty high performance for this solution;
  - Can be reused for Proxy.

Disadvantage:
  - Difficult for developers, they need to know every hooks.

Plan B:

Provide independent DataSources, every DataSources process oneself only.
And we may have lots of DataSource, such as ShardingDataSource,
MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
TransactionDataSource, OrchestrationDataSource and so on.
Every DataSources should composite by users themselves, ShardingSphere can
privide a Facade class to simplify the usage.

Advantage:
  - Isolation by design level, every DataSource has clear and simple
responsibility;
  - Simple for developer, they need to know independent logic for one
DataSource only, do not need expose internal implementations.

Disadvantage:
  - Can not reuse by Proxy;
  - Difficult for user to find every possible combination of DataSource;
  - Hide the internal implementation may cause low performance;

By the way, I just discuss about how to design Proxy for Plan B. It should
be every features have a independent proxy process (may consider about put
them into one process in future).
ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
the performance should not enough.

Conclusion:
Even the Plan B has better design, but I worry about the performance and
the difficulty for user. So I prefer Plan A for now.

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo


guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：

> +1 It sounds great.
> And is the platform suitable for JDTX? And Sharding may not be a necessary
> function, maybe transaction will be more widely used. So, will JDTX will be
> a plugin which is easily used in the platform?
>
> Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
>
> > +1, That sounds great!
> >
> >
> > For a pluggable platform, contributors will be very convenient to
> > contribute code, This will attract more people.
> >
> >
> > Zonglei Dong
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
> > +1
> >
> >
> > Good idea. It’s a good long-term planning for ShardingSphere.
> >
> >
> > ―――――――――
> > Haisheng Sun (sunbufu)
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
> > Agree with Liang,
> >
> >
> > Maybe there is no feature benefit apparently, however we make its
> > architecture become…a structured plaza with many rooms. Consequently,
> more
> > contributors could fill in specific implements for those rooms with less
> > harms to this plaza, i.e our project. Oh, i have a great imagination,
> don’t
> > you think so? :)
> >
> >
> > Regards,
> > Trista
> >
> >
> > Juan Pan
> >
> >
> > panjuan@apache.org
> > Juan Pan(Trista), Apache ShardingSphere
> >
> >
> > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
> > Hi guys,
> > What features will be planned in pluggable platform? And how to assign
> > features splits from sharding-core? Based on user concept，what does user
> > benefit from this pluggable platform?
> >
> > Thanks,
> > Nicholas Jiang
> >
> > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
> > wrote:
> > Hi, ShardingSphere community,
> >
> > More and more features are added into ShardingSphere now, as you know,
> the
> > scope of ShardingSphere is no longer for sharding only.
> > There are more and more features related with sharding, such as
> distributed
> > transaction, distributed orchestration, observability and so on; and
> there
> > are couple of features did not relate with sharding obviously, for
> example:
> > encrypt, shadow data source, SQL audit and so on.
> >
> > I'd like to discuss about establishing a pluggable platform of
> > ShardingSphere. The proposal of pluggable platform is decoupling all
> > features and technical implementations, the benefits are:
> >
> > 1. Flexible for add new feature.
> > 2. Reduce the negative effects if problem occur on one feature.
> > 3. Provide a platform to make more contributors work together without
> > interact on each other.
> >
> > I plan split all features from sharding-core first, and then use SPI to
> > introduce features into the pluggable platform(same thing with sharding,
> > sharding can be remove from main process too).
> >
> > The pluggable platform is a blank JDBC and database protocol finally,
> and
> > provide assist technical features such as SQL parser and SQL rewrite.
> >
> > Any advice?
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
> >
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by "zhangliang@apache.org" <zh...@apache.org>.

I have 2 ideas for the pluggable platform.

Plan A:

Design a unify DataSource to process all rules, it should be a transparent
DataSource if no rule provided (but it should be valid). We can name it as
`ShardingSphereDataSource`.
There are lots of hooks in `ShardingSphereDataSource`, such as:
  - route hook: for sharding, master-slave and shadow;
  - SQL rewrite hook: for sharding, encrypt and shadow;
  - execution hook: for sharding and transaction;
  - merge hook: for sharding and encrypt;
  - init hook: for orchestration;
  - config refresh hook: for orchestration;

The hook should load feature implementations via SPI and appended
by pluggable platform automatically. (SPI should design the process order)

Advantage:
  - Only one API for `ShardingSphereDataSource` which should not confuse
the users;
  - Process logics internally, pretty high performance for this solution;
  - Can be reused for Proxy.

Disadvantage:
  - Difficult for developers, they need to know every hooks.

Plan B:

Provide independent DataSources, every DataSources process oneself only.
And we may have lots of DataSource, such as ShardingDataSource,
MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
TransactionDataSource, OrchestrationDataSource and so on.
Every DataSources should composite by users themselves, ShardingSphere can
privide a Facade class to simplify the usage.

Advantage:
  - Isolation by design level, every DataSource has clear and simple
responsibility;
  - Simple for developer, they need to know independent logic for one
DataSource only, do not need expose internal implementations.

Disadvantage:
  - Can not reuse by Proxy;
  - Difficult for user to find every possible combination of DataSource;
  - Hide the internal implementation may cause low performance;

By the way, I just discuss about how to design Proxy for Plan B. It should
be every features have a independent proxy process (may consider about put
them into one process in future).
ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
the performance should not enough.

Conclusion:
Even the Plan B has better design, but I worry about the performance and
the difficulty for user. So I prefer Plan A for now.

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo


guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：

> +1 It sounds great.
> And is the platform suitable for JDTX? And Sharding may not be a necessary
> function, maybe transaction will be more widely used. So, will JDTX will be
> a plugin which is easily used in the platform?
>
> Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
>
> > +1, That sounds great!
> >
> >
> > For a pluggable platform, contributors will be very convenient to
> > contribute code, This will attract more people.
> >
> >
> > Zonglei Dong
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
> > +1
> >
> >
> > Good idea. It’s a good long-term planning for ShardingSphere.
> >
> >
> > —————————
> > Haisheng Sun (sunbufu)
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
> > Agree with Liang,
> >
> >
> > Maybe there is no feature benefit apparently, however we make its
> > architecture become…a structured plaza with many rooms. Consequently,
> more
> > contributors could fill in specific implements for those rooms with less
> > harms to this plaza, i.e our project. Oh, i have a great imagination,
> don’t
> > you think so? :)
> >
> >
> > Regards,
> > Trista
> >
> >
> > Juan Pan
> >
> >
> > panjuan@apache.org
> > Juan Pan(Trista), Apache ShardingSphere
> >
> >
> > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
> > Hi guys,
> > What features will be planned in pluggable platform? And how to assign
> > features splits from sharding-core? Based on user concept，what does user
> > benefit from this pluggable platform?
> >
> > Thanks,
> > Nicholas Jiang
> >
> > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
> > wrote:
> > Hi, ShardingSphere community,
> >
> > More and more features are added into ShardingSphere now, as you know,
> the
> > scope of ShardingSphere is no longer for sharding only.
> > There are more and more features related with sharding, such as
> distributed
> > transaction, distributed orchestration, observability and so on; and
> there
> > are couple of features did not relate with sharding obviously, for
> example:
> > encrypt, shadow data source, SQL audit and so on.
> >
> > I'd like to discuss about establishing a pluggable platform of
> > ShardingSphere. The proposal of pluggable platform is decoupling all
> > features and technical implementations, the benefits are:
> >
> > 1. Flexible for add new feature.
> > 2. Reduce the negative effects if problem occur on one feature.
> > 3. Provide a platform to make more contributors work together without
> > interact on each other.
> >
> > I plan split all features from sharding-core first, and then use SPI to
> > introduce features into the pluggable platform(same thing with sharding,
> > sharding can be remove from main process too).
> >
> > The pluggable platform is a blank JDBC and database protocol finally,
> and
> > provide assist technical features such as SQL parser and SQL rewrite.
> >
> > Any advice?
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
> >
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by guangyuan wang <wa...@apache.org>.

+1 It sounds great.
And is the platform suitable for JDTX? And Sharding may not be a necessary
function, maybe transaction will be more widely used. So, will JDTX will be
a plugin which is easily used in the platform?

Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：

> +1, That sounds great!
>
>
> For a pluggable platform, contributors will be very convenient to
> contribute code, This will attract more people.
>
>
> Zonglei Dong
> Apache ShardingSphere
>
>
> On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
> +1
>
>
> Good idea. It’s a good long-term planning for ShardingSphere.
>
>
> —————————
> Haisheng Sun (sunbufu)
> Apache ShardingSphere
>
>
> On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
> Agree with Liang,
>
>
> Maybe there is no feature benefit apparently, however we make its
> architecture become…a structured plaza with many rooms. Consequently, more
> contributors could fill in specific implements for those rooms with less
> harms to this plaza, i.e our project. Oh, i have a great imagination, don’t
> you think so? :)
>
>
> Regards,
> Trista
>
>
> Juan Pan
>
>
> panjuan@apache.org
> Juan Pan(Trista), Apache ShardingSphere
>
>
> On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
> Hi guys,
> What features will be planned in pluggable platform? And how to assign
> features splits from sharding-core? Based on user concept，what does user
> benefit from this pluggable platform?
>
> Thanks,
> Nicholas Jiang
>
> On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
> wrote:
> Hi, ShardingSphere community,
>
> More and more features are added into ShardingSphere now, as you know, the
> scope of ShardingSphere is no longer for sharding only.
> There are more and more features related with sharding, such as distributed
> transaction, distributed orchestration, observability and so on; and there
> are couple of features did not relate with sharding obviously, for example:
> encrypt, shadow data source, SQL audit and so on.
>
> I'd like to discuss about establishing a pluggable platform of
> ShardingSphere. The proposal of pluggable platform is decoupling all
> features and technical implementations, the benefits are:
>
> 1. Flexible for add new feature.
> 2. Reduce the negative effects if problem occur on one feature.
> 3. Provide a platform to make more contributors work together without
> interact on each other.
>
> I plan split all features from sharding-core first, and then use SPI to
> introduce features into the pluggable platform(same thing with sharding,
> sharding can be remove from main process too).
>
> The pluggable platform is a blank JDBC and database protocol finally,  and
> provide assist technical features such as SQL parser and SQL rewrite.
>
> Any advice?
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by Zonglei Dong <dz...@163.com>.

+1, That sounds great!

For a pluggable platform, contributors will be very convenient to contribute code, This will attract more people.

Zonglei Dong
Apache ShardingSphere

On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
+1

Good idea. It’s a good long-term planning for ShardingSphere.

—————————
Haisheng Sun (sunbufu)
Apache ShardingSphere

On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
Agree with Liang,

Maybe there is no feature benefit apparently, however we make its architecture become…a structured plaza with many rooms. Consequently, more contributors could fill in specific implements for those rooms with less harms to this plaza, i.e our project. Oh, i have a great imagination, don’t you think so? :)

Regards,
Trista

Juan Pan

panjuan@apache.org
Juan Pan(Trista), Apache ShardingSphere

On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
Hi guys,
What features will be planned in pluggable platform? And how to assign features splits from sharding-core? Based on user concept，what does user benefit from this pluggable platform?

Thanks,
Nicholas Jiang

On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org> wrote:
Hi, ShardingSphere community,

More and more features are added into ShardingSphere now, as you know, the
scope of ShardingSphere is no longer for sharding only.
There are more and more features related with sharding, such as distributed
transaction, distributed orchestration, observability and so on; and there
are couple of features did not relate with sharding obviously, for example:
encrypt, shadow data source, SQL audit and so on.

I'd like to discuss about establishing a pluggable platform of
ShardingSphere. The proposal of pluggable platform is decoupling all
features and technical implementations, the benefits are:

1. Flexible for add new feature.
2. Reduce the negative effects if problem occur on one feature.
3. Provide a platform to make more contributors work together without
interact on each other.

I plan split all features from sharding-core first, and then use SPI to
introduce features into the pluggable platform(same thing with sharding,
sharding can be remove from main process too).

The pluggable platform is a blank JDBC and database protocol finally, and
provide assist technical features such as SQL parser and SQL rewrite.

Any advice?

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo

Re: Discussion of pluggable platform of ShardingSphere

Posted by sunbufu <su...@apache.org>.

Good idea. It’s a good long-term planning for ShardingSphere.

—————————
Haisheng Sun (sunbufu)
Apache ShardingSphere

On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
Agree with Liang,

Regards,
Trista

Juan Pan

panjuan@apache.org
Juan Pan(Trista), Apache ShardingSphere

Thanks,
Nicholas Jiang

On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org> wrote:
Hi, ShardingSphere community,

I'd like to discuss about establishing a pluggable platform of
ShardingSphere. The proposal of pluggable platform is decoupling all
features and technical implementations, the benefits are:

1. Flexible for add new feature.
2. Reduce the negative effects if problem occur on one feature.
3. Provide a platform to make more contributors work together without
interact on each other.

I plan split all features from sharding-core first, and then use SPI to
introduce features into the pluggable platform(same thing with sharding,
sharding can be remove from main process too).

The pluggable platform is a blank JDBC and database protocol finally, and
provide assist technical features such as SQL parser and SQL rewrite.

Any advice?

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo

Re: Discussion of pluggable platform of ShardingSphere

Posted by Juan Pan <pa...@apache.org>.

Agree with Liang,

Regards,
Trista

Juan Pan

panjuan@apache.org
Juan Pan(Trista), Apache ShardingSphere

Thanks,
Nicholas Jiang

On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org> wrote:
Hi, ShardingSphere community,

I'd like to discuss about establishing a pluggable platform of
ShardingSphere. The proposal of pluggable platform is decoupling all
features and technical implementations, the benefits are:

1. Flexible for add new feature.
2. Reduce the negative effects if problem occur on one feature.
3. Provide a platform to make more contributors work together without
interact on each other.

I plan split all features from sharding-core first, and then use SPI to
introduce features into the pluggable platform(same thing with sharding,
sharding can be remove from main process too).

The pluggable platform is a blank JDBC and database protocol finally, and
provide assist technical features such as SQL parser and SQL rewrite.

Any advice?

------------------

Liang Zhang (John)
Apache ShardingSphere & Dubbo

Re: Discussion of pluggable platform of ShardingSphere

Posted by Nicholas <th...@gmail.com>.

Hi guys,
    What features will be planned in pluggable platform? And how to assign features splits from sharding-core? Based on user concept，what does user benefit from this pluggable platform?

Thanks,
Nicholas Jiang

On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org> wrote: 
> Hi, ShardingSphere community,
> 
> More and more features are added into ShardingSphere now, as you know, the
> scope of ShardingSphere is no longer for sharding only.
> There are more and more features related with sharding, such as distributed
> transaction, distributed orchestration, observability and so on; and there
> are couple of features did not relate with sharding obviously, for example:
> encrypt, shadow data source, SQL audit and so on.
> 
> I'd like to discuss about establishing a pluggable platform of
> ShardingSphere. The proposal of pluggable platform is decoupling all
> features and technical implementations, the benefits are:
> 
> 1. Flexible for add new feature.
> 2. Reduce the negative effects if problem occur on one feature.
> 3. Provide a platform to make more contributors work together without
> interact on each other.
> 
> I plan split all features from sharding-core first, and then use SPI to
> introduce features into the pluggable platform(same thing with sharding,
> sharding can be remove from main process too).
> 
> The pluggable platform is a blank JDBC and database protocol finally,  and
> provide assist technical features such as SQL parser and SQL rewrite.
> 
> Any advice?
> 
> ------------------
> 
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>

Re: Discussion of pluggable platform of ShardingSphere

Posted by zhaojun <zh...@126.com>.

+1

Pluggable data platform would be a brand new start.
Looking forward to combining  variety powerful features through SPI, not just inside the sharding-core.

------------------
Zhao Jun (cherrylzhao)
Apache ShardingSphere & ServiceComb

> On Dec 9, 2019, at 1:29 PM, zhangliang@apache.org wrote:
> 
> Hi, ShardingSphere community,
> 
> More and more features are added into ShardingSphere now, as you know, the
> scope of ShardingSphere is no longer for sharding only.
> There are more and more features related with sharding, such as distributed
> transaction, distributed orchestration, observability and so on; and there
> are couple of features did not relate with sharding obviously, for example:
> encrypt, shadow data source, SQL audit and so on.
> 
> I'd like to discuss about establishing a pluggable platform of
> ShardingSphere. The proposal of pluggable platform is decoupling all
> features and technical implementations, the benefits are:
> 
> 1. Flexible for add new feature.
> 2. Reduce the negative effects if problem occur on one feature.
> 3. Provide a platform to make more contributors work together without
> interact on each other.
> 
> I plan split all features from sharding-core first, and then use SPI to
> introduce features into the pluggable platform(same thing with sharding,
> sharding can be remove from main process too).
> 
> The pluggable platform is a blank JDBC and database protocol finally,  and
> provide assist technical features such as SQL parser and SQL rewrite.
> 
> Any advice?
> 
> ------------------
> 
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo