You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shardingsphere.apache.org by "zhangliang@apache.org" <zh...@apache.org> on 2020/01/02 16:13:07 UTC
Re: Discussion of pluggable platform of ShardingSphere
I have 2 ideas for the pluggable platform.
Plan A:
Design a unify DataSource to process all rules, it should be a transparent
DataSource if no rule provided (but it should be valid). We can name it as
`ShardingSphereDataSource`.
There are lots of hooks in `ShardingSphereDataSource`, such as:
- route hook: for sharding, master-slave and shadow;
- SQL rewrite hook: for sharding, encrypt and shadow;
- execution hook: for sharding and transaction;
- merge hook: for sharding and encrypt;
- init hook: for orchestration;
- config refresh hook: for orchestration;
The hook should load feature implementations via SPI and appended
by pluggable platform automatically. (SPI should design the process order)
Advantage:
- Only one API for `ShardingSphereDataSource` which should not confuse
the users;
- Process logics internally, pretty high performance for this solution;
- Can be reused for Proxy.
Disadvantage:
- Difficult for developers, they need to know every hooks.
Plan B:
Provide independent DataSources, every DataSources process oneself only.
And we may have lots of DataSource, such as ShardingDataSource,
MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
TransactionDataSource, OrchestrationDataSource and so on.
Every DataSources should composite by users themselves, ShardingSphere can
privide a Facade class to simplify the usage.
Advantage:
- Isolation by design level, every DataSource has clear and simple
responsibility;
- Simple for developer, they need to know independent logic for one
DataSource only, do not need expose internal implementations.
Disadvantage:
- Can not reuse by Proxy;
- Difficult for user to find every possible combination of DataSource;
- Hide the internal implementation may cause low performance;
By the way, I just discuss about how to design Proxy for Plan B. It should
be every features have a independent proxy process (may consider about put
them into one process in future).
ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
the performance should not enough.
Conclusion:
Even the Plan B has better design, but I worry about the performance and
the difficulty for user. So I prefer Plan A for now.
------------------
Liang Zhang (John)
Apache ShardingSphere & Dubbo
guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：
> +1 It sounds great.
> And is the platform suitable for JDTX? And Sharding may not be a necessary
> function, maybe transaction will be more widely used. So, will JDTX will be
> a plugin which is easily used in the platform?
>
> Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
>
> > +1, That sounds great!
> >
> >
> > For a pluggable platform, contributors will be very convenient to
> > contribute code, This will attract more people.
> >
> >
> > Zonglei Dong
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
> > +1
> >
> >
> > Good idea. It’s a good long-term planning for ShardingSphere.
> >
> >
> > —————————
> > Haisheng Sun (sunbufu)
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
> > Agree with Liang,
> >
> >
> > Maybe there is no feature benefit apparently, however we make its
> > architecture become…a structured plaza with many rooms. Consequently,
> more
> > contributors could fill in specific implements for those rooms with less
> > harms to this plaza, i.e our project. Oh, i have a great imagination,
> don’t
> > you think so? :)
> >
> >
> > Regards,
> > Trista
> >
> >
> > Juan Pan
> >
> >
> > panjuan@apache.org
> > Juan Pan(Trista), Apache ShardingSphere
> >
> >
> > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
> > Hi guys,
> > What features will be planned in pluggable platform? And how to assign
> > features splits from sharding-core? Based on user concept，what does user
> > benefit from this pluggable platform?
> >
> > Thanks,
> > Nicholas Jiang
> >
> > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
> > wrote:
> > Hi, ShardingSphere community,
> >
> > More and more features are added into ShardingSphere now, as you know,
> the
> > scope of ShardingSphere is no longer for sharding only.
> > There are more and more features related with sharding, such as
> distributed
> > transaction, distributed orchestration, observability and so on; and
> there
> > are couple of features did not relate with sharding obviously, for
> example:
> > encrypt, shadow data source, SQL audit and so on.
> >
> > I'd like to discuss about establishing a pluggable platform of
> > ShardingSphere. The proposal of pluggable platform is decoupling all
> > features and technical implementations, the benefits are:
> >
> > 1. Flexible for add new feature.
> > 2. Reduce the negative effects if problem occur on one feature.
> > 3. Provide a platform to make more contributors work together without
> > interact on each other.
> >
> > I plan split all features from sharding-core first, and then use SPI to
> > introduce features into the pluggable platform(same thing with sharding,
> > sharding can be remove from main process too).
> >
> > The pluggable platform is a blank JDBC and database protocol finally,
> and
> > provide assist technical features such as SQL parser and SQL rewrite.
> >
> > Any advice?
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
> >
>
Re: Discussion of pluggable platform of ShardingSphere
Posted by Zheng Feng <zh...@gmail.com>.
Hi Liang,
The Plan B looks like similar to the netty pipeline architecture and will
be much more flexible.
Regards,
Zheng Feng
zhangliang@apache.org <zh...@apache.org> 于2020年1月3日周五 上午12:13写道：
> I have 2 ideas for the pluggable platform.
>
> Plan A:
>
> Design a unify DataSource to process all rules, it should be a transparent
> DataSource if no rule provided (but it should be valid). We can name it as
> `ShardingSphereDataSource`.
> There are lots of hooks in `ShardingSphereDataSource`, such as:
> - route hook: for sharding, master-slave and shadow;
> - SQL rewrite hook: for sharding, encrypt and shadow;
> - execution hook: for sharding and transaction;
> - merge hook: for sharding and encrypt;
> - init hook: for orchestration;
> - config refresh hook: for orchestration;
>
> The hook should load feature implementations via SPI and appended
> by pluggable platform automatically. (SPI should design the process order)
>
> Advantage:
> - Only one API for `ShardingSphereDataSource` which should not confuse
> the users;
> - Process logics internally, pretty high performance for this solution;
> - Can be reused for Proxy.
>
> Disadvantage:
> - Difficult for developers, they need to know every hooks.
>
> Plan B:
>
> Provide independent DataSources, every DataSources process oneself only.
> And we may have lots of DataSource, such as ShardingDataSource,
> MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
> TransactionDataSource, OrchestrationDataSource and so on.
> Every DataSources should composite by users themselves, ShardingSphere can
> privide a Facade class to simplify the usage.
>
> Advantage:
> - Isolation by design level, every DataSource has clear and simple
> responsibility;
> - Simple for developer, they need to know independent logic for one
> DataSource only, do not need expose internal implementations.
>
> Disadvantage:
> - Can not reuse by Proxy;
> - Difficult for user to find every possible combination of DataSource;
> - Hide the internal implementation may cause low performance;
>
> By the way, I just discuss about how to design Proxy for Plan B. It should
> be every features have a independent proxy process (may consider about put
> them into one process in future).
> ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
> the performance should not enough.
>
> Conclusion:
> Even the Plan B has better design, but I worry about the performance and
> the difficulty for user. So I prefer Plan A for now.
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
>
> guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：
>
> > +1 It sounds great.
> > And is the platform suitable for JDTX? And Sharding may not be a
> necessary
> > function, maybe transaction will be more widely used. So, will JDTX will
> be
> > a plugin which is easily used in the platform?
> >
> > Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
> >
> > > +1, That sounds great!
> > >
> > >
> > > For a pluggable platform, contributors will be very convenient to
> > > contribute code, This will attract more people.
> > >
> > >
> > > Zonglei Dong
> > > Apache ShardingSphere
> > >
> > >
> > > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
> > > +1
> > >
> > >
> > > Good idea. It’s a good long-term planning for ShardingSphere.
> > >
> > >
> > > —————————
> > > Haisheng Sun (sunbufu)
> > > Apache ShardingSphere
> > >
> > >
> > > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
> > > Agree with Liang,
> > >
> > >
> > > Maybe there is no feature benefit apparently, however we make its
> > > architecture become…a structured plaza with many rooms. Consequently,
> > more
> > > contributors could fill in specific implements for those rooms with
> less
> > > harms to this plaza, i.e our project. Oh, i have a great imagination,
> > don’t
> > > you think so? :)
> > >
> > >
> > > Regards,
> > > Trista
> > >
> > >
> > > Juan Pan
> > >
> > >
> > > panjuan@apache.org
> > > Juan Pan(Trista), Apache ShardingSphere
> > >
> > >
> > > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
> > > Hi guys,
> > > What features will be planned in pluggable platform? And how to assign
> > > features splits from sharding-core? Based on user concept，what does
> user
> > > benefit from this pluggable platform?
> > >
> > > Thanks,
> > > Nicholas Jiang
> > >
> > > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zhangliang@apache.org
> >
> > > wrote:
> > > Hi, ShardingSphere community,
> > >
> > > More and more features are added into ShardingSphere now, as you know,
> > the
> > > scope of ShardingSphere is no longer for sharding only.
> > > There are more and more features related with sharding, such as
> > distributed
> > > transaction, distributed orchestration, observability and so on; and
> > there
> > > are couple of features did not relate with sharding obviously, for
> > example:
> > > encrypt, shadow data source, SQL audit and so on.
> > >
> > > I'd like to discuss about establishing a pluggable platform of
> > > ShardingSphere. The proposal of pluggable platform is decoupling all
> > > features and technical implementations, the benefits are:
> > >
> > > 1. Flexible for add new feature.
> > > 2. Reduce the negative effects if problem occur on one feature.
> > > 3. Provide a platform to make more contributors work together without
> > > interact on each other.
> > >
> > > I plan split all features from sharding-core first, and then use SPI to
> > > introduce features into the pluggable platform(same thing with
> sharding,
> > > sharding can be remove from main process too).
> > >
> > > The pluggable platform is a blank JDBC and database protocol finally,
> > and
> > > provide assist technical features such as SQL parser and SQL rewrite.
> > >
> > > Any advice?
> > >
> > > ------------------
> > >
> > > Liang Zhang (John)
> > > Apache ShardingSphere & Dubbo
> > >
> > >
> >
>
回复: Discussion of pluggable platform of ShardingSphere
Posted by Sun Kane <ka...@outlook.com>.
I personally recommend solution A, just like the current hardware system,although the hardware outside the mainboard is pluggable, users rarely plug hardware such like graphics cards or CPU.
reduce the difficulty for users to get started will the best
________________________________
发件人: zhangliang@apache.org <zh...@apache.org>
发送时间: 2020年1月3日 0:13
收件人: dev@shardingsphere.apache.org <de...@shardingsphere.apache.org>
主题: Re: Discussion of pluggable platform of ShardingSphere
I have 2 ideas for the pluggable platform.
Plan A:
Design a unify DataSource to process all rules, it should be a transparent
DataSource if no rule provided (but it should be valid). We can name it as
`ShardingSphereDataSource`.
There are lots of hooks in `ShardingSphereDataSource`, such as:
- route hook: for sharding, master-slave and shadow;
- SQL rewrite hook: for sharding, encrypt and shadow;
- execution hook: for sharding and transaction;
- merge hook: for sharding and encrypt;
- init hook: for orchestration;
- config refresh hook: for orchestration;
The hook should load feature implementations via SPI and appended
by pluggable platform automatically. (SPI should design the process order)
Advantage:
- Only one API for `ShardingSphereDataSource` which should not confuse
the users;
- Process logics internally, pretty high performance for this solution;
- Can be reused for Proxy.
Disadvantage:
- Difficult for developers, they need to know every hooks.
Plan B:
Provide independent DataSources, every DataSources process oneself only.
And we may have lots of DataSource, such as ShardingDataSource,
MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
TransactionDataSource, OrchestrationDataSource and so on.
Every DataSources should composite by users themselves, ShardingSphere can
privide a Facade class to simplify the usage.
Advantage:
- Isolation by design level, every DataSource has clear and simple
responsibility;
- Simple for developer, they need to know independent logic for one
DataSource only, do not need expose internal implementations.
Disadvantage:
- Can not reuse by Proxy;
- Difficult for user to find every possible combination of DataSource;
- Hide the internal implementation may cause low performance;
By the way, I just discuss about how to design Proxy for Plan B. It should
be every features have a independent proxy process (may consider about put
them into one process in future).
ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
the performance should not enough.
Conclusion:
Even the Plan B has better design, but I worry about the performance and
the difficulty for user. So I prefer Plan A for now.
------------------
Liang Zhang (John)
Apache ShardingSphere & Dubbo
guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：
> +1 It sounds great.
> And is the platform suitable for JDTX? And Sharding may not be a necessary
> function, maybe transaction will be more widely used. So, will JDTX will be
> a plugin which is easily used in the platform?
>
> Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
>
> > +1, That sounds great!
> >
> >
> > For a pluggable platform, contributors will be very convenient to
> > contribute code, This will attract more people.
> >
> >
> > Zonglei Dong
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
> > +1
> >
> >
> > Good idea. It’s a good long-term planning for ShardingSphere.
> >
> >
> > ―――――――――
> > Haisheng Sun (sunbufu)
> > Apache ShardingSphere
> >
> >
> > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
> > Agree with Liang,
> >
> >
> > Maybe there is no feature benefit apparently, however we make its
> > architecture become…a structured plaza with many rooms. Consequently,
> more
> > contributors could fill in specific implements for those rooms with less
> > harms to this plaza, i.e our project. Oh, i have a great imagination,
> don’t
> > you think so? :)
> >
> >
> > Regards,
> > Trista
> >
> >
> > Juan Pan
> >
> >
> > panjuan@apache.org
> > Juan Pan(Trista), Apache ShardingSphere
> >
> >
> > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
> > Hi guys,
> > What features will be planned in pluggable platform? And how to assign
> > features splits from sharding-core? Based on user concept，what does user
> > benefit from this pluggable platform?
> >
> > Thanks,
> > Nicholas Jiang
> >
> > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
> > wrote:
> > Hi, ShardingSphere community,
> >
> > More and more features are added into ShardingSphere now, as you know,
> the
> > scope of ShardingSphere is no longer for sharding only.
> > There are more and more features related with sharding, such as
> distributed
> > transaction, distributed orchestration, observability and so on; and
> there
> > are couple of features did not relate with sharding obviously, for
> example:
> > encrypt, shadow data source, SQL audit and so on.
> >
> > I'd like to discuss about establishing a pluggable platform of
> > ShardingSphere. The proposal of pluggable platform is decoupling all
> > features and technical implementations, the benefits are:
> >
> > 1. Flexible for add new feature.
> > 2. Reduce the negative effects if problem occur on one feature.
> > 3. Provide a platform to make more contributors work together without
> > interact on each other.
> >
> > I plan split all features from sharding-core first, and then use SPI to
> > introduce features into the pluggable platform(same thing with sharding,
> > sharding can be remove from main process too).
> >
> > The pluggable platform is a blank JDBC and database protocol finally,
> and
> > provide assist technical features such as SQL parser and SQL rewrite.
> >
> > Any advice?
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
> >
>
Re:Re: Discussion of pluggable platform of ShardingSphere
Posted by Sion Yang <sc...@163.com>.
I prefer Plan A too.
--
Yi Yang(Sion)
Apache ShardingSphere
At 2020-01-03 00:13:07, "zhangliang@apache.org" <zh...@apache.org> wrote:
>I have 2 ideas for the pluggable platform.
>
>Plan A:
>
>Design a unify DataSource to process all rules, it should be a transparent
>DataSource if no rule provided (but it should be valid). We can name it as
>`ShardingSphereDataSource`.
>There are lots of hooks in `ShardingSphereDataSource`, such as:
> - route hook: for sharding, master-slave and shadow;
> - SQL rewrite hook: for sharding, encrypt and shadow;
> - execution hook: for sharding and transaction;
> - merge hook: for sharding and encrypt;
> - init hook: for orchestration;
> - config refresh hook: for orchestration;
>
>The hook should load feature implementations via SPI and appended
>by pluggable platform automatically. (SPI should design the process order)
>
>Advantage:
> - Only one API for `ShardingSphereDataSource` which should not confuse
>the users;
> - Process logics internally, pretty high performance for this solution;
> - Can be reused for Proxy.
>
>Disadvantage:
> - Difficult for developers, they need to know every hooks.
>
>Plan B:
>
>Provide independent DataSources, every DataSources process oneself only.
>And we may have lots of DataSource, such as ShardingDataSource,
>MasterSlaveDataSource, EncryptDataSource, ShadowDataSource,
>TransactionDataSource, OrchestrationDataSource and so on.
>Every DataSources should composite by users themselves, ShardingSphere can
>privide a Facade class to simplify the usage.
>
>Advantage:
> - Isolation by design level, every DataSource has clear and simple
>responsibility;
> - Simple for developer, they need to know independent logic for one
>DataSource only, do not need expose internal implementations.
>
>Disadvantage:
> - Can not reuse by Proxy;
> - Difficult for user to find every possible combination of DataSource;
> - Hide the internal implementation may cause low performance;
>
>By the way, I just discuss about how to design Proxy for Plan B. It should
>be every features have a independent proxy process (may consider about put
>them into one process in future).
>ShardingProxy visit EncryptProxy, and EncryptProxy visit ShadowProxy,
>the performance should not enough.
>
>Conclusion:
>Even the Plan B has better design, but I worry about the performance and
>the difficulty for user. So I prefer Plan A for now.
>
>------------------
>
>Liang Zhang (John)
>Apache ShardingSphere & Dubbo
>
>
>guangyuan wang <wa...@apache.org> 于2019年12月11日周三 下午2:21写道：
>
>> +1 It sounds great.
>> And is the platform suitable for JDTX? And Sharding may not be a necessary
>> function, maybe transaction will be more widely used. So, will JDTX will be
>> a plugin which is easily used in the platform?
>>
>> Zonglei Dong <dz...@163.com> 于2019年12月9日周一 下午9:24写道：
>>
>> > +1, That sounds great!
>> >
>> >
>> > For a pluggable platform, contributors will be very convenient to
>> > contribute code, This will attract more people.
>> >
>> >
>> > Zonglei Dong
>> > Apache ShardingSphere
>> >
>> >
>> > On 12/9/2019 19:13，sunbufu<su...@apache.org> wrote：
>> > +1
>> >
>> >
>> > Good idea. It’s a good long-term planning for ShardingSphere.
>> >
>> >
>> > —————————
>> > Haisheng Sun (sunbufu)
>> > Apache ShardingSphere
>> >
>> >
>> > On 12/9/2019 18:43，Juan Pan<pa...@apache.org> wrote：
>> > Agree with Liang,
>> >
>> >
>> > Maybe there is no feature benefit apparently, however we make its
>> > architecture become…a structured plaza with many rooms. Consequently,
>> more
>> > contributors could fill in specific implements for those rooms with less
>> > harms to this plaza, i.e our project. Oh, i have a great imagination,
>> don’t
>> > you think so? :)
>> >
>> >
>> > Regards,
>> > Trista
>> >
>> >
>> > Juan Pan
>> >
>> >
>> > panjuan@apache.org
>> > Juan Pan(Trista), Apache ShardingSphere
>> >
>> >
>> > On 12/9/2019 14:13，Nicholas<th...@gmail.com> wrote：
>> > Hi guys,
>> > What features will be planned in pluggable platform? And how to assign
>> > features splits from sharding-core? Based on user concept，what does user
>> > benefit from this pluggable platform?
>> >
>> > Thanks,
>> > Nicholas Jiang
>> >
>> > On 2019/12/09 05:29:47, "zhangliang@apache.org" <zh...@apache.org>
>> > wrote:
>> > Hi, ShardingSphere community,
>> >
>> > More and more features are added into ShardingSphere now, as you know,
>> the
>> > scope of ShardingSphere is no longer for sharding only.
>> > There are more and more features related with sharding, such as
>> distributed
>> > transaction, distributed orchestration, observability and so on; and
>> there
>> > are couple of features did not relate with sharding obviously, for
>> example:
>> > encrypt, shadow data source, SQL audit and so on.
>> >
>> > I'd like to discuss about establishing a pluggable platform of
>> > ShardingSphere. The proposal of pluggable platform is decoupling all
>> > features and technical implementations, the benefits are:
>> >
>> > 1. Flexible for add new feature.
>> > 2. Reduce the negative effects if problem occur on one feature.
>> > 3. Provide a platform to make more contributors work together without
>> > interact on each other.
>> >
>> > I plan split all features from sharding-core first, and then use SPI to
>> > introduce features into the pluggable platform(same thing with sharding,
>> > sharding can be remove from main process too).
>> >
>> > The pluggable platform is a blank JDBC and database protocol finally,
>> and
>> > provide assist technical features such as SQL parser and SQL rewrite.
>> >
>> > Any advice?
>> >
>> > ------------------
>> >
>> > Liang Zhang (John)
>> > Apache ShardingSphere & Dubbo
>> >
>> >
>>