You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shardingsphere.apache.org by zhaojun <zh...@126.com> on 2020/01/15 04:26:48 UTC
[DISCUSS] Optimize model of ColumnMetaData
Hi, all
I have created a issue[1] for discussing about optimizing metadata model.
Currently ShardingSphere metadata was only used by sharding & encrypt scenario,
master-slave case have to get them from underlying ResultSet meta in sharding-proxy.
It is better to establish an global MetaData mechanism which provide
uniform interface for initialization, read, refreshing instead of distinguish with usage scenario
I think it would be a big refactor for sharding-jdbc & sharding-proxy.
assume that task list is as below
1. investigate the usage of current metadata, clarify the function list of metadata (SQLParse, Route,Rewrite etc)
2. design a MetaData Model & MetaData service, at first only consider about storing in memory
3. then consider about migrating to zookeeper or other registration
[1]: https://github.com/apache/incubator-shardingsphere/issues/3922 <https://github.com/apache/incubator-shardingsphere/issues/3922>
Any thought?
------------------
Zhao Jun (cherrylzhao)
Apache ShardingSphere & ServiceComb
Re:Re: [DISCUSS] Optimize model of ColumnMetaData
Posted by KimmKing <ki...@apache.org>.
We have persist meta data of logic schema to center repository like zk. And add a global notification to all proxy/jdbc nodes when a ddl statement executed.
More details also see:
[1] [DISCUSS]MetadataCenter Design 5.x https://github.com/apache/incubator-shardingsphere/issues/4896
At 2020-01-20 22:11:13, "guangyuan wang" <wa...@apache.org> wrote:
>For byte[] value, I think we could find a way to resolve.
>But refresh metadata without affecting running instance, I am not sure.
>Registry Center provides a watching function, does it could be used to
>refresh metadata?
>
>zhaojun <zh...@126.com> 于2020年1月20日周一 下午5:23写道：
>
>> If we store raw column metadata into orchestration, byte[] value should be
>> supported in orchestration.
>> metaData could be serialized into byte array using kryo framework etc
>>
>> Also we should consider about refresh metadata without affecting running
>> instance.
>>
>> ------------------
>> Zhao Jun (cherrylzhao)
>> Apache ShardingSphere & ServiceComb
>>
>> > On Jan 20, 2020, at 5:02 PM, guangyuan wang <wa...@apache.org>
>> wrote:
>> >
>> > What kind of help do you need in the orchestration module? The
>> > orchestration module provides functions: writing and reading key-value
>> pair.
>> >
>> > Nicholas <th...@gmail.com> 于2020年1月17日周五 下午4:02写道：
>> >
>> >> Hi zhaojun,
>> >> About ColumnMeta storage, we need ShardingSphere's orchestration
>> module
>> >> to help register column metadata. And at best establish metadata module
>> to
>> >> manage metadata of table, column etc. Field property like primary key
>> etc
>> >> should be put into ColumnMeta to maintain. At last, could you please
>> >> provide the design document of MetaData Model & MetaData service, and
>> take
>> >> distributed scenario into consideration.
>> >>
>> >> Thanks,
>> >> Nicholas
>> >>
>> >> On 2020/01/15 04:26:48, zhaojun <zh...@126.com> wrote:
>> >>> Hi, all
>> >>>
>> >>> I have created a issue[1] for discussing about optimizing metadata
>> model.
>> >>>
>> >>> Currently ShardingSphere metadata was only used by sharding & encrypt
>> >> scenario,
>> >>> master-slave case have to get them from underlying ResultSet meta in
>> >> sharding-proxy.
>> >>> It is better to establish an global MetaData mechanism which provide
>> >>> uniform interface for initialization, read, refreshing instead of
>> >> distinguish with usage scenario
>> >>>
>> >>> I think it would be a big refactor for sharding-jdbc & sharding-proxy.
>> >>> assume that task list is as below
>> >>>
>> >>> 1. investigate the usage of current metadata, clarify the function list
>> >> of metadata (SQLParse, Route,Rewrite etc)
>> >>> 2. design a MetaData Model & MetaData service, at first only consider
>> >> about storing in memory
>> >>> 3. then consider about migrating to zookeeper or other registration
>> >>>
>> >>> [1]: https://github.com/apache/incubator-shardingsphere/issues/3922 <
>> >> https://github.com/apache/incubator-shardingsphere/issues/3922>
>> >>>
>> >>> Any thought?
>> >>>
>> >>> ------------------
>> >>> Zhao Jun (cherrylzhao)
>> >>> Apache ShardingSphere & ServiceComb
>> >>>
>> >>>
>> >>
>>
>>
Re: [DISCUSS] Optimize model of ColumnMetaData
Posted by zhaojun <zh...@126.com>.
Yes, using watch function of register center has the minimal side effects to current architecture.
Metadata in local memory will refresh according to the refresh event.
------------------
Zhao Jun (cherrylzhao)
Apache ShardingSphere & ServiceComb
> On Jan 20, 2020, at 10:11 PM, guangyuan wang <wa...@apache.org> wrote:
>
> For byte[] value, I think we could find a way to resolve.
> But refresh metadata without affecting running instance, I am not sure.
> Registry Center provides a watching function, does it could be used to
> refresh metadata?
>
> zhaojun <zh...@126.com> 于2020年1月20日周一 下午5:23写道：
>
>> If we store raw column metadata into orchestration, byte[] value should be
>> supported in orchestration.
>> metaData could be serialized into byte array using kryo framework etc
>>
>> Also we should consider about refresh metadata without affecting running
>> instance.
>>
>> ------------------
>> Zhao Jun (cherrylzhao)
>> Apache ShardingSphere & ServiceComb
>>
>>> On Jan 20, 2020, at 5:02 PM, guangyuan wang <wa...@apache.org>
>> wrote:
>>>
>>> What kind of help do you need in the orchestration module? The
>>> orchestration module provides functions: writing and reading key-value
>> pair.
>>>
>>> Nicholas <th...@gmail.com> 于2020年1月17日周五 下午4:02写道：
>>>
>>>> Hi zhaojun,
>>>> About ColumnMeta storage, we need ShardingSphere's orchestration
>> module
>>>> to help register column metadata. And at best establish metadata module
>> to
>>>> manage metadata of table, column etc. Field property like primary key
>> etc
>>>> should be put into ColumnMeta to maintain. At last, could you please
>>>> provide the design document of MetaData Model & MetaData service, and
>> take
>>>> distributed scenario into consideration.
>>>>
>>>> Thanks,
>>>> Nicholas
>>>>
>>>> On 2020/01/15 04:26:48, zhaojun <zh...@126.com> wrote:
>>>>> Hi, all
>>>>>
>>>>> I have created a issue[1] for discussing about optimizing metadata
>> model.
>>>>>
>>>>> Currently ShardingSphere metadata was only used by sharding & encrypt
>>>> scenario,
>>>>> master-slave case have to get them from underlying ResultSet meta in
>>>> sharding-proxy.
>>>>> It is better to establish an global MetaData mechanism which provide
>>>>> uniform interface for initialization, read, refreshing instead of
>>>> distinguish with usage scenario
>>>>>
>>>>> I think it would be a big refactor for sharding-jdbc & sharding-proxy.
>>>>> assume that task list is as below
>>>>>
>>>>> 1. investigate the usage of current metadata, clarify the function list
>>>> of metadata (SQLParse, Route,Rewrite etc)
>>>>> 2. design a MetaData Model & MetaData service, at first only consider
>>>> about storing in memory
>>>>> 3. then consider about migrating to zookeeper or other registration
>>>>>
>>>>> [1]: https://github.com/apache/incubator-shardingsphere/issues/3922 <
>>>> https://github.com/apache/incubator-shardingsphere/issues/3922>
>>>>>
>>>>> Any thought?
>>>>>
>>>>> ------------------
>>>>> Zhao Jun (cherrylzhao)
>>>>> Apache ShardingSphere & ServiceComb
>>>>>
>>>>>
>>>>
>>
>>
Re: [DISCUSS] Optimize model of ColumnMetaData
Posted by guangyuan wang <wa...@apache.org>.
For byte[] value, I think we could find a way to resolve.
But refresh metadata without affecting running instance, I am not sure.
Registry Center provides a watching function, does it could be used to
refresh metadata?
zhaojun <zh...@126.com> 于2020年1月20日周一 下午5:23写道：
> If we store raw column metadata into orchestration, byte[] value should be
> supported in orchestration.
> metaData could be serialized into byte array using kryo framework etc
>
> Also we should consider about refresh metadata without affecting running
> instance.
>
> ------------------
> Zhao Jun (cherrylzhao)
> Apache ShardingSphere & ServiceComb
>
> > On Jan 20, 2020, at 5:02 PM, guangyuan wang <wa...@apache.org>
> wrote:
> >
> > What kind of help do you need in the orchestration module? The
> > orchestration module provides functions: writing and reading key-value
> pair.
> >
> > Nicholas <th...@gmail.com> 于2020年1月17日周五 下午4:02写道：
> >
> >> Hi zhaojun,
> >> About ColumnMeta storage, we need ShardingSphere's orchestration
> module
> >> to help register column metadata. And at best establish metadata module
> to
> >> manage metadata of table, column etc. Field property like primary key
> etc
> >> should be put into ColumnMeta to maintain. At last, could you please
> >> provide the design document of MetaData Model & MetaData service, and
> take
> >> distributed scenario into consideration.
> >>
> >> Thanks,
> >> Nicholas
> >>
> >> On 2020/01/15 04:26:48, zhaojun <zh...@126.com> wrote:
> >>> Hi, all
> >>>
> >>> I have created a issue[1] for discussing about optimizing metadata
> model.
> >>>
> >>> Currently ShardingSphere metadata was only used by sharding & encrypt
> >> scenario,
> >>> master-slave case have to get them from underlying ResultSet meta in
> >> sharding-proxy.
> >>> It is better to establish an global MetaData mechanism which provide
> >>> uniform interface for initialization, read, refreshing instead of
> >> distinguish with usage scenario
> >>>
> >>> I think it would be a big refactor for sharding-jdbc & sharding-proxy.
> >>> assume that task list is as below
> >>>
> >>> 1. investigate the usage of current metadata, clarify the function list
> >> of metadata (SQLParse, Route,Rewrite etc)
> >>> 2. design a MetaData Model & MetaData service, at first only consider
> >> about storing in memory
> >>> 3. then consider about migrating to zookeeper or other registration
> >>>
> >>> [1]: https://github.com/apache/incubator-shardingsphere/issues/3922 <
> >> https://github.com/apache/incubator-shardingsphere/issues/3922>
> >>>
> >>> Any thought?
> >>>
> >>> ------------------
> >>> Zhao Jun (cherrylzhao)
> >>> Apache ShardingSphere & ServiceComb
> >>>
> >>>
> >>
>
>
Re: [DISCUSS] Optimize model of ColumnMetaData
Posted by zhaojun <zh...@126.com>.
If we store raw column metadata into orchestration, byte[] value should be supported in orchestration.
metaData could be serialized into byte array using kryo framework etc
Also we should consider about refresh metadata without affecting running instance.
------------------
Zhao Jun (cherrylzhao)
Apache ShardingSphere & ServiceComb
> On Jan 20, 2020, at 5:02 PM, guangyuan wang <wa...@apache.org> wrote:
>
> What kind of help do you need in the orchestration module? The
> orchestration module provides functions: writing and reading key-value pair.
>
> Nicholas <th...@gmail.com> 于2020年1月17日周五 下午4:02写道：
>
>> Hi zhaojun,
>> About ColumnMeta storage, we need ShardingSphere's orchestration module
>> to help register column metadata. And at best establish metadata module to
>> manage metadata of table, column etc. Field property like primary key etc
>> should be put into ColumnMeta to maintain. At last, could you please
>> provide the design document of MetaData Model & MetaData service, and take
>> distributed scenario into consideration.
>>
>> Thanks,
>> Nicholas
>>
>> On 2020/01/15 04:26:48, zhaojun <zh...@126.com> wrote:
>>> Hi, all
>>>
>>> I have created a issue[1] for discussing about optimizing metadata model.
>>>
>>> Currently ShardingSphere metadata was only used by sharding & encrypt
>> scenario,
>>> master-slave case have to get them from underlying ResultSet meta in
>> sharding-proxy.
>>> It is better to establish an global MetaData mechanism which provide
>>> uniform interface for initialization, read, refreshing instead of
>> distinguish with usage scenario
>>>
>>> I think it would be a big refactor for sharding-jdbc & sharding-proxy.
>>> assume that task list is as below
>>>
>>> 1. investigate the usage of current metadata, clarify the function list
>> of metadata (SQLParse, Route,Rewrite etc)
>>> 2. design a MetaData Model & MetaData service, at first only consider
>> about storing in memory
>>> 3. then consider about migrating to zookeeper or other registration
>>>
>>> [1]: https://github.com/apache/incubator-shardingsphere/issues/3922 <
>> https://github.com/apache/incubator-shardingsphere/issues/3922>
>>>
>>> Any thought?
>>>
>>> ------------------
>>> Zhao Jun (cherrylzhao)
>>> Apache ShardingSphere & ServiceComb
>>>
>>>
>>
Re: [DISCUSS] Optimize model of ColumnMetaData
Posted by guangyuan wang <wa...@apache.org>.
What kind of help do you need in the orchestration module? The
orchestration module provides functions: writing and reading key-value pair.
Nicholas <th...@gmail.com> 于2020年1月17日周五 下午4:02写道：
> Hi zhaojun,
> About ColumnMeta storage, we need ShardingSphere's orchestration module
> to help register column metadata. And at best establish metadata module to
> manage metadata of table, column etc. Field property like primary key etc
> should be put into ColumnMeta to maintain. At last, could you please
> provide the design document of MetaData Model & MetaData service, and take
> distributed scenario into consideration.
>
> Thanks,
> Nicholas
>
> On 2020/01/15 04:26:48, zhaojun <zh...@126.com> wrote:
> > Hi, all
> >
> > I have created a issue[1] for discussing about optimizing metadata model.
> >
> > Currently ShardingSphere metadata was only used by sharding & encrypt
> scenario,
> > master-slave case have to get them from underlying ResultSet meta in
> sharding-proxy.
> > It is better to establish an global MetaData mechanism which provide
> > uniform interface for initialization, read, refreshing instead of
> distinguish with usage scenario
> >
> > I think it would be a big refactor for sharding-jdbc & sharding-proxy.
> > assume that task list is as below
> >
> > 1. investigate the usage of current metadata, clarify the function list
> of metadata (SQLParse, Route,Rewrite etc)
> > 2. design a MetaData Model & MetaData service, at first only consider
> about storing in memory
> > 3. then consider about migrating to zookeeper or other registration
> >
> > [1]: https://github.com/apache/incubator-shardingsphere/issues/3922 <
> https://github.com/apache/incubator-shardingsphere/issues/3922>
> >
> > Any thought?
> >
> > ------------------
> > Zhao Jun (cherrylzhao)
> > Apache ShardingSphere & ServiceComb
> >
> >
>
Re: [DISCUSS] Optimize model of ColumnMetaData
Posted by Nicholas <th...@gmail.com>.
Hi zhaojun,
About ColumnMeta storage, we need ShardingSphere's orchestration module to help register column metadata. And at best establish metadata module to manage metadata of table, column etc. Field property like primary key etc should be put into ColumnMeta to maintain. At last, could you please provide the design document of MetaData Model & MetaData service, and take distributed scenario into consideration.
Thanks,
Nicholas
On 2020/01/15 04:26:48, zhaojun <zh...@126.com> wrote:
> Hi, all
>
> I have created a issue[1] for discussing about optimizing metadata model.
>
> Currently ShardingSphere metadata was only used by sharding & encrypt scenario,
> master-slave case have to get them from underlying ResultSet meta in sharding-proxy.
> It is better to establish an global MetaData mechanism which provide
> uniform interface for initialization, read, refreshing instead of distinguish with usage scenario
>
> I think it would be a big refactor for sharding-jdbc & sharding-proxy.
> assume that task list is as below
>
> 1. investigate the usage of current metadata, clarify the function list of metadata (SQLParse, Route,Rewrite etc)
> 2. design a MetaData Model & MetaData service, at first only consider about storing in memory
> 3. then consider about migrating to zookeeper or other registration
>
> [1]: https://github.com/apache/incubator-shardingsphere/issues/3922 <https://github.com/apache/incubator-shardingsphere/issues/3922>
>
> Any thought?
>
> ------------------
> Zhao Jun (cherrylzhao)
> Apache ShardingSphere & ServiceComb
>
>