[Discuess] How to implement the getChildren api for some orchestration framework, like nacos, etcd.

[Zonglei Dong <do...@apache.org>]

Hi ShardingShpere community,


We are refactoring the ShardingSphere's orchestration module, the branch is "orchestration-5.x".


the issue's link is: https://github.com/apache/incubator-shardingsphere/issues/3185 .


Now we meet a problem and want to discuss it.


The problem is that ShardingSphere use "Zookeeper" as a RegistryCenter, and Zookeeper support "getChildren" api to get one key's children info.


At ShardingSphere, some modules use the “getChildren" api, for example, the Sharding-Proxy use “getChildren" api to load all schema data.


the "getChildren" api's use case for ShardingSphere, reference this link: http://ww1.sinaimg.cn/large/7dad8649ly1gaxg6ti728j22rn1gmnbd.jpg .


But for other framework， like "apollo", “nacos" and “etcd", don't support “getChildren" api.


the solution that we discuss does add an additional key to store the relationship between orchestration config keys.


e.g. the key would be like "shardingsphere.orchestration.key.relationship", and the value is json format.


for example is, json's key is parent path and json's value are children nodes.


 {
    "/orchestration_ds":[
        "state",
        "config"
    ],
    "/orchestration_ds/state":[
        "instances",
        "datasources"
    ],
    "/orchestration_ds/config":[
        "schema"
    ],
    "/orchestration_ds/config/schema":[
        "db_test"
    ],
    "/orchestration_ds/config/schema/db_test":[
        "rule",
        "datasource"
    ]
}


If ShardingSphere creates key, the program will divide two step, first parse and save the relationship and then save the key's data.


There maybe cause another serious problem, if maintain two keys at the same time, maybe one is success and another is failure, cause data inconsistency.


We want to save the relationship firstly and then save the key's data. If user start up system or modify one key, we will check all keys relationship and make them correctly.


If data is not consistency, the program will print log and user must deal the error data manually.


Do you think this is a good way or have any other good suggestions, Thanks!


Best wishes!
Zonglei Dong
Apache ShardingSphere

Re: [Discuess] How to implement the getChildren api for some orchestration framework, like nacos, etcd.

[guangyuan wang <wa...@apache.org>]

In the case, "If data is not consistent, the program will print log and
user must deal the error data manually."
 I think if data is not consistent, the program could delete keys which
does not exist in the relationship and then prints related log.

Zonglei Dong <do...@apache.org> 于2020年1月15日周三 下午8:44写道：

> Hi ShardingShpere community,
>
>
> We are refactoring the ShardingSphere's orchestration module, the branch
> is "orchestration-5.x".
>
>
> the issue's link is:
> https://github.com/apache/incubator-shardingsphere/issues/3185 .
>
>
> Now we meet a problem and want to discuss it.
>
>
> The problem is that ShardingSphere use "Zookeeper" as a RegistryCenter,
> and Zookeeper support "getChildren" api to get one key's children info.
>
>
> At ShardingSphere, some modules use the “getChildren" api, for example,
> the Sharding-Proxy use “getChildren" api to load all schema data.
>
>
> the "getChildren" api's use case for ShardingSphere, reference this link:
> http://ww1.sinaimg.cn/large/7dad8649ly1gaxg6ti728j22rn1gmnbd.jpg .
>
>
> But for other framework， like "apollo", “nacos" and “etcd", don't support
> “getChildren" api.
>
>
> the solution that we discuss does add an additional key to store the
> relationship between orchestration config keys.
>
>
> e.g. the key would be like
> "shardingsphere.orchestration.key.relationship", and the value is json
> format.
>
>
> for example is, json's key is parent path and json's value are children
> nodes.
>
>
>  {
>     "/orchestration_ds":[
>         "state",
>         "config"
>     ],
>     "/orchestration_ds/state":[
>         "instances",
>         "datasources"
>     ],
>     "/orchestration_ds/config":[
>         "schema"
>     ],
>     "/orchestration_ds/config/schema":[
>         "db_test"
>     ],
>     "/orchestration_ds/config/schema/db_test":[
>         "rule",
>         "datasource"
>     ]
> }
>
>
> If ShardingSphere creates key, the program will divide two step, first
> parse and save the relationship and then save the key's data.
>
>
> There maybe cause another serious problem, if maintain two keys at the
> same time, maybe one is success and another is failure, cause data
> inconsistency.
>
>
> We want to save the relationship firstly and then save the key's data. If
> user start up system or modify one key, we will check all keys relationship
> and make them correctly.
>
>
> If data is not consistency, the program will print log and user must deal
> the error data manually.
>
>
> Do you think this is a good way or have any other good suggestions, Thanks!
>
>
> Best wishes!
> Zonglei Dong
> Apache ShardingSphere
>
>