You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2020/02/28 11:57:03 UTC

[GitHub] [incubator-shardingsphere] kimmking opened a new issue #4522: [DISCUSS]Orchestration 5.x Design

kimmking opened a new issue #4522: [DISCUSS]Orchestration 5.x Design
URL: https://github.com/apache/incubator-shardingsphere/issues/4522

### Orchestration Design
[TOC]

This document focuses on the design of the new orchestration module of shardingsphere version 5.0.0.

#### Overview
Orchestration is designed as the distributed governance module of shardingsphere platform.
In the future, it will be renamed as governance or governor module, according to the development and improvement of the module.

#### Planning
This module includes the following sub modules:
- Config Center: manage system configuration paramters
- Registry center: manage the service registration, discovery and coordination capabilities of the system
- Metadata Center: manage core metadata information used by each node (todo)

There are several governance components in other modules:
- Circuit break: in the Sharding-JDBC orchestration module, it can disable a library instance or break a node's access.
- APM integration: in the Sharding-Openingtracing module, it integrates with APM.
- Data Masking: in the Encrypt-Core module, it provides the function of encrypting to output sensitive data.
- Shadow table: in the Shadow-Core module, it provide some specified tables as shadow tables. This is a new feature.

Features of cluster management and distributed coordinator in further planning.

The current version of orchestration focuses on the capability design and implementation of the above three centers.

![image](https://user-images.githubusercontent.com/807508/75408853-fa9c6d00-5951-11ea-9fe7-0cc7ad51e0bd.png)

#### Structure
Current projects structure of orchestration module:

```yaml
- sharding-orchestration
- sharding-orchestration-center
- sharding-orchestration-center-api
- sharding-orchestration-center-apollo
- sharding-orchestration-center-nacos
- sharding-orchestration-center-zookeeper-curator
- sharding-orchestration-core
```

New structure:

```yaml
- sharding-orchestration
- sharding-orchestration-center
- sharding-orchestration-center-spi
- sharding-orchestration-center-model
- sharding-orchestration-center-configuration
- sharding-orchestration-center-apollo
- sharding-orchestration-center-nacos
- sharding-orchestration-center-zookeeper(-curator) *
- sharding-orchestration-center-etcd
- sharding-orchestration-core
- sharding-orchestration-core-configuration
- sharding-orchestration-core-model * #may be moved to underly project
- sharding-orchestration-core-common
-...

```

We need clarify several concepts here:
- The `config` in Config-Center is related to the `configuration` read from the configuration file into XXConfiguration class instance in runtime, but the two things are not the same. That should be clarified clearly. The `configuration` is only used once when the configuration file is used to launch our node, proxy etc. The `config` is used dynamically during the system runtime, and needs to maintain the state and own behavior all the time in system lifecycle.

- There are three types of centers, config-center,registry-center and metadata-center. We can use zookeeper, etcd, nacos and other thirdparty library to implement one or all of them. But in practice, we can only use one implementation for the certain center. Therefore, we can provide two base implementations supported by default, for long term maintenance.

Center support matrix for popular thirdparty libraries in planning, as follow:

| thirdparty library | default | ConfigCenter enable | RegistryCenter enable | MetadataCenter enbale |
| ------------ | :------: | :----------: | :----------: | :------------: |
| zookeeper | √ | √ | √ | √ |
| etcd | √ | √ | √ | √ |
| apollo | | √ | | |
| nacos | | √ | √ | |
| redis | | √ | | |
| eureka | | | √ | |
| consul | | | √ | |

Some topic should be noted and further discussed later:
- Migrate/partially migrate the components in other modules or not?
- How to choose the implementation of curator and native client for zookeeper center?
- Is the thirdparty library allowed to be configured independently(so that more than one modules can reference it), rather than inside the center?

#### Center Design

Based on the above analysis, our design principles for centers can be summarized as follows:
- Flexibility: the design of each center is independent of the specific implement library, so the implementation can be replaced flexibly.
- Encapsulation: we need to encapsulate the domain models and core logic, which can be used by different implementations of each center.

Therefore, we propose that what needs to implement a common base, not a complete and specific config center. It can be an adapter with some behavioral capabilities. In this way, we need to distinguish the domain models from the adapter instance. On another side, a unified set of domain models should be defined and maintained, the common operating logic of modules is the same.

Note that there maybe more and more centers later, they just need implement the SPI: CenterRepository. Contributors who implements centers don't need learn the domain models details and how many centers there are. They can focus on integration with thirdparty library only.

##### Domain models

The preliminary design is that domain model includes three main objects:
- ShardingSphereConfig -> ConfigCenter
- ShardingSphereState -> RegistryCenter
- ShardingSphereMetadata - > MetadataCenter

Let's explains them:
1. ShardingSphereConfig corresponds to the configuration model of the configuration center. These config parameters are used when the system starts and are updated or adjusted when parameters changed. The config parameters are that they have no impact on the core business capabilities of the system, and the requirements for consistency and real-time are not so high (relatively to others).
2. ShardingSphereState corresponds to the state model of the whole system. For example, when a new node starts/stops completed, joins a cluster/group, it can show us which nodes are available now. This model is useful to manage and control the behavior of the whole cluster. The state data is that they are continuously generated in the process with system running. Generally, they don't need to be saved before cluster stop/restart. If they are missing/lost, cluster will take a consistent recovery in time.
3. ShardingSphereMetadata corresponds to the core metadata information of whole system/cluster. If they are wrong or out of date, will cause whole system/cluster error or critical data error. High consistency requirements.

##### CenterRepository implememt

We extract an interface of adapter concept: `CenterRepository`, adapt the integration libraries, and provide several capabilities:
- The ability to get configuration data from the outter services by key: get/getchildren method
- The ability to persist data to a outter service: persist method
- The ability to notify and invoke callback when related data changed: watch method

![image](https://user-images.githubusercontent.com/807508/75407961-4d285a00-594f-11ea-995e-f3cc8b4b39e1.png)

##### Runtime

1. When Sharding-Proxy starting, loading the configuration from the local yaml files, or Java/Spring config, and then construct the runtime data model in memory based on those configuration.
2. If there is a ConfigCenter configured, the proxy will serialize the runtime data model into yaml format, store it in ConfigCenter(via CenterRepository get/persist method). And then retrieve the configuration data, monitor data changes through ConfigCenter(via CenterRepository watch method).
3. Each proxy node will register itself as an instance to the registry center, and it could be managed and controlled when state changed (such as enabling and disabling).

![image](https://user-images.githubusercontent.com/807508/75407914-2a964100-594f-11ea-8b2c-ded75634c060.png)

#### Parameter configuration

For existing parameters, see the Haoran's list.
- Organize and optimize parameters(todo)

#### Task list

1.API design
- 1.1 CenterRepository design (done)
- Extract CenterRepository interface
- Refactoring ConfigCenterRepository
- Refactoring the RegistryCenterRepository
- Adjust unit tests and UI backend references
- 1.2 distributed_lock_manager and leaf removal (done)
- Integrate third-party leaf and migrated to opensharding
- Remove the distributed_lock_manager and leaf implementation
- Optimize the CenterRepository interface
- Sync examples and documents
- 1.3 config domain object design (in progress)
- 1.4 registry domain object design (in progress)
2. Project structure design
- 2.1 configuration project (in progress)
- Add configuration project
- Extract existing code to configuration project
- Improve unit tests
- 2.2 model design (todo)
- Add model project
- Extract existing code to model project
- Refactoring model project code and design
- Improve unit tests
- Update references to other projects
3. Metadata design (todo)
4. Center etcd implements or migration (todo)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services