You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by ki...@apache.org on 2020/07/02 08:37:38 UTC
[shardingsphere] branch master updated: translate the solution from
solution.cn.md to solution.en.md (#6229)
This is an automated email from the ASF dual-hosted git repository.
kimmking pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new 47322ac translate the solution from solution.cn.md to solution.en.md (#6229)
47322ac is described below
commit 47322ac7074df09bf13d345036e99988b8e3088b
Author: mlw12345678 <40...@users.noreply.github.com>
AuthorDate: Thu Jul 2 16:37:18 2020 +0800
translate the solution from solution.cn.md to solution.en.md (#6229)
---
docs/blog/content/material/solution.en.md | 250 +++++++++++++++++++++++++++++-
1 file changed, 249 insertions(+), 1 deletion(-)
diff --git a/docs/blog/content/material/solution.en.md b/docs/blog/content/material/solution.en.md
index 1d51f1d..3cc60ab 100644
--- a/docs/blog/content/material/solution.en.md
+++ b/docs/blog/content/material/solution.en.md
@@ -4,4 +4,252 @@ weight = 9
chapter = true
+++
-## TODO
+## The mixed open-source distributed transaction solution
+
+### Author
+
+Liang Zhang，Head of Data R&D of JD.com, initiator of Apache ShardingSphere & PPMC
+
+Love open source, currently leading open source projects ShardingSphere (formerly known as Sharding-JDBC) and Elastic-Job. Good at using Java as the main distributed architecture and Kubernetes and Mesos as the main cloud platform, admiring elegant code, and having more research on how to write expressive code.
+
+At present, the main energy is invested in building ShardingSphere into the industry's first-class financial data solution. ShardingSphere has entered the Apache incubator, is the first open source project of the Jingdong Group to enter the Apache Foundation, and is also the first distributed database middleware of the Apache Foundation.
+
+---
+Ning Jiang，Technical expert of Huawei Open Source Competency Center, project leader of Apache ServiceComb. Former chief software engineer of Red Hat Software, he has more than ten years of experience in enterprise-level open source middleware development, rich experience in Java development and use, and he also is the enthusiast of functional programming. Since 2006, he has been engaged in the development of the Apache open source middleware project, and has participated in the developme [...]
+
+Blog address：https://willemjiang.github.io/
+
+---
+
+Zheng Feng is a software engineer at Red Hat. Joined Red Hat Software in 2009, mainly engaged in the work of the transaction manager. As a core developer, he participated in the Narayan and Blacktie projects. He had contributed to the integration of transaction processing of multiple application servers (Wildfly, Karaf, Tomcat) and frameworks (Common DBCP, Spring Boot ). Since 2017, he has participated in the Apache ServiceComb project and is currently a member of PMC. He has in-depth re [...]
+
+### Guide
+
+Compared with the gradual maturity of data sharding solutions, distributed transaction solutions that combine performance, transparency, automation, strong consistency, and can be applied to various application scenarios in one are very rare. Based on the performance bottlenecks of distributed transactions submitted in two (three) stages and the business transformation of flexible transactions, distributed transactions are still a headache for architects.
+
+At the beginning of 2019, Apache ShardingSphere (Incubating) provided a rigid and flexible integrated distributed transaction solution. If your application system is being troubled by this aspect, why not pour a cup of coffee and spend ten minutes reading this article, maybe you will gain something?
+
+### Background
+
+Database transactions need to meet the four characteristics of ACID (Atomicity, Consistency, Isolation, Durability).
+
+- Atomicity Refers to the execution of the transaction as a whole, either all or no execution.
+
+- Consistency Refers to the transaction should ensure that the data changes from one consistent state to another consistent state.
+
+- Isolation When multiple transactions are executed concurrently, the execution of one transaction should not affect the execution of other transactions.
+
+- Durability Refers to the committed transaction modification data will be persisted.
+
+In a single data node, the transaction is limited to the access control of a single database resource, called local transaction. Almost all mature relational databases provide native support for local transactions. However, in a distributed application environment based on microservices, more and more application scenarios require that access to multiple services and corresponding multiple database resources can be incorporated into the same transaction, then the distributed transactions [...]
+
+Although the relational database provides perfect ACID native support for local transactions. But in a distributed scenario, it has become a shackle in system performance. How to make the database meet the characteristics of ACID or find corresponding alternatives in a distributed scenario is the focus of distributed transactions.
+
+#### Local transaction
+
+Without opening any distributed transaction manager, let each data node manage its own transaction. There is no coordination and communication between them, and they do not know each other's success of other data node transactions. There is no loss in performance for local transactions, but it is inadequate in terms of strong consistency and eventual consistency.
+
+#### Two-phase commit
+
+The earliest distributed transaction model of the XA protocol is the X/Open Distributed Transaction Processing (DTP) model proposed by the X/Open International Alliance, referred to as the XA protocol.
+
+The distributed transaction based on the XA protocol has little intrusion on the business. Its biggest advantage is that it is transparent to the user. Users can use distributed transactions based on the XA protocol like local transactions. The XA protocol can strictly guarantee the ACID characteristics of transactions.
+
+Strictly guaranteeing ACID characteristics of transactions is a double-edged sword. In the process of transaction execution, all required resources need to be locked, so the XA protocol is more suitable for short transactions whose execution time is determined. For long transactions, the exclusive use of data during the entire transaction will cause the concurrency performance of business systems that rely on hot data to decline significantly. Therefore, in high-concurrency performance-o [...]
+
+#### Flexible transaction
+
+If the transaction that implements the ACID transaction element is called a rigid transaction, the transaction based on the BASE transaction element is called a flexible transaction. BASE is an abbreviation of the three elements of basic availability, flexible state and final consistency.
+
+- Basically Available Ensure that participants in distributed transactions are not necessarily online at the same time.
+
+- Soft state It allows a certain delay in the system status update, and this delay may not be noticeable to customers.
+
+- Eventually consistent Usually, the final consistency of the system is ensured by means of message passing.
+
+In ACID transactions, the requirements for consistency and isolation are very high. During the execution of the transaction, all resources must be occupied. The idea of flexible transactions is to move the mutex operation from the resource level to the business level through business logic. By relaxing the requirements for strong consistency and isolation, only when the entire transaction ends, the data is consistent. During the execution of the transaction, any data obtained by the read [...]
+
+Saga is a typical flexible transaction manager. The concept of Sagas comes from a database paper more than thirty years ago [http://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf], a Saga transaction is A long-term transaction which is composed of multiple short-term transactions. In the distributed transaction scenario, we regard a Saga distributed transaction as a transaction composed of multiple local transactions, and each local transaction has a corresponding compensation t [...]
+
+TCC (Try-Cancel/Confirm implementation) is another kind of flexible transaction coordination implementation. TCC provides a more perfect recovery method with the help of a two-phase submission agreement. In TCC mode, cancel compensation obviously needs to execute business logic in the second stage to cancel the consequences of the first stage. Try is to perform related business operations in the first stage to complete the occupation of related business resources, such as pre-allocating [...]
+
+The strong consistency transaction based on ACID and the final consistency transaction based on BASE are not silver bullets, and their greatest strengths can only be used in the most suitable scenarios. The following table can be used to compare the differences between them in detail to help developers choose technologies.
+
+<center>
+
+
+| Contrast | Local Transaction | Two-phase Submission | Flexible Transaction |
+| :---------------------: | :---------------------------------------: | :---------------------------------: | :---------------------------------: |
+| Business Transformation | No need | No need | Implement related interfaces |
+| Consistency | Not support | Support | Final consistency |
+| Isolation | Not support | Support | Business guarantee |
+| Concurrent Performance | No effect | Severe decline | Slight decline |
+| Suitable Scenarios | Inconsistent processing by business party | Short transaction & Low concurrency | Long transaction & High concurrency |
+
+</center>
+
+#### Challenge
+
+Due to different application scenarios, developers need to be able to reasonably choose various distributed transactions between performance and functionality.
+
+The API and functions of the two-phase commit and flexible transactions are not exactly the same, and there is no free and transparent switching between them. In the development decision-making stage, you have to choose between transactions submitted in two stages and flexible transactions, which greatly increases the cost of design and development.
+
+The XA-based two-phase commit transaction is relatively simple to use, but it cannot deal well with the Internet's high concurrency or long transaction scenarios of complex systems; flexible transactions require developers to transform applications, access costs are very high, and developers are required Implement resource occupancy and reverse compensation on your own.
+
+### Distributed transactions of ShardingSphere
+
+Integrate existing mature transaction solutions, provide a unified distributed transaction interface for local transactions, two-phase commit and flexible transactions, and make up for the shortcomings of the current solution, the main design goal of the Apache ShardingSphere (Incubating) distribution transactional module is providing a one-stop distributed transaction solution. The name of the module is sharding-transaction. The three key words of combination of hardness and softness, a [...]
+
+1.The combination of hardness and softness
+
+At the same time provide XA-based two-phase commit transaction and Saga-based flexible transaction solution, and can be used together.
+
+2.Automation
+
+XA transactions and Saga transactions are completed in an automated manner, and the user has no awareness. XA transactions do not require the use of XADataSource interface and JTA transaction manager; Saga transactions also do not require users to implement compensation interfaces themselves.
+
+3.Transparency
+
+In the two access points of Apache ShardingSphere (Incubating), Sharding-JDBC and Sharding-Proxy, respectively provide the encapsulation for the local transaction interface. The user can fully use the multiple data sources of horizontal sharding managed by ShardingSphere as one database, and the complete distributed transaction capability can be achieved through the local transaction API. Users can transparently switch transaction types in the application.
+
+The sharding-transaction module consists of three sub-modules: sharding-transaction-core, sharding-transaction-2pc and sharding-transaction-base.
+
+- sharding-transaction-core:
+
+Provides APIs for users and SPIs for developers.
+
+- sharding-transaction-2pc:
+
+The parent module of two-stage commit transaction. Currently only the sharding-transaction-xa module provides XA protocol support. In the future, more types of transactions based on two-phase commit will be introduced, such as: percolator, see:
+
+[https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36726.pdf]。
+
+- sharding-transaction-base:
+
+The parent module of flexible transaction. Currently, there is only the sharding-transaction-saga module, which uses the Saga executor provided by Apache ServiceComb Saga Actuator to provide flexible transaction support, and on the basis of it provides reverse SQL and snapshot capabilities, and thus realizes automatic reverse compensation.
+
+The function highlights of ShardingSphere's XA and Saga transaction modules will be described below.
+
+#### XA transaction-three XA transaction managers escort together
+
+There are many mature XA transaction managers. Apache ShardingSphere (Incubating) did not choose to reinvent the wheel. Instead, it hoped to create an ecology that organically integrates the appropriate wheels to provide mature and stable distributed transaction processing capabilities. Its main functions are as follows:
+
+**1.Reuse mature engine, automatically switch the underlying implementation**
+
+The Sharding-transaction-xa module further defines the SPI for XA transaction manager developers. Developers only need to implement the SPI-defined interface to automatically join the Apache ShardingSphere (Incubating) ecosystem as their XA transaction manager.
+
+Apache ShardingSphere (Incubating) officially implements SPI based on Atomikos and Bitronix, and invited the Radhat JBoss XA transaction engine Narayana [https://github.com/jbosstm/narayana] development team to implement JBoss SPI. Users can choose their favorite XA transaction manager among Atomikos, Bitronix and Narayana.
+
+
+
+Limited by the license of the Apache Foundation project, Apache ShardingSphere (Incubating) uses Atomikos of the Apache protocol as its default implementation. Regarding Bitronix based on the LGPL protocol and Narayana based on the LGPL protocol, users can refer to the corresponding jar package to the classpath of project.
+
+
+
+If these three XA transaction managers still do not meet user needs, developers can implement customized XA transaction managers by extending SPI.
+
+
+
+**2.Automatic access to transparent data sources**
+
+
+
+Apache ShardingSphere (Incubating) can automatically connect XADataSource as a database-driven data source to the XA transaction manager. For applications that use DataSource as a database driver, users do not need to change their encoding and configuration. Apache ShardingSphere (Incubating) converts it into XADataSource and XAConnection that support the XA protocol through automatic adaptation. It is registered as an XA resource in the underlying XA transaction manager.
+
+
+
+The architecture diagram of the XA module is as follows：
+
+
+
+
+#### Saga transactions—overcome the limitations of flexible transactions and realize automatic compensation
+
+In flexible transactions, each update operation to the database will actually submit the data to the database to achieve the best resource release effect in a highly concurrent system. When data needs to be rolled back, the flexible transaction manager maintains the final consistency of the data and the isolation behavior. Apache ShardingSphere (Incubating) uses Apache ServiceComb Saga Actuator [https://github.com/apache/servicecomb-saga-actuator] as the Saga transaction manager. Its mai [...]
+
+
+
+**1. Automatic reverse compensation**
+
+
+
+Saga defines that each sub-transaction in a transaction has a corresponding reverse compensation operation. The Saga transaction manager generates a directed acyclic graph based on the program execution results, and when the rollback operation needs to be performed, reverse compensation operations are called in reverse order according to the graph. The Saga transaction manager is only used to control when to retry, appropriate compensation, and is not responsible for the content of compe [...]
+
+
+
+Another flexible transaction manager, TCC, is similar to the Saga concept, and both require compensation operations from the user developer. In addition to compensation, TCC also provides the ability to occupy resources, but it also needs to be provided by the developer of the user. Although functionally stronger than Saga, the cost of using TCC is also higher than that of Saga.
+
+
+
+The user developer provides resource occupancy and compensation operations, which makes it difficult for flexible transaction solutions to be implemented in business systems on a large scale. And because of the intervention of the business system, the use scope of the flexible transaction framework has always been positioned as a service rather than a database. The mature flexible transaction manager that the database can directly use is still rare.
+
+
+
+Apache ShardingSphere (Incubating) uses reverse SQL technology to automatically generate data snapshots and reverse SQL for the SQL that updates the database, and it is executed by Apache ServiceComb Saga Actuator. The user does not need to pay attention to how to implement the compensation method. The application category of the transaction manager is successfully positioned back to the source of the transaction -- the database level.
+
+
+
+For the Apache ShardingSphere (Incubating) SQL parsing engine that can process complex query statements, the difficulty of parsing statements such as insert/update/delete is much smaller; ShardingSphere intercepts the SQL executed by the user to perform data sharding. All SQL Can be directly controlled by it. Therefore, the combination of reverse SQL and compensation capabilities with Apache ServiceComb Saga Actuator achieves the ability to automate flexible transactions, which is a mode [...]
+
+
+
+The architecture diagram of the Saga module is as follows：
+
+
+
+#### Access point -- Distributed transaction for native transaction interface
+
+The goal of Apache ShardingSphere (Incubating) is to use sharded multi-database like a database. In the transaction module, this goal is still applicable. No matter how fragmented the database managed by ShardingSphere, there is always only one logical database for developers. Therefore, ShardingSphere's transaction interface is still the native local transaction interface, namely the setAutoCommit, commit and rollback methods of JDBC's java.sql.Connection; and the begin, commit and roll [...]
+
+
+
+Since the native transaction interface does not support transaction types, ShardingSphere provides 3 ways for users to switch transaction types.
+
+
+
+1.Switch the current transaction type through SCTL (sharding-ctl, the database management command provided by ShardingSphere). Just enter it in SQL execution mode, and it is applicable to Sharding-JDBC and Sharding-Proxy. For example: SCTL:SET TRANSACTION_TYPE=BASE
+
+2.Switch the current transaction type through Threadlocal, suitable for Sharding-JDBC. For example: TransactionTypeHolder.set (TransactionType.XA)
+
+3.Through meta-annotation, and used with Spring to switch the current transaction type, suitable for Sharding-JDBC and Sharding-Proxy. For example: @ShardingTransactionType (TransactionType.BASE)
+
+### Future plan
+
+The distributed transaction module in the development branch of github [https://github.com/apache/incubator-shardingsphere] is basically available and will be released with the version of 4.0.0.M1, which will also be the first release version when ShardingSphere into the Apache Foundation incubator. Distributed transactions are an important part of data sharding and microservice architecture. They are also the focus of Apache ShardingSphere (Incubating). After the release, they will cont [...]
+
+#### Transaction isolation engine
+
+After the SQL reverse engine is stabilized, the focus of flexible transactions will be on creating transaction isolation. Since the isolation of transactions is not the scope of Saga's plan, Apache ShardingSphere (Incubating) will improve it outside of Saga, together with the SQL reverse engine as an integral part of the entire flexible transaction.
+
+
+
+Apache ShardingSphere (Incubating) will support the isolation level of read committed, read uncommitted, repeatable read, and serialization through several strategies such as optimistic locking, pessimistic locking, and no isolation. And through the multi-version snapshot to further improve the concurrency of the system.
+
+#### External XA transaction interface
+
+The two access ends of Apache ShardingSphere (Incubating), Sharding-JDBC and Sharding-Proxy, after supporting their own internal transaction issues, will provide the ability to integrate with other data sources to be managed by distributed transaction manager such as JTA.
+
+
+
+After the external XA transaction interface is implemented, the Sharding-JDBC DataSource will implement the XADataSource interface, providing the possibility to join with other data sources in an XA transaction; Sharding-Proxy's database protocol will also implement a two-stage XA-based submission protocol; Let it become the resource manager loaded by XA.
+
+
+
+In addition, ShardingSphere will also implement the recovery part of the XA protocol, that is, when the transaction processor crashes, it can provide in-doubt transactions to achieve transaction recovery.
+
+
+### summary
+
+The distributed transaction capabilities provided by Apache ShardingSphere (Incubating) can be summarized by the following table. Readers may wish to compare with the table at the beginning of the article to see the changes brought by ShardingSphere's distributed transaction module.
+
+
+| Contrast | Local Transaction | Two-phase Submission | Flexible Transaction |
+| :---------------------: | :---------------------------------------: | :---------------------------------: | :---------------------------------: |
+| Business Transformation | No need | No need | <font color=#ff0000>No need</font> |
+| Consistency | Not support | Support | Final consistency |
+| Isolation | Not support | Support | <font color=#ff0000>Planning</font> |
+| Concurrent Performance | No effect | Severe decline | Slight decline |
+| Suitable Scenarios | Inconsistent processing by business party | Short transaction & Low concurrency | Long transaction & High concurrency |
+
+In the fast-developing Apache ShardingSphere (Incubating), the prototype of distributed transactions has been established. We will build it into a usable product as soon as possible and continue to provide quality solutions to the community. For an article that is not short, after reading this article, I believe you must be interested in this field. Let’s try it first. Does it meet your expectations? Or simply join our community to create a more complete distributed transaction solution.
+