You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by me...@apache.org on 2022/01/28 14:32:39 UTC
[shardingsphere] branch master updated: Stress testing blog (#15158)
This is an automated email from the ASF dual-hosted git repository.
menghaoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new 3a617a9 Stress testing blog (#15158)
3a617a9 is described below
commit 3a617a9710a8f7ad74d06f5912b77257891e8fd7
Author: Yacine Si Tayeb <86...@users.noreply.github.com>
AuthorDate: Fri Jan 28 22:30:20 2022 +0800
Stress testing blog (#15158)
---
...ingSphere_Shadow_Database_Feature_Upgrade.en.md | 238 +++++++++++++++++++++
.../img/Full_Link_Online_Stress_Testing_img_1.png | Bin 0 -> 7747 bytes
.../img/Full_Link_Online_Stress_Testing_img_10.png | Bin 0 -> 19383 bytes
.../img/Full_Link_Online_Stress_Testing_img_2.png | Bin 0 -> 29778 bytes
.../img/Full_Link_Online_Stress_Testing_img_3.jpeg | Bin 0 -> 59470 bytes
.../img/Full_Link_Online_Stress_Testing_img_4.png | Bin 0 -> 52649 bytes
.../img/Full_Link_Online_Stress_Testing_img_5.jpg | Bin 0 -> 87875 bytes
.../img/Full_Link_Online_Stress_Testing_img_6.png | Bin 0 -> 12014 bytes
.../img/Full_Link_Online_Stress_Testing_img_7.png | Bin 0 -> 30466 bytes
.../img/Full_Link_Online_Stress_Testing_img_8.png | Bin 0 -> 20625 bytes
.../img/Full_Link_Online_Stress_Testing_img_9.png | Bin 0 -> 23255 bytes
docs/blog/static/img/Hou_Yang_Photo.png | Bin 0 -> 94106 bytes
12 files changed, 238 insertions(+)
diff --git a/docs/blog/content/material/Jan_28_Full_Link_Online_Stress_Testing_for_Production_Database_Apache_ShardingSphere_Shadow_Database_Feature_Upgrade.en.md b/docs/blog/content/material/Jan_28_Full_Link_Online_Stress_Testing_for_Production_Database_Apache_ShardingSphere_Shadow_Database_Feature_Upgrade.en.md
new file mode 100644
index 0000000..f6c5c1e
--- /dev/null
+++ b/docs/blog/content/material/Jan_28_Full_Link_Online_Stress_Testing_for_Production_Database_Apache_ShardingSphere_Shadow_Database_Feature_Upgrade.en.md
@@ -0,0 +1,238 @@
+# Full Link Online Stress Testing for Production Database: Apache ShardingSphere Shadow Database Feature Upgrade
+
+## What is full link stress testing?
+
+As the Internet industry is growing rapidly, businesses that operate with large amounts of data are seeing a rapid expansion.
+
+Predictably, ever changing customer demand is having a significant impact on the stability of their whole systems. For example, online food delivery platforms receive most of their customer orders at noon and in the evening. Online shopping sprees and time-limited sales promotions are also good examples.
+
+All businesses are served by a series of business systems which are distributedly deployed in different machines. “Data Plannig” cannot only ensure systems’ stability but also save costs, which are some of the major problems for technology teams.
+
+To precisely get the right service from a specific machine, stress testing should be conducted in the production environment. This can ensure the authenticity of the environment and the data, significantly improving the precision of “data planning”.
+
+## Shadow Database and Full-Link Stress Testing
+
+Performing a stress test on an online business system is evidently risky. For example, data corruption or performance problems may arise.
+
+Just imagine how it would compromise customer experience, if customers found their order has been lost or an unpaid order popped up?
+
+Full-link online stress testing implies a significant amount of complicated work which requires cooperation between microservices and middlewares. [Apache ShardingSphere](https://shardingsphere.apache.org/) focuses on database-level solutions in full-link stress testing.
+
+With ShardingSphere’s powerful SQL parsing capability, Apache ShardingSphere released the shadow database stress testing feature to determine shadow databases by executing SQL, and to meet the online stress testing needs in complicated business scenarios with the flexible configuration of shadow algorithms. By routing stress testing traffic to shadow databases, and normal online traffic to production database, stress testing data will be isolated and the data corruption problem is solved.
+
+## Shadow Database Function Upgrading
+
+Shadow database function was initially realized in version 4.1.0 by adding a logic shadow column. By parsing, executing, routing and rewriting SQL, Apache ShardingSphere deleted the shadow column and column value. Users do not need to setup or do anything during this process. They only need to modify SQL accordingly, add shadow fields and corresponding configurations.
+
+### Two pain points occur when adding shadow column:
+
+1. Before performing stress testings, users need to modify test-related SQL according to actual business needs.
+2. SQL modification will increase implementation damage and reduce the accuracy of stress testing results. After discussion in the ShardingSphere community, we decided to upgrade shadow database functions. Apache ShardingSphere 4.1.1 GA shadow database API had relatively simple functionaloity. Whether to open shadow database is determined by corresponding values of the logicColumn.
+
+
+
+The upgraded shadow database API of 5.0.0 GA is more powerful. Users can determine whether the shadow database function is enabled through the “enable” attribute. Configurable shadow table can determine what needs to be stress tested by means of a table, and supports multiple shadow algorithms. For example, column value matching algorithm, regular expression matching and SQL comment matching algorithm.
+
+
+
+## Shadow Database in Practice
+
+Online full-link stress testing diagram:
+
+
+
+### Prepare stress testing environment:
+Suppose an e-commerce website needs to perform online stress testing for an order (demonstrating how to use stand-alone deployment). Suppose stress testing table `t_order` is an order table, and the ID of the test user is 0.
+
+The data generated by test user order is executed on ds_shadow shadow database, and the production data is executed on ds production database.
+
+**Prepare testing environment:**
+
+1. Download ShardingSphere-Proxy 5.0.0 GA from the [Download Page](https://shardingsphere.apache.org/document/5.0.0/en/downloads/), and for installation configuration details, please refer to [ShardingSphere-Proxy-Quick-Start](https://shardingsphere.apache.org/document/current/en/quick-start/shardingsphere-proxy-quick-start/).
+2. Configure ShardingSphere-Proxy in the hypothetical stress testing scenario mentioned above:
+
+**server.yaml**
+
+
+
+**config-shadow.yaml**
+
+
+
+3. Order service
+
+Order-related businesses are not discussed here. Considering the simplest request taking and order table inserting as an example, the order table structure is as follows:
+
+- Order Table Structure
+
+
+
+### Stress Testing Process Simulation
+- Use `postman` to simulate the order created by test user, which is as follows:
+
+
+
+- SQL executor being routed to and executed in shadow database can be seen in ShardingSphere-Proxy execution log:
+
+
+
+### Verify Stress Testing Results :
+
+- Shadow database `ds_shadow` executes query sentence `SELECT * FROM t_order;`
+
+Query result:
+
+
+
+- Production database ds executes query sentence `SELECT * FROM t_order;`
+
+Query result:
+
+
+
+The data generated from test user order creation will be routed to shadow database. For more complicated configurations, please refer to Shadow Database Stress Testing in the [ShardingSphere official document](https://shardingsphere.apache.org/document/5.0.0/cn/features/shadow/).
+
+### Complete Solution for Full-Link Online Stress Testing — CyborgFlow
+
+As mentioned in the introduction, full-link online stress testing is a complicated task that requires collaboration between microservices and middlewares to meet the needs of different traffic and stress testing tag transmissions.
+
+Additonally, the testing service should be stateless and immediately available. [CyborgFlow](https://github.com/SphereEx/CyborgFlow), which is jointly maintained by Apache ShardingSphere, Apache APISIX and Apache SkyWalking provides out-of-the-box (OoTB) solution to run load test in your online system.
+
+[Apache APISIX](https://apisix.apache.org/) is responsible for making tags on testing data at the gateway layer, while [Apache SkyWalking](https://skywalking.apache.org/) is responsible for transmission through the whole scheduling link, and finally, Apache ShardingSphere-Proxy will isolate data and route testing data to the shadow database.
+
+The 0.1.0 version of CyborgFlow has been [released and is available for download](https://github.com/SphereEx/CyborgFlow/releases).
+
+## Apache ShardingSphere Open Source Project Links:
+[ShardingSphere Github](https://github.com/apache/shardingsphere)
+
+[ShardingSphere Twitter](https://twitter.com/ShardingSphere)
+
+[ShardingSphere Slack Channel](https://apacheshardingsphere.slack.com/join/shared_invite/zt-sbdde7ie-SjDqo9~I4rYcR18bq0SYTg)
+
+[Contributor Guide](https://shardingsphere.apache.org/community/cn/contribute/)
+
+## Author
+
+HouYang
+
+> SphereEx Middleware Developer, Apache ShardingSphere Contributor.
+>
+> Currently he focuses on the design and development of ShadowDB and full-link stress testing.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_1.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_1.png
new file mode 100644
index 0000000..d4b4e52
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_1.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_10.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_10.png
new file mode 100644
index 0000000..66ed351
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_10.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_2.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_2.png
new file mode 100644
index 0000000..1f385c0
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_2.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_3.jpeg b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_3.jpeg
new file mode 100644
index 0000000..18ded1f
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_3.jpeg differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_4.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_4.png
new file mode 100644
index 0000000..250d89a
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_4.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_5.jpg b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_5.jpg
new file mode 100644
index 0000000..dc60d9d
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_5.jpg differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_6.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_6.png
new file mode 100644
index 0000000..3419390
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_6.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_7.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_7.png
new file mode 100644
index 0000000..9927e38
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_7.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_8.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_8.png
new file mode 100644
index 0000000..00a915d
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_8.png differ
diff --git a/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_9.png b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_9.png
new file mode 100644
index 0000000..f6e5bb0
Binary files /dev/null and b/docs/blog/static/img/Full_Link_Online_Stress_Testing_img_9.png differ
diff --git a/docs/blog/static/img/Hou_Yang_Photo.png b/docs/blog/static/img/Hou_Yang_Photo.png
new file mode 100644
index 0000000..281729b
Binary files /dev/null and b/docs/blog/static/img/Hou_Yang_Photo.png differ