You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by pa...@apache.org on 2023/06/22 13:27:50 UTC
[shardingsphere] branch master updated: Fix : fix the incorrect link in blog (#26456) (#26500)
This is an automated email from the ASF dual-hosted git repository.
panjuan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/shardingsphere.git
The following commit(s) were added to refs/heads/master by this push:
new e6a99e9af56 Fix : fix the incorrect link in blog (#26456) (#26500)
e6a99e9af56 is described below
commit e6a99e9af56c3f27269c318c761fcb593d81ab5c
Author: 孙念君 Nianjun Sun <su...@apache.org>
AuthorDate: Thu Jun 22 21:27:43 2023 +0800
Fix : fix the incorrect link in blog (#26456) (#26500)
---
.github/workflows/nightly-check.yml | 2 +-
...nical_Deep_Dive_by_Apache_ShardingSphere.en.md} | 0
...e_ShardingSphere_Enterprise_Applications.en.md} | 0
..._Performance_Record_with_10_Million_tpmC.en.md} | 0
...y_Database Protocol_Interaction_Explained.en.md | 251 ---------------------
...pache_ShardingSphere_5.1.0_Now_Avaliable.en.md} | 0
...d_SQL_Statement_in_Apache_ShardingSphere.en.md} | 0
7 files changed, 1 insertion(+), 252 deletions(-)
diff --git a/.github/workflows/nightly-check.yml b/.github/workflows/nightly-check.yml
index ce5977b3d56..6ab20e76a4c 100644
--- a/.github/workflows/nightly-check.yml
+++ b/.github/workflows/nightly-check.yml
@@ -145,7 +145,7 @@ jobs:
- run: sudo npm install -g markdown-link-check@3.10.0
- name: Check dead link
run: |
- IGNORED_PATH_LIST=("./docs/document/themes" "./docs/community/content/powered-by")
+ IGNORED_PATH_LIST=("./docs/document/themes" "./docs/community/content/powered-by" "./docs/blog/content/material" "./docs/blog/themes")
ignore_current_file=false
for file in $(find . -name "*.md"); do
diff --git a/docs/blog/content/material/2022_03_09_SQL_Parse_Format_Function_A _Technical_Deep_Dive_by_Apache_ShardingSphere.en.md b/docs/blog/content/material/2022_03_09_SQL_Parse_Format_Function_A_Technical_Deep_Dive_by_Apache_ShardingSphere.en.md
similarity index 100%
rename from docs/blog/content/material/2022_03_09_SQL_Parse_Format_Function_A _Technical_Deep_Dive_by_Apache_ShardingSphere.en.md
rename to docs/blog/content/material/2022_03_09_SQL_Parse_Format_Function_A_Technical_Deep_Dive_by_Apache_ShardingSphere.en.md
diff --git a/docs/blog/content/material/2022_03_29_CITIC_Industrial_Cloud_Apache_ShardingSphere_ Enterprise_Applications.en.md b/docs/blog/content/material/2022_03_29_CITIC_Industrial_Cloud_Apache_ShardingSphere_Enterprise_Applications.en.md
similarity index 100%
rename from docs/blog/content/material/2022_03_29_CITIC_Industrial_Cloud_Apache_ShardingSphere_ Enterprise_Applications.en.md
rename to docs/blog/content/material/2022_03_29_CITIC_Industrial_Cloud_Apache_ShardingSphere_Enterprise_Applications.en.md
diff --git a/docs/blog/content/material/2022_04_21_Apache_ShardingSphere_openGauss_Breaking_the Distributed_Database_Performance_Record_with_10_Million_tpmC.en.md b/docs/blog/content/material/2022_04_21_Apache_ShardingSphere_openGauss_Breaking_the_Distributed_Database_Performance_Record_with_10_Million_tpmC.en.md
similarity index 100%
rename from docs/blog/content/material/2022_04_21_Apache_ShardingSphere_openGauss_Breaking_the Distributed_Database_Performance_Record_with_10_Million_tpmC.en.md
rename to docs/blog/content/material/2022_04_21_Apache_ShardingSphere_openGauss_Breaking_the_Distributed_Database_Performance_Record_with_10_Million_tpmC.en.md
diff --git a/docs/blog/content/material/2023_02_23_ShardingSphere-Proxy_Database Protocol_Interaction_Explained.en.md b/docs/blog/content/material/2023_02_23_ShardingSphere-Proxy_Database Protocol_Interaction_Explained.en.md
deleted file mode 100644
index 9278e23f3cf..00000000000
--- a/docs/blog/content/material/2023_02_23_ShardingSphere-Proxy_Database Protocol_Interaction_Explained.en.md
+++ /dev/null
@@ -1,251 +0,0 @@
-+++
-title = "ShardingSphere-Proxy Database Protocol Interaction Explained"
-weight = 89
-chapter = true
-
-+++
-
-Database protocols are considered relatively unknown knowledge for most developers. Generally, users and developers use databases through off-the-shelf database clients and drivers, without manipulating database protocols directly.
-
-However, having some basic understanding of the features and processes of database protocols can help developers provide some ideas to identify troubleshooting problems when it comes to database functionality and performance issues.
-
-In this post, we will introduce the features of commonly used open source database protocols such as MySQL and PostgreSQL, and explain how ShardingSphere-Proxy interacts with clients at the database protocol level.
-
-# Introduction to the ShardingSphere Access Side
-
-The Apache ShardingSphere ecosystem includes ShardingSphere-JDBC and ShardingSphere-Proxy, deployable independently or in a hybrid deployment.
-
-They both provide standardised incremental functionality based on databases as storage nodes for a variety of application scenarios such as Java homogeneous, heterogeneous languages, cloud native and more.
-
-
-
-ShardingSphere-JDBC is a Java-based SDK that implements the standard JDBC and is lightweight and high performance, but also has obvious limitations. Nevertheless, the limitations of ShardingSphere-JDBC in terms of access are addressed by ShardingSphere-Proxy on the access side.
-
-- ShardingSphere-Proxy supports connections via any database client, database driver, and is not limited to JVM-based languages such as Java.
-- Simplifies data management. In particular, in scenarios where data sharding or encryption is used, ShardingSphere-Proxy can be used as a unified portal to manipulate data without having to consider the actual node where the data is stored or to manually decrypt it, etc.
-- Provides unified operation and maintenance control capabilities. In cluster mode, ShardingSphere-Proxy can be used to unify the management of ShardingSphere rules and configurations.
-- Heavyweight operations can be performed. ShardingSphere-JDBC is in the same process as the application, where heavyweight computation and I/O operations may impact application performance, while ShardingSphere-Proxy starts as a separate process and supports horizontal scaling, and performing heavyweight operations without impacting application performance.
-
-# Database Protocol Features
-
-There are already a number of specific interpretations of MySQL or PostgreSQL protocols on the Internet, so we won't go into detail in this article.
-
-This section focuses on the features of each database protocol, such as its support for Pipelining and how bulk operations are represented in the protocol.
-
-## MySQL Protocol
-
-The MySQL protocol is typically a “one question, one answer" protocol, e.g. to execute SQL using Prepared Statement. At the protocol level, you need to execute `COM_STMT_PREPARE` and `COM_STMT_EXECUTE` respectively.
-
-
-
-**Image source:** [MySQL documentation](https://dev.mysql.com/doc/dev/mysql-server/latest/mysqlx_protocol_use_cases.html).
-
-MySQL has added an X Plugin since 5.7.12, which allows MySQL to add support for document type stores while maintaining the original relational store.
-
-The X Plugin uses a new communication protocol, X Protocol, which uses port 33060 by default and supports pipelining, i.e. the client can send a batch of commands to the client at a time, reducing the RTT (Round-trip time) associated with the “one question, one answer" model (Round-trip time).
-
-For example, if a SQL is executed using `Prepared Statement`, at the protocol level there are `Prepare` and `Execute` steps, but at the network transport level, these two steps can be combined and sent. This can theoretically reduce the RTT by one compared to the original protocol.
-
-
-
-**Image source:** [MySQL documentation](https://dev.mysql.com/doc/dev/mysql-server/latest/mysqlx_protocol_use_cases.html).
-
-However, the X Plugin for MySQL does not seem to be catching on at the moment, and most scenarios are still based on the original MySQL protocol for client and server communication.
-
-In the case of batch operations, the MySQL protocol command `COM_STMT_EXECUTE`, which executes the `Prepared Statement `statement, can only send one set of parameters at a time, making the "one question, one answer" approach somewhat inefficient:
-
-
-
-**Image source:** [MySQL documentation](https://dev.mysql.com/doc/dev/mysql-server/latest/mysqlx_protocol_use_cases.html).
-
-The protocol itself is not designed to support batch operations, so they can only be optimised at the client level.
-
-In the case of MySQL Connector/J, for example, when the parameter `rewriteBatchedStatements` is enabled, MySQL Connector/J will internally take the multiple sets of parameters set by the `addBatch` method, combine insert values or compose update/delete into multiple statements, and send them at the protocol level sent in one go.
-
-This is a small increase in CPU overhead in exchange for a reduction in RTT. For example, for a Prepared Statement insert statement, add multiple parameters to execute:
-
-```Mysql
-INSERT INTO tbl VALUES (?, ?, ?);
-addBatch [1, "foo", "bar"]
-addBatch [2, "baz", "fuz"]
-```
-
-MySQL Connector/J Actual Execution Statement:
-
-```Mysql
-INSERT INTO tbl VALUES (1, "foo", "bar"),(2, "baz", "fuz");
-```
-
-For bulk update / delete, MySQL Connector/J will execute multiple statements via `COM_QUERY`, e.g:
-
-```Mysql
-UPDATE tbl SET name = ? WHERE id = ?;
-addBatch ["foo", 1]
-addBatch ["bar", 2]
-```
-
-MySQL Connector/J Actual Execution Statement:
-
-```Mysql
-UPDATE tbl SET name = "foo" WHERE id = 1;UPDATE tbl SET name = "bar" WHERE id = 2;
-```
-
-## PostgreSQL Protocol
-
-Compared to the MySQL protocol, the PostgreSQL protocol definition looks simpler and the PostgreSQL protocol supports Pipelining:
-
-PostgreSQL's Extended Query breaks up SQL execution into multiple steps, with the following common operations.
-
-- **Parse:** parses the SQL into a `Prepared Statement`.
-- **Describe:** Get the metadata of a `Prepared Statement` or `Portal`.
-- **Bind:** Bind the actual parameters of the `Prepared Statement` to produce an executable `Portal`; `Bind: Bind `the actual parameters of the `Prepared Statement` to produce an executable `Portal`.
-- **Execute:** executing the `Portal`.
-- **Close:** closes the `Prepared Statement` or `Portal`.
-
-An example of the PostgreSQL JDBC protocol interaction with the database is as follows:
-
-
-
-In a bulk operation scenario, the client can send multiple sets of parameters in one continuous `Bind`, `Execute`, which is multiple packets at the protocol level, but at the TCP transport level, one batch of packets can be sent out at a time.
-
-
-
-Pipelining-enabled protocols, combined with I/O multiplexing, offer certain advantages in terms of throughput. For example, Vert.x PostgreSQL, a database driver based on multiplexed I/O, scored first place in the TechEmpower Benchmark Round 15 test in the Single Query scenario (database spotting).
-
-
-
-Image source: [Tech Empower](https://www.techempower.com/benchmarks/#section=data-r15&test=db).
-
-## openGauss Protocol
-
-openGauss adds a Batch Bind message to the PostgreSQL Protocol 3.0, which only sends one set of parameters at a time.
-
-openGauss adds a Batch Bind message to support sending multiple sets of parameters at the same time.
-
-
-
-Additionally, openGauss has been enhanced in terms of authentication security. The general flow of the protocol is identical to that of PostgreSQL.
-
-# ShardingSphere-Proxy Front-End Interaction Flow Explained
-
-## Relationship between ShardingSphere-Proxy and Database Protocol
-
-Database protocols, like HTTP and other protocols, are the standard for communication between the client and the server. Each database defines its own protocol.
-
-For example, the MySQL database defines its own set of protocols, as well as the Protobuf-based X Protocol; PostgreSQL also defines its own set of protocols and etc.
-
-The average user or developer uses an off-the-shelf client or corresponding driver and the protocol is relatively transparent to them. Therefore, ShardingSphere-Proxy implements the database protocol and serves it to the public, so that users can use ShardingSphere-Proxy as if it were a database.
-
-The specific database protocol versions currently supported by ShardingSphere-Proxy are:
-
-- MySQL Protocol 4.1 (since MySQL 4.1)
-- PostgreSQL Protocol 3.0 (since PostgreSQL 7.4)
-- openGauss Protocol 3.00 / 3.50 / 3.51
-
-## Overall Process for the Access Side
-
-ShardingSphere-Proxy and ShardingSphere-JDBC share the ShardingSphere kernel module and provide different access to users.
-
-ShardingSphere-Proxy exists as a standalone process and provides services to the outside world as a database protocol. ShardingSphere-JDBC is a set of SDKs that can be called directly by users through code.
-
-
-
-## ShardingSphere-Proxy Front-end Process
-
-The ShardingSphere-Proxy front-end uses Netty to implement the database protocol. The Netty event-driven approach to front-end connections allows the ShardingSphere-Proxy front-end to maintain a large number of client connections.
-
-The protocol unwrapping and encoding logic is mainly performed in the Netty `EventLoop` thread. As the ShardingSphere-Proxy backend still uses JDBC to interact with the database, a dedicated thread pool is used to execute the ShardingSphere kernel logic and database interaction after the protocol data has been unpacked to avoid blocking in the Netty `EventLoop` thread.
-
-
-
-`PacketCodecs` are mainly used for unpacking and encoding data. As mentioned earlier in the introduction to database protocols, the PostgreSQL protocol supports pipelining and can send a batch of packets at a time.
-
-## Example:
-
-The following diagram shows the request protocol for a PostgreSQL client to execute the `select current_schema()` statement using `Prepared Statement`, where the SQL parsing and execution steps of `Prepared Statement` are sent by the client to the server for execution in one go.
-
-
-
-What the server receives is a stream of bytes. How can this stream be split into multiple protocol packets?
-
-Taking the PostgreSQL protocol format as an example, except for the Startup Message, the format of each protocol packet is 1 byte of message type + 4 bytes of data length (including the length itself) + data, with the following structure:
-
-
-
-The MySQL protocol packet format is similar.
-
-The `PacketCodec` simply follows the protocol format definition and splits the incoming byte stream correctly.
-
-After the byte stream has been split, the remaining steps are to parse the data according to the database protocol and get the SQL and parameters to be executed.
-
-Once the SQL and parameters are available, the rest of the execution process is essentially the same as executing SQL via ShardingSphere-JDBC.
-
-After the ShardingSphere-Proxy backend executes SQL via JDBC, the result set is a Java object and the `PacketCodec` calls the specific encoding logic to convert the Java object into a byte stream according to the database protocol, assembles it into a packet and responds to the client.
-
-
-
-This is the general flow of the ShardingSphere-Proxy front-end database protocol interaction.
-
-## How to Feedback Suspected Proxy Protocol Issues to the ShardingSphere Community?
-
-Due to the differences in computing power between ShardingSphere-Proxy and databases, the Proxy does not yet have 100% support for database protocols, and there are inevitably some unsupported cases in the process.
-
-This article gives some suggestions for feedback when users encounter problems with Proxy that are suspected to be caused by imperfect implementation of the Proxy protocol.
-
-## Demo Available for Simple Problem Replication
-
-If you have a problem that can be reproduced by constructing simple code (e.g. just using Python and installing a few simple dependencies), you can provide the code and steps to reproduce the problem directly in the Github issue.
-
-**Case:** a community member has submitted a problem with aDjango.db [transaction to ShardingSphere-Proxy MySQL](https://github.com/apache/shardingsphere/issues/18461).
-
-The author provided a reproduction in the issue to help the ShardingSphere team fix the problem.
-
-## Directly Submit a Problem Fix PR
-
-For relatively simple issues, the ShardingSphere team can provide ideas for fixing them, and community members who are in a position to do so can consider submitting a PR to fix them directly.
-
-**Case:** a community member gave feedback on an issue with ShardingSphere-Proxy that was reporting an error[ when connecting to it via Python asyncpg](https://github.com/apache/shardingsphere/issues/23885).
-
-The problem is that the Python asyncpg database driver adds quotes to the encoding name when sending `client_encoding` to ShardingSphere-Proxy. The ShardingSphere-Proxy PostgreSQL does not take into account that the encoding name contains quotes (which is supported by the PostgreSQL database), resulting in an encoding recognition error.
-
-The issue's author was already in a position to reproduce the problem, and submitted a PR to fix the problem directly under the guidance of the ShardingSphere team.
-
-## Capture the Traffic Between the Client and the Proxy Using a Packet Capture Tool
-
-Some heterogeneous languages users may encounter issues with ShardingSphere-Proxy that are not related to a specific functionality and are suspected to be at the protocol level.
-
-Due to differences in the technology stack between the user and the ShardingSphere team, the ShardingSphere team may not be able to quickly reproduce the issue locally. In this case, consider feeding back to the ShardingSphere community by capturing network traffic between the client and the ShardingSphere-Proxy.
-
-There is a lot of information available on the Internet on how to use this tool, so we won't mention them here.
-
-**Case:** A .NET MySqlConnector using ShardingSphere-Proxy issue was submitted by a community member some time ago [with the following problem](https://github.com/apache/shardingsphere/issues/23857).
-
-There is an error about the .NET connection to ShardingSphere-Proxy reported in the Issue. According to the stack, the error is caused during `TryResetConnectionAsync` and the last exception is thrown under the Protocol-related code, so this could be a ShardingSphere-Proxy protocol implementation does not behave the same way as MySQL.
-
-```plaintext
-An error occurred using the connection to database .....
-
- MySqlConnector.MySqlProtocolException: Packet received out-of-order. Expected 1; got 2.
- at MySqlConnector.Protocol.Serialization.ProtocolUtility.<DoReadPayloadAsync>g__AddContinuation|5_0(ValueTask`1 readPacketTask, BufferedByteReader bufferedByteReader, IByteHandler byteHandler, F
-unc`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:
-line 476
- at MySqlConnector.Core.ServerSession.ReceiveReplyAsyncAwaited(ValueTask`1 task) in /_/src/MySqlConnector/Core/ServerSession.cs:line 943
- at MySqlConnector.Core.ServerSession.TryResetConnectionAsync(ConnectionSettings cs, MySqlConnection connection, IOBehavior ioBehavior, CancellationToken cancellationToken) in /_/src/MySqlConnect
-or/Core/ServerSession.cs:line 616
-```
-
-As this issue is costly to reproduce, it's not easy for the ShardingSphere team to reproduce the issue locally, and the community has provided protocol traffic between the client and the Proxy.
-
-
-
-Based on the protocol packet capture results, the ShardingSphere team immediately identified the issue as a problem with the implementation of the ShardingSphere-Proxy MySQL packet encoding logic.
-
-## Relevant Links:
-
-**🔗** [**ShardingSphere Official Website**](https://shardingsphere.apache.org/)
-
-**🔗** [**ShardingSphere Official Project Repo**](https://github.com/apache/shardingsphere)
-
-**🔗** [**ShardingSphere Twitter**](https://twitter.com/ShardingSphere)
-
-**🔗** [**ShardingSphere Slack**](https://join.slack.com/t/apacheshardingsphere/shared_invite/zt-sbdde7ie-SjDqo9~I4rYcR18bq0SYTg)
\ No newline at end of file
diff --git a/docs/blog/content/material/Feb_18_Apache_ShardingSphere_5.1.0_Now Avaliable.en.md b/docs/blog/content/material/Feb_18_Apache_ShardingSphere_5.1.0_Now_Avaliable.en.md
similarity index 100%
rename from docs/blog/content/material/Feb_18_Apache_ShardingSphere_5.1.0_Now Avaliable.en.md
rename to docs/blog/content/material/Feb_18_Apache_ShardingSphere_5.1.0_Now_Avaliable.en.md
diff --git a/docs/blog/content/material/Jan_28_Blog_X_How_to Develop_Your_Distributed_SQL_Statement_in_Apache_ShardingSphere.en.md b/docs/blog/content/material/Jan_28_Blog_X_How_to_Develop_Your_Distributed_SQL_Statement_in_Apache_ShardingSphere.en.md
similarity index 100%
rename from docs/blog/content/material/Jan_28_Blog_X_How_to Develop_Your_Distributed_SQL_Statement_in_Apache_ShardingSphere.en.md
rename to docs/blog/content/material/Jan_28_Blog_X_How_to_Develop_Your_Distributed_SQL_Statement_in_Apache_ShardingSphere.en.md