You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/09/26 09:57:32 UTC
[GitHub] [skywalking] wankai123 opened a new pull request, #9689: Support sharding MySQL database instances and tables.
wankai123 opened a new pull request, #9689:
URL: https://github.com/apache/skywalking/pull/9689
- Support sharding MySQL database instances and tables by [Shardingsphere-Proxy](https://shardingsphere.apache.org/document/current/en/overview/#shardingsphere-proxy).
relate to https://github.com/apache/skywalking/issues/9477
- Optimize `Duration` and `DurationUtils`, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
### Support sharding MySQL database instances and tables
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [X] Update the documentation to include this new feature.
- [X] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #9689:
URL: https://github.com/apache/skywalking/pull/9689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r983568144
##########
docs/en/setup/backend/backend-storage.md:
##########
@@ -211,6 +212,34 @@ All connection-related settings, including URL link, username, and password, are
Only part of the settings is listed here. See the [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
To understand the function of the parameter `rewriteBatchedStatements=true` in MySQL, see the [MySQL official document](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-performance-extensions.html#cj-conn-prop_rewriteBatchedStatements).
+## MySQL-Sharding
Review Comment:
Is this only applicable to MySQL? Or is it applicable to other JDBC protocol?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r983575308
##########
docs/en/setup/backend/backend-storage.md:
##########
@@ -211,6 +212,34 @@ All connection-related settings, including URL link, username, and password, are
Only part of the settings is listed here. See the [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
To understand the function of the parameter `rewriteBatchedStatements=true` in MySQL, see the [MySQL official document](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-performance-extensions.html#cj-conn-prop_rewriteBatchedStatements).
+## MySQL-Sharding
Review Comment:
We are not certain about ShardingSphere compatible capability crossing MySQL and PostgreSQL.
Kai has noticed and reported to their community, we could only work with ShardingSphere 5.1.2, but 5.2.0 doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980063735
##########
test/e2e-v2/cases/storage/mysql/sharding/docker-compose.yml:
##########
@@ -0,0 +1,114 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ sharding-proxy:
+ extends:
+ file: ../../../../script/docker-compose/base-compose.yml
+ service: sharding-proxy
+ networks:
+ - e2e
+ ports:
+ - 3307
+ depends_on:
+ mysql-1:
+ condition: service_healthy
+ mysql-2:
+ condition: service_healthy
+
+ mysql-1:
+ image: mysql/mysql-server:8.0.13
+ networks:
+ - e2e
+ ports:
+ - 3306
+ environment:
+ MYSQL_ROOT_PASSWORD: "root@1234"
+ MYSQL_DATABASE: "swtest"
+ MYSQL_ROOT_HOST: "%"
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ mysql-2:
+ image: mysql/mysql-server:8.0.13
+ networks:
+ - e2e
+ ports:
+ - 3306
+ environment:
+ MYSQL_ROOT_PASSWORD: "root@1234"
+ MYSQL_DATABASE: "swtest"
+ MYSQL_ROOT_HOST: "%"
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
Review Comment:
```suggestion
mysql-1: &mysql
image: mysql/mysql-server:8.0.13
networks:
- e2e
ports:
- 3306
environment:
MYSQL_ROOT_PASSWORD: "root@1234"
MYSQL_DATABASE: "swtest"
MYSQL_ROOT_HOST: "%"
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
interval: 5s
timeout: 60s
retries: 120
mysql-2:
<<: *mysql
```
##########
test/e2e-v2/cases/zipkin/mysql/sharding/docker-compose.yml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ sharding-proxy:
+ extends:
+ file: ../../../../script/docker-compose/base-compose.yml
+ service: sharding-proxy
+ networks:
+ - e2e
+ ports:
+ - 3307
+ depends_on:
+ mysql-1:
+ condition: service_healthy
+ mysql-2:
+ condition: service_healthy
+
+ mysql-1:
+ image: mysql/mysql-server:8.0.13
+ networks:
+ - e2e
+ ports:
+ - 3306
+ environment:
+ MYSQL_ROOT_PASSWORD: "root@1234"
+ MYSQL_DATABASE: "swtest"
+ MYSQL_ROOT_HOST: "%"
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ mysql-2:
Review Comment:
Same here
##########
test/e2e-v2/cases/log/mysql/sharding/docker-compose.yml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ sharding-proxy:
+ extends:
+ file: ../../../../script/docker-compose/base-compose.yml
+ service: sharding-proxy
+ networks:
+ - e2e
+ ports:
+ - 3307
+ depends_on:
+ mysql-1:
+ condition: service_healthy
+ mysql-2:
+ condition: service_healthy
+
+ mysql-1:
+ image: mysql/mysql-server:8.0.13
+ networks:
+ - e2e
+ ports:
+ - 3306
+ environment:
+ MYSQL_ROOT_PASSWORD: "root@1234"
+ MYSQL_DATABASE: "swtest"
+ MYSQL_ROOT_HOST: "%"
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ mysql-2:
+ image: mysql/mysql-server:8.0.13
+ networks:
+ - e2e
+ ports:
+ - 3306
+ environment:
+ MYSQL_ROOT_PASSWORD: "root@1234"
+ MYSQL_DATABASE: "swtest"
+ MYSQL_ROOT_HOST: "%"
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
Review Comment:
Try to use yaml anchor to reduce duplications
```suggestion
mysql-1: &mysql
image: mysql/mysql-server:8.0.13
networks:
- e2e
ports:
- 3306
environment:
MYSQL_ROOT_PASSWORD: "root@1234"
MYSQL_DATABASE: "swtest"
MYSQL_ROOT_HOST: "%"
healthcheck:
test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
interval: 5s
timeout: 60s
retries: 120
mysql-2:
<<: *mysql
```
##########
test/e2e-v2/cases/browser/mysql/sharding/docker-compose.yml:
##########
@@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+ sharding-proxy:
+ extends:
+ file: ../../../../script/docker-compose/base-compose.yml
+ service: sharding-proxy
+ networks:
+ - e2e
+ ports:
+ - 3307
+ depends_on:
+ mysql-1:
+ condition: service_healthy
+ mysql-2:
+ condition: service_healthy
+
+ mysql-1:
+ image: mysql/mysql-server:8.0.13
+ networks:
+ - e2e
+ ports:
+ - 3306
+ environment:
+ MYSQL_ROOT_PASSWORD: "root@1234"
+ MYSQL_DATABASE: "swtest"
+ MYSQL_ROOT_HOST: "%"
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/3306"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ mysql-2:
Review Comment:
And here
##########
test/e2e-v2/script/docker-compose/base-compose.yml:
##########
@@ -114,5 +114,21 @@ services:
timeout: 60s
retries: 120
+ sharding-proxy:
Review Comment:
This base docker is for some very basic components that all cases should use, I'd rather not to add this here, otherwise future developers tend to add all other components here and this file will lose its point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r983613765
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/DurationWithinTTL.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere;
+
+import lombok.Setter;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.config.ConfigService;
+import org.apache.skywalking.oap.server.core.query.DurationUtils;
+import org.apache.skywalking.oap.server.core.query.enumeration.Step;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.joda.time.DateTime;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+public enum DurationWithinTTL {
+ INSTANCE;
+
+ private static final DateTimeFormatter YYYYMMDD = DateTimeFormat.forPattern("yyyyMMdd");
+ private static final DateTimeFormatter YYYYMMDDHH = DateTimeFormat.forPattern("yyyyMMddHH");
+ private static final DateTimeFormatter YYYYMMDDHHMM = DateTimeFormat.forPattern("yyyyMMddHHmm");
+ private static final DateTimeFormatter YYYYMMDDHHMMSS = DateTimeFormat.forPattern("yyyyMMddHHmmss");
+
+ @Setter
+ private ConfigService configService;
Review Comment:
*[ImmutableEnumChecker](https://errorprone.info/bugpattern/ImmutableEnumChecker):* enums should be immutable: 'DurationWithinTTL' has non-final field 'configService'
---
```suggestion
private final ConfigService configService;
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=339931909&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=339931909&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339931909&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339931909&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=339931909&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/dao/ShardingTopologyQueryDAO.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.dao;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.skywalking.oap.server.core.query.input.Duration;
+import org.apache.skywalking.oap.server.core.query.type.Call;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.DurationWithinTTL;
+
+public class ShardingTopologyQueryDAO extends H2TopologyQueryDAO {
+ private JDBCHikariCPClient h2Client;
Review Comment:
*[UnusedVariable](https://errorprone.info/bugpattern/UnusedVariable):* The field 'h2Client' is never read.
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=339932091&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=339932091&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339932091&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339932091&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=339932091&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980003962
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/ShardingAlgorithm.java:
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.storage;
+
+public enum ShardingAlgorithm {
+ TIME_RELATIVE_ID_SHARDING_ALGORITHM,
+ TIME_RANGE_SHARDING_ALGORITHM,
+ TIME_BUCKET_SHARDING_ALGORITHM
+}
Review Comment:
Let's add a full description of these three algorithms, as they are very important from a design perspective.
Put some demo layouts of the target data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980189513
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -250,6 +250,14 @@ private void retrieval(final Class<?> clazz,
}
}
+ if (clazz.isAnnotationPresent(SQLDatabase.Sharding.class)) {
+ SQLDatabase.Sharding sharding = clazz.getAnnotation(SQLDatabase.Sharding.class);
Review Comment:
`SQLDatabase.Sharding.class` is present in the parent class. `FunctionCategory#doGetAnnotation` use a recursion to read **MeterFunction** annotation from parent class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980050352
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/input/Duration.java:
##########
@@ -27,45 +27,50 @@
@Getter
@Setter
+/**
+ * To optimize the query, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
+ */
public class Duration {
private String start;
private String end;
private Step step;
/**
- * See {@link DurationUtils#convertToTimeBucket(Step, String)}
+ * See {@link DurationUtils#trimToStartTimeBucket(Step, String, boolean)}
*/
- public long getStartTimeBucket() {
- return DurationUtils.INSTANCE.convertToTimeBucket(step, start);
+ public long getStartTimeBucket(boolean isRecord) {
Review Comment:
When I review and read this method, I am wondering, whether we separate this method as two, is that better for reading?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980058151
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/input/Duration.java:
##########
@@ -27,45 +27,50 @@
@Getter
@Setter
+/**
+ * To optimize the query, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
+ */
public class Duration {
private String start;
private String end;
private Step step;
/**
- * See {@link DurationUtils#convertToTimeBucket(Step, String)}
+ * See {@link DurationUtils#trimToStartTimeBucket(Step, String, boolean)}
*/
- public long getStartTimeBucket() {
- return DurationUtils.INSTANCE.convertToTimeBucket(step, start);
+ public long getStartTimeBucket(boolean isRecord) {
+ return DurationUtils.INSTANCE.trimToStartTimeBucket(step, start, isRecord);
}
/**
- * See {@link DurationUtils#convertToTimeBucket(Step, String)}
+ * See {@link DurationUtils#trimToEndTimeBucket(Step, String)}
*/
public long getEndTimeBucket() {
- return DurationUtils.INSTANCE.convertToTimeBucket(step, end);
+ return DurationUtils.INSTANCE.trimToEndTimeBucket(step, end);
}
- public long getStartTimestamp() {
- return DurationUtils.INSTANCE.startTimeToTimestamp(step, start);
+ public long getStartTimestamp(boolean isRecord) {
+ return DurationUtils.INSTANCE.startTimeToTimestamp(step, DurationUtils.INSTANCE.trimToStartTimeBucket(step, start, isRecord));
}
public long getEndTimestamp() {
- return DurationUtils.INSTANCE.endTimeToTimestamp(step, end);
+ return DurationUtils.INSTANCE.endTimeToTimestamp(step, DurationUtils.INSTANCE.trimToEndTimeBucket(step, end));
}
- public long getStartTimeBucketInSec() {
- return DurationUtils.INSTANCE.startTimeDurationToSecondTimeBucket(step, start);
+ public long getStartTimeBucketInSec(boolean isRecord) {
+ return DurationUtils.INSTANCE.startTimeDurationToSecondTimeBucket(
+ step, DurationUtils.INSTANCE.trimToStartTimeBucket(step, start, isRecord));
}
public long getEndTimeBucketInSec() {
- return DurationUtils.INSTANCE.endTimeDurationToSecondTimeBucket(step, end);
+ return DurationUtils.INSTANCE.endTimeDurationToSecondTimeBucket(
+ step, DurationUtils.INSTANCE.trimToEndTimeBucket(step, end));
}
/**
* Assemble time point based on {@link #step} and {@link #start} / {@link #end}
*/
- public List<PointOfTime> assembleDurationPoints() {
- return DurationUtils.INSTANCE.getDurationPoints(step, getStartTimeBucket(), getEndTimeBucket());
+ public List<PointOfTime> assembleDurationPoints(boolean isRecord) {
+ return DurationUtils.INSTANCE.getDurationPoints(step, getStartTimeBucket(isRecord), getEndTimeBucket());
Review Comment:
This change is going to break UI if the query happens on the edge or out of the scope of TTL range. Because the `PointOfTime` is shorter than expected.
From my understanding, there are not enough elements in the list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980036466
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/SQLDatabaseModelExtension.java:
##########
@@ -37,6 +39,12 @@ public class SQLDatabaseModelExtension {
//exclude the columns from the main table
private final List<ModelColumn> excludeColumns = new ArrayList<>(5);
+ @Setter
+ private boolean isShardingTable = false;
+ @Setter
+ @Getter
+ private Sharding sharding;
Review Comment:
Merge these two as `Optional<Sharding>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980132997
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/HistogramFunction.java:
##########
@@ -31,19 +31,25 @@
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.query.type.Bucket;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ID;
+
/**
* Histogram includes data range buckets and the amount matched/grouped in the buckets. This is for original histogram
* graph visualization
*/
@MeterFunction(functionName = "sumHistogram")
@Slf4j
@ToString
+@SQLDatabase.Sharding(shardingAlgorithm = ShardingAlgorithm.TIME_RELATIVE_ID_SHARDING_ALGORITHM, tableShardingColumn = ID, dsShardingColumn = ENTITY_ID)
Review Comment:
I think we should move this into `Meter`? As it has `getEntityId` method, all implementations should have this field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#issuecomment-1259152535
:warning: **6 God Classes** were detected by Lift in this project. [Visit the Lift web console](https://lift.sonatype.com/results/github.com/apache/skywalking/01GDYY5Q1HR8PGK8KSF0S9Z37H?tab=technical-debt&utm_source=github.com&utm_campaign=lift-comment&utm_content=apache\%20skywalking) for more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980969591
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationServerSideMetrics.java:
##########
@@ -39,6 +41,7 @@
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338301995&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338301995&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338301995&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338301995&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338301995&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980914531
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationClientSideMetrics.java:
##########
@@ -27,16 +27,22 @@
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
+
@Stream(name = ServiceInstanceRelationClientSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_INSTANCE_RELATION,
builder = ServiceInstanceRelationClientSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338284173&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338284173&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284173&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284173&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338284173&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingHistoryDeleteDAO.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2HistoryDeleteDAO;
+import org.joda.time.DateTime;
+
+@Slf4j
+public class MySQLShardingHistoryDeleteDAO extends H2HistoryDeleteDAO {
+
+ private final JDBCHikariCPClient client;
+ private final MySQLShardingStorageConfig config;
+ private final ModuleManager manager;
+ private final Set<String> dataSources;
+ private final Map<String, Long> tableLatestSuccess;
+
+ public MySQLShardingHistoryDeleteDAO(JDBCHikariCPClient client, MySQLShardingStorageConfig config,
+ ModuleManager manager) {
+ super(client);
+ this.client = client;
+ this.config = config;
+ this.manager = manager;
+ this.dataSources = config.getDataSources();
+ this.tableLatestSuccess = new HashMap<>();
+ }
+
+ @Override
+ public void deleteHistory(Model model, String timeBucketColumnName, int ttl) throws IOException {
+ if (!model.isRecord()) {
+ if (!DownSampling.Minute.equals(model.getDownsampling())) {
+ return;
+ }
+ }
+
+ long deadline = Long.parseLong(DateTime.now().plusDays(-ttl).toString("yyyyMMdd"));
+ //If it's a sharding table drop expired tables
+ if (model.getSqlDBModelExtension().isShardingTable()) {
+ boolean isRuleExecuted = false;
+ Long latestSuccessDeadline = this.tableLatestSuccess.get(model.getName());
+ if (latestSuccessDeadline != null && deadline <= latestSuccessDeadline) {
+ if (log.isDebugEnabled()) {
+ log.debug("Table = {} already deleted, skip, deadline = {}, ttl = {}", model.getName(), deadline, ttl);
+ }
+ return;
+ }
+ try {
+ //refresh sharding rules
+ isRuleExecuted = ShardingRulesOperator.INSTANCE.createOrUpdateShardingRule(client, model, this.dataSources, ttl);
+ if (isRuleExecuted) {
+ MySQLShardingTableInstaller installer = new MySQLShardingTableInstaller(client, manager, config);
+ installer.createTable(model);
+ }
+ } catch (StorageException e) {
+ throw new IOException(e.getMessage(), e);
+ }
+ List<String> realTables = new ArrayList<>();
+ try (Connection connection = client.getConnection()) {
+ ResultSet resultSet = connection.getMetaData()
+ .getTables(connection.getCatalog(), null, model.getName() + "_20%", null);
+ while (resultSet.next()) {
+ realTables.add(resultSet.getString("TABLE_NAME"));
+ }
+
+ //delete additional tables
+ for (String additionalTable : model.getSqlDBModelExtension().getAdditionalTables().keySet()) {
+ ResultSet additionalTableRS = connection.getMetaData()
+ .getTables(connection.getCatalog(), null,
+ additionalTable + "_20%", null);
+ while (additionalTableRS.next()) {
+ realTables.add(additionalTableRS.getString("TABLE_NAME"));
+ }
+ }
+ } catch (JDBCClientException | SQLException e) {
+ throw new IOException(e.getMessage(), e);
+ }
+
+ List<String> prepareDeleteTables = new ArrayList<>();
+ for (String table : realTables) {
+ long timeSeries = isolateTimeFromTableName(table);
+ if (deadline >= timeSeries) {
+ prepareDeleteTables.add(table);
+ }
+ }
+ if (log.isDebugEnabled()) {
+ log.debug("Tables to be dropped: {}", prepareDeleteTables);
Review Comment:
*[CRLF_INJECTION_LOGS](https://find-sec-bugs.github.io/bugs.htm#CRLF_INJECTION_LOGS):* This use of org/slf4j/Logger.debug(Ljava/lang/String;Ljava/lang/Object;)V might be used to include CRLF characters into log messages
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338284296&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338284296&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284296&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284296&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338284296&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingTableInstaller.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.config.ConfigService;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseExtension;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.TableMetaInfo;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLTableInstaller;
+
+@Slf4j
+public class MySQLShardingTableInstaller extends MySQLTableInstaller {
+ private final ConfigService configService;
+ private final Set<String> dataSources;
+
+ public MySQLShardingTableInstaller(Client client,
+ ModuleManager moduleManager,
+ MySQLShardingStorageConfig config) {
+ super(client, moduleManager);
+ this.configService = moduleManager.find(CoreModule.NAME).provider().getService(ConfigService.class);
+ this.dataSources = config.getDataSources();
+ }
+
+ @Override
+ protected boolean isExists(Model model) throws StorageException {
+ boolean isRuleExecuted = false;
+ boolean isTableExist = false;
+ if (configService == null) {
+ throw new UnexpectedException("ConfigService can not be null, should set ConfigService first.");
+ }
+ int ttl = model.isRecord() ? configService.getRecordDataTTL() : configService.getMetricsDataTTL();
+ TableMetaInfo.addModel(model);
+ JDBCHikariCPClient h2Client = (JDBCHikariCPClient) client;
+
+ try (Connection conn = h2Client.getConnection()) {
+ try (ResultSet rset = conn.getMetaData().getTables(conn.getCatalog(), null, model.getName(), null)) {
+ if (rset.next()) {
+ isTableExist = true;
+ }
+ }
+ } catch (SQLException | JDBCClientException e) {
+ throw new StorageException(e.getMessage(), e);
+ }
+ if (model.getSqlDBModelExtension().isShardingTable()) {
+ isRuleExecuted = ShardingRulesOperator.INSTANCE.createOrUpdateShardingRule(h2Client, model, this.dataSources, ttl);
+ }
+ return isTableExist && !isRuleExecuted;
+ }
+
+ @Override
+ protected void createTable(Model model) throws StorageException {
+ //todo: init
+ super.createTable(model);
+ }
+
+ @Override
+ protected void createTable(JDBCHikariCPClient client,
+ Connection connection,
+ String tableName,
+ List<ModelColumn> columns,
+ boolean additionalTable) throws JDBCClientException {
+ SQLBuilder tableCreateSQL = new SQLBuilder("CREATE TABLE IF NOT EXISTS " + tableName + " (");
+ tableCreateSQL.append(ID_COLUMN).append(" VARCHAR(512) ");
+ if (!additionalTable) {
+ tableCreateSQL.appendLine("PRIMARY KEY, ");
+ } else {
+ tableCreateSQL.appendLine(", ");
+ }
+ for (int i = 0; i < columns.size(); i++) {
+ ModelColumn column = columns.get(i);
+ tableCreateSQL.appendLine(
+ getColumn(column) + (i != columns.size() ? "," : ""));
+ }
+
+ int indexSeq = 0;
+
+ //Add indexes
+ List<String> columnList = columns.stream().map(column -> column.getColumnName().getStorageName()).collect(
+ Collectors.toList());
+ for (int i = 0; i < columns.size(); i++) {
+ ModelColumn column = columns.get(i);
+ for (final SQLDatabaseExtension.MultiColumnsIndex index : column.getSqlDatabaseExtension()
+ .getIndices()) {
+ final String[] multiColumns = index.getColumns();
+ //Create MultiColumnsIndex on the additional table only when it contains all need columns.
+ if (additionalTable && !columnList.containsAll(Arrays.asList(multiColumns))) {
+ continue;
+ }
+ tableCreateSQL.append("KEY K")
+ .append(String.valueOf(indexSeq++));
+ tableCreateSQL.append(" (");
+ for (int j = 0; j < multiColumns.length; j++) {
+ tableCreateSQL.append(multiColumns[j]);
+ if (j < multiColumns.length - 1) {
+ tableCreateSQL.append(",");
+ }
+ }
+ tableCreateSQL.appendLine("),");
+ }
+ if (column.shouldIndex() && column.getLength() < 513) {
+ tableCreateSQL.append("KEY K")
+ .append(String.valueOf(indexSeq++));
+ tableCreateSQL.append(" (").append(column.getColumnName().getStorageName()).append(")");
+ tableCreateSQL.appendLine(i != columns.size() - 1 ? "," : "");
+ }
+ }
+ tableCreateSQL.appendLine(")");
+
+ if (log.isDebugEnabled()) {
+ log.debug("creating table: " + tableCreateSQL.toStringInNewLine());
+ }
+
+ client.execute(connection, tableCreateSQL.toString());
Review Comment:
*[SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE](https://find-sec-bugs.github.io/bugs.htm#SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE):* org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.MySQLShardingTableInstaller.createTable(JDBCHikariCPClient, Connection, String, List, boolean) passes a nonconstant String to an execute or addBatch method on an SQL statement
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338284304&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338284304&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284304&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284304&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338284304&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/ShardingRulesOperator.java:
##########
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseModelExtension;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule;
+import org.joda.time.DateTime;
+import org.joda.time.Days;
+
+public enum ShardingRulesOperator {
+ INSTANCE;
+
+ private final Map<String, ShardingRule> modelShardingRules = new HashMap<>();
+
+ private static final String TIME_RELATIVE_ID_SHARDING_EXPRESSION =
+ "${long time_bucket = Long.parseLong(id.substring(0,id.indexOf('_')));\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ private static final String TIME_SEC_RANGE_SHARDING_EXPRESSION =
+ "\"datetime-pattern\"=\"yyyyMMddHHmmss\"," +
+ "\"datetime-interval-unit\"=\"days\"," +
+ "\"datetime-interval-amount\"=\"1\"," +
+ "\"sharding-suffix-pattern\"=\"yyyyMMdd\"," +
+ "\"datetime-lower\"=\"20220101000000\"," +
+ "\"datetime-upper\"=\"20991201000000\"";
+
+ private static final String TIME_MIN_RANGE_SHARDING_EXPRESSION =
+ "\"datetime-pattern\"=\"yyyyMMddHHmm\"," +
+ "\"datetime-interval-unit\"=\"days\"," +
+ "\"datetime-interval-amount\"=\"1\"," +
+ "\"sharding-suffix-pattern\"=\"yyyyMMdd\"," +
+ "\"datetime-lower\"=\"202201010000\"," +
+ "\"datetime-upper\"=\"209912010000\"";
+
+ private static final String TIME_BUCKET_SHARDING_EXPRESSION =
+ "${\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ public void start(JDBCHikariCPClient client) throws IOException, SQLException, StorageException {
+ initShardingRules(client);
+ }
+
+ boolean createOrUpdateShardingRule(JDBCHikariCPClient client, Model model, Set<String> dataSources, int ttl) throws StorageException {
+ boolean isExecuted;
+ ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
+ builder.table(model.getName());
+ String tableName = model.getName();
+ SQLDatabaseModelExtension.Sharding sharding = model.getSqlDBModelExtension().getSharding().orElseThrow(
+ () -> new UnexpectedException("Sharding should not be empty."));
+ isExecuted = executeShardingRule(
+ buildShardingRule(builder, tableName, dataSources, sharding.getShardingAlgorithm(),
+ sharding.getTableShardingColumn(),
+ sharding.getDsShardingColumn(),
+ ttl,
+ DateTime.now()
+ ),
+ client,
+ tableName
+ );
+ // additional tables
+ for (String additionalTable : model.getSqlDBModelExtension().getAdditionalTables().keySet()) {
+ ShardingRule.ShardingRuleBuilder additionalBuilder = ShardingRule.builder();
+ additionalBuilder.table(additionalTable);
+ isExecuted = executeShardingRule(
+ buildShardingRule(additionalBuilder, additionalTable, dataSources, sharding.getShardingAlgorithm(),
+ sharding.getTableShardingColumn(),
+ sharding.getDsShardingColumn(), ttl, DateTime.now()
+ ), client, additionalTable
+ );
+ }
+
+ return isExecuted;
+ }
+
+ private void initShardingRules(JDBCHikariCPClient client) throws IOException {
+ SQLBuilder sql = new SQLBuilder("SHOW SHARDING TABLE RULES");
+ try (Connection connection = client.getConnection()) {
+ ResultSet resultSet = client.executeQuery(connection, sql.toString());
+ while (resultSet.next()) {
+ ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
+ builder.table(resultSet.getString("TABLE"));
+ builder.actualDataNodes(resultSet.getString("ACTUAL_DATA_NODES"));
+ builder.actualDataSources(resultSet.getString("ACTUAL_DATA_SOURCES"));
+ // todo: shardingsphere-5.1.2 response DATABASE_STRATEGY and TABLE_STRATEGY type "inline" but should "standard"
+ builder.databaseStrategyType("standard");
+ builder.databaseShardingColumn(resultSet.getString("DATABASE_SHARDING_COLUMN"));
+ builder.databaseShardingAlgorithmType(resultSet.getString("DATABASE_SHARDING_ALGORITHM_TYPE"));
+ builder.databaseShardingAlgorithmProps(resultSet.getString("DATABASE_SHARDING_ALGORITHM_PROPS"));
+ // todo: shardingsphere-5.1.2 response DATABASE_STRATEGY and TABLE_STRATEGY type "inline" but should "standard"
+ builder.tableStrategyType("standard");
+ builder.tableShardingColumn(resultSet.getString("TABLE_SHARDING_COLUMN"));
+ builder.tableShardingAlgorithmType(resultSet.getString("TABLE_SHARDING_ALGORITHM_TYPE"));
+ builder.tableShardingAlgorithmProps(resultSet.getString("TABLE_SHARDING_ALGORITHM_PROPS"));
+ builder.keyGenerateColumn(resultSet.getString("KEY_GENERATE_COLUMN"));
+ builder.keyGeneratorType(resultSet.getString("KEY_GENERATOR_TYPE"));
+ builder.keyGeneratorProps(resultSet.getString("KEY_GENERATOR_PROPS"));
+
+ ShardingRule shardingRule = builder.build();
+ modelShardingRules.put(shardingRule.getTable(), shardingRule);
+ }
+ } catch (JDBCClientException | SQLException e) {
+ throw new IOException(e.getMessage(), e);
+ }
+ }
+
+ private void registerShardingRule(String tableName, ShardingRule rule) {
+ modelShardingRules.put(tableName, rule);
+ }
+
+ private ShardingRule.ShardingRuleBuilder buildShardingRule(ShardingRule.ShardingRuleBuilder builder,
+ String tableName,
+ Set<String> dataSources,
+ ShardingAlgorithm shardingAlgorithm,
+ String tableShardingColumn,
+ String dsShardingColumn,
+ int ttl,
+ DateTime currentDate) {
+ buildDataNodes(builder, tableName, dataSources, ttl, currentDate);
+ buildDatabaseStrategy(builder, dsShardingColumn, dataSources.size());
+
+ switch (shardingAlgorithm) {
+ case TIME_SEC_RANGE_SHARDING_ALGORITHM:
+ buildTimeRangeTableStrategy(builder, tableShardingColumn, TIME_SEC_RANGE_SHARDING_EXPRESSION);
+ break;
+ case TIME_MIN_RANGE_SHARDING_ALGORITHM:
+ buildTimeRangeTableStrategy(builder, tableShardingColumn, TIME_MIN_RANGE_SHARDING_EXPRESSION);
+ break;
+ case TIME_RELATIVE_ID_SHARDING_ALGORITHM:
+ buildExpressionTableStrategy(builder, tableName, tableShardingColumn,
+ TIME_RELATIVE_ID_SHARDING_EXPRESSION);
+ break;
+ case TIME_BUCKET_SHARDING_ALGORITHM:
+ buildExpressionTableStrategy(builder, tableName, tableShardingColumn, TIME_BUCKET_SHARDING_EXPRESSION);
+ break;
+ default:
+ throw new UnexpectedException("Unsupported sharding algorithm " + shardingAlgorithm);
+ }
+ return builder;
+ }
+
+ private void buildDataNodes(ShardingRule.ShardingRuleBuilder builder,
+ String tableName,
+ Set<String> dataSources,
+ int ttl,
+ DateTime currentDate) {
+ DateTime startDateTime = currentDate.plusDays(1 - ttl);
+ DateTime endDate = currentDate.plusDays(1);
+ int days = Days.daysBetween(startDateTime, endDate).getDays();
+ List<DateTime> timeRanges = new ArrayList<>();
+ StringBuilder nodesBuilder = new StringBuilder();
+ for (int i = 0; i <= days; i++) {
+ timeRanges.add(startDateTime.plusDays(i));
+ }
+ dataSources.forEach(dataSource -> {
+ timeRanges.forEach(dateTime -> {
+ nodesBuilder.append("\"")
+ .append(dataSource)
+ .append(".")
+ .append(tableName)
+ .append("_")
+ .append(dateTime.toString("yyyyMMdd"))
+ .append("\"")
+ .append(",");
+ });
+ });
+ builder.actualDataNodes(nodesBuilder.substring(0, nodesBuilder.length() - 1));
+ }
+
+ private void buildDatabaseStrategy(ShardingRule.ShardingRuleBuilder builder,
+ String shardingColumn,
+ int dsCount) {
+ StringBuilder propsBuilder = new StringBuilder();
+ propsBuilder.append("\"algorithm-expression\"=\"ds_${")
+ .append(shardingColumn)
+ .append(".hashCode()&Integer.MAX_VALUE%")
+ .append(dsCount)
+ .append("}\"");
+
+ builder.databaseStrategyType("\"standard\"")
+ .databaseShardingColumn(shardingColumn)
+ .databaseShardingAlgorithmType("\"inline\"")
+ .databaseShardingAlgorithmProps(propsBuilder.toString());
+
+ }
+
+ private void buildTimeRangeTableStrategy(ShardingRule.ShardingRuleBuilder builder,
+ String tableShardingColumn, String algorithmExpression) {
+ builder.tableStrategyType("\"standard\"")
+ .tableShardingColumn(tableShardingColumn)
+ .tableShardingAlgorithmType("\"interval\"")
+ .tableShardingAlgorithmProps(algorithmExpression);
+ }
+
+ private void buildExpressionTableStrategy(ShardingRule.ShardingRuleBuilder builder,
+ String tableName,
+ String tableShardingColumn,
+ String algorithmExpression
+ ) {
+ StringBuilder propsBuilder = new StringBuilder();
+ propsBuilder.append("\"allow-range-query-with-inline-sharding\"=\"true\",")
+ .append("\"algorithm-expression\"=\"")
+ .append(tableName)
+ .append("_")
+ .append(algorithmExpression)
+ .append("\"");
+
+ builder.tableStrategyType("\"standard\"")
+ .tableShardingColumn(tableShardingColumn)
+ .tableShardingAlgorithmType("\"inline\"")
+ .tableShardingAlgorithmProps(propsBuilder.toString());
+
+ }
+
+ private boolean executeShardingRule(ShardingRule.ShardingRuleBuilder builder,
+ JDBCHikariCPClient client,
+ String tableName) throws StorageException {
+ ShardingRule existRule = modelShardingRules.get(tableName);
+ ShardingRule shardingRule;
+ String shardingRuleSQL;
+ if (existRule == null) {
+ builder.operation("CREATE");
+ shardingRule = builder.build();
+ shardingRuleSQL = shardingRule.toShardingRuleSQL();
+ } else {
+ builder.operation("ALTER");
+ existRule.setOperation("ALTER");
+ shardingRule = builder.build();
+ shardingRuleSQL = shardingRule.toShardingRuleSQL();
+ if (existRule.toShardingRuleSQL().equals(shardingRuleSQL.replaceAll("\"", ""))) {
Review Comment:
*[UNSAFE_HASH_EQUALS](https://find-sec-bugs.github.io/bugs.htm#UNSAFE_HASH_EQUALS):* Unsafe comparison of hash that are susceptible to timing attack
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338284288&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338284288&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284288&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284288&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338284288&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/h2/dao/H2SQLExecutor.java:
##########
@@ -225,23 +226,30 @@ protected <T extends StorageData> SQLExecutor getUpdateExecutor(String modelName
storageBuilder.entity2Storage(metrics, toStorage);
Map<String, Object> objectMap = toStorage.obtain();
- SQLBuilder sqlBuilder = new SQLBuilder("UPDATE " + modelName + " SET ");
- List<ModelColumn> columns = TableMetaInfo.get(modelName).getColumns();
+ StringBuilder sqlBuilder = new StringBuilder("UPDATE " + modelName + " SET ");
+ Model model = TableMetaInfo.get(modelName);
+ List<ModelColumn> columns = model.getColumns();
List<Object> param = new ArrayList<>();
for (int i = 0; i < columns.size(); i++) {
ModelColumn column = columns.get(i);
- sqlBuilder.append(column.getColumnName().getStorageName() + "= ?");
- if (i != columns.size() - 1) {
- sqlBuilder.append(",");
+ String columnName = column.getColumnName().getName();
+ if (model.getSqlDBModelExtension().isShardingTable()) {
+ SQLDatabaseModelExtension.Sharding sharding = model.getSqlDBModelExtension().getSharding().orElseThrow(
+ () -> new UnexpectedException("Sharding should not be empty."));
+ if (columnName.equals(sharding.getDsShardingColumn()) || columnName.equals(sharding.getTableShardingColumn())) {
+ continue;
+ }
}
+ sqlBuilder.append(column.getColumnName().getStorageName()).append("= ?,");
- Object value = objectMap.get(column.getColumnName().getName());
+ Object value = objectMap.get(columnName);
Review Comment:
*NULL_DEREFERENCE:* object `objectMap` last assigned on line 227 could be null and is dereferenced at line 245.
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338284372&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338284372&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284372&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338284372&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338284372&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980922662
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/process/ProcessRelationClientSideMetrics.java:
##########
@@ -39,6 +41,7 @@
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338287591&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338287591&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338287591&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338287591&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338287591&lift_comment_rating=5) ]
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationServerSideMetrics.java:
##########
@@ -27,17 +27,23 @@
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
+
@Stream(name = ServiceRelationServerSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_RELATION,
builder = ServiceRelationServerSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338287675&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338287675&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338287675&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338287675&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338287675&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wankai123 commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wankai123 commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r983572211
##########
docs/en/setup/backend/backend-storage.md:
##########
@@ -211,6 +212,34 @@ All connection-related settings, including URL link, username, and password, are
Only part of the settings is listed here. See the [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
To understand the function of the parameter `rewriteBatchedStatements=true` in MySQL, see the [MySQL official document](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-performance-extensions.html#cj-conn-prop_rewriteBatchedStatements).
+## MySQL-Sharding
Review Comment:
This PR only applicable to MySQL, postgre needs some adaptation and verification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r983176121
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/networkalias/NetworkAddressAlias.java:
##########
@@ -43,6 +45,7 @@
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=339720716&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=339720716&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339720716&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339720716&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=339720716&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980189513
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/StorageModels.java:
##########
@@ -250,6 +250,14 @@ private void retrieval(final Class<?> clazz,
}
}
+ if (clazz.isAnnotationPresent(SQLDatabase.Sharding.class)) {
+ SQLDatabase.Sharding sharding = clazz.getAnnotation(SQLDatabase.Sharding.class);
Review Comment:
`SQLDatabase.Sharding.class` is present in the parent class. `FunctionCategory#doGetAnnotation` use a recursion to read **MeterFunction** annotation from parent class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980058151
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/input/Duration.java:
##########
@@ -27,45 +27,50 @@
@Getter
@Setter
+/**
+ * To optimize the query, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
+ */
public class Duration {
private String start;
private String end;
private Step step;
/**
- * See {@link DurationUtils#convertToTimeBucket(Step, String)}
+ * See {@link DurationUtils#trimToStartTimeBucket(Step, String, boolean)}
*/
- public long getStartTimeBucket() {
- return DurationUtils.INSTANCE.convertToTimeBucket(step, start);
+ public long getStartTimeBucket(boolean isRecord) {
+ return DurationUtils.INSTANCE.trimToStartTimeBucket(step, start, isRecord);
}
/**
- * See {@link DurationUtils#convertToTimeBucket(Step, String)}
+ * See {@link DurationUtils#trimToEndTimeBucket(Step, String)}
*/
public long getEndTimeBucket() {
- return DurationUtils.INSTANCE.convertToTimeBucket(step, end);
+ return DurationUtils.INSTANCE.trimToEndTimeBucket(step, end);
}
- public long getStartTimestamp() {
- return DurationUtils.INSTANCE.startTimeToTimestamp(step, start);
+ public long getStartTimestamp(boolean isRecord) {
+ return DurationUtils.INSTANCE.startTimeToTimestamp(step, DurationUtils.INSTANCE.trimToStartTimeBucket(step, start, isRecord));
}
public long getEndTimestamp() {
- return DurationUtils.INSTANCE.endTimeToTimestamp(step, end);
+ return DurationUtils.INSTANCE.endTimeToTimestamp(step, DurationUtils.INSTANCE.trimToEndTimeBucket(step, end));
}
- public long getStartTimeBucketInSec() {
- return DurationUtils.INSTANCE.startTimeDurationToSecondTimeBucket(step, start);
+ public long getStartTimeBucketInSec(boolean isRecord) {
+ return DurationUtils.INSTANCE.startTimeDurationToSecondTimeBucket(
+ step, DurationUtils.INSTANCE.trimToStartTimeBucket(step, start, isRecord));
}
public long getEndTimeBucketInSec() {
- return DurationUtils.INSTANCE.endTimeDurationToSecondTimeBucket(step, end);
+ return DurationUtils.INSTANCE.endTimeDurationToSecondTimeBucket(
+ step, DurationUtils.INSTANCE.trimToEndTimeBucket(step, end));
}
/**
* Assemble time point based on {@link #step} and {@link #start} / {@link #end}
*/
- public List<PointOfTime> assembleDurationPoints() {
- return DurationUtils.INSTANCE.getDurationPoints(step, getStartTimeBucket(), getEndTimeBucket());
+ public List<PointOfTime> assembleDurationPoints(boolean isRecord) {
+ return DurationUtils.INSTANCE.getDurationPoints(step, getStartTimeBucket(isRecord), getEndTimeBucket());
Review Comment:
This change is going to break UI if the query happens on the edge. Because the `PointOfTime` is shorter than expected.
From my understanding, there are not enough elements in the list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980145757
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/MaxLongMetrics.java:
##########
@@ -23,12 +23,18 @@
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Entrance;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.MetricsFunction;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.SourceFrom;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
+
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ID;
/**
*
**/
Review Comment:
Remove empty comments(legacy codes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r979858902
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java:
##########
@@ -42,6 +44,9 @@ public enum DurationUtils {
private static final DateTimeFormatter YYYYMMDDHHMM = DateTimeFormat.forPattern("yyyyMMddHHmm");
private static final DateTimeFormatter YYYYMMDDHHMMSS = DateTimeFormat.forPattern("yyyyMMddHHmmss");
+ @Setter
+ private ConfigService configService;
Review Comment:
*[ImmutableEnumChecker](https://errorprone.info/bugpattern/ImmutableEnumChecker):* enums should be immutable: 'DurationUtils' has non-final field 'configService'
---
```suggestion
private final ConfigService configService;
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688227&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688227&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688227&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688227&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688227&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/ShardingRulesOperator.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseModelExtension;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule;
+import org.joda.time.DateTime;
+import org.joda.time.Days;
+
+public enum ShardingRulesOperator {
+ INSTANCE;
+
+ private final Map<String, ShardingRule> modelShardingRules = new HashMap<>();
Review Comment:
*[ImmutableEnumChecker](https://errorprone.info/bugpattern/ImmutableEnumChecker):* enums should be immutable: 'ShardingRulesOperator' has field 'modelShardingRules' of type 'java.util.Map<java.lang.String,org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule>', 'Map' is mutable
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688245&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688245&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688245&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688245&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688245&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/ShardingRulesOperator.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseModelExtension;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule;
+import org.joda.time.DateTime;
+import org.joda.time.Days;
+
+public enum ShardingRulesOperator {
+ INSTANCE;
+
+ private final Map<String, ShardingRule> modelShardingRules = new HashMap<>();
+
+ private static final String TIME_RELATIVE_ID_SHARDING_EXPRESSION =
+ "${long time_bucket = Long.parseLong(id.substring(0,id.indexOf('_')));\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ private static final String TIME_RANGE_SHARDING_EXPRESSION =
+ "\"datetime-pattern\"=\"yyyyMMddHHmmss\"," +
+ "\"datetime-interval-unit\"=\"days\"," +
+ "\"datetime-interval-amount\"=\"1\"," +
+ "\"sharding-suffix-pattern\"=\"yyyyMMdd\"," +
+ "\"datetime-lower\"=\"20220101000000\"," +
+ "\"datetime-upper\"=\"20991201000000\"";
+
+ private static final String TIME_BUCKET_SHARDING_EXPRESSION =
+ "${\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ public void start(JDBCHikariCPClient client) throws IOException, SQLException, StorageException {
+ initShardingRules(client);
+ }
+
+ boolean createOrUpdateShardingRule(JDBCHikariCPClient client, Model model, Set<String> dataSources, int ttl) throws StorageException {
+ boolean isExecuted;
+ ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
+ builder.table(model.getName());
+ String tableName = model.getName();
+ SQLDatabaseModelExtension.Sharding sharding = model.getSqlDBModelExtension().getSharding();
+ isExecuted = executeShardingRule(
+ buildShardingRule(builder, tableName, dataSources, sharding.getShardingAlgorithm(),
+ sharding.getTableShardingColumn(),
+ sharding.getDsShardingColumn(),
+ ttl,
+ new DateTime()
Review Comment:
*[JodaConstructors](https://errorprone.info/bugpattern/JodaConstructors):* Use of certain JodaTime constructors are not allowed.
---
```suggestion
DateTime.now()
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688188&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688188&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688188&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688188&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688188&lift_comment_rating=5) ]
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java:
##########
@@ -188,4 +191,49 @@ public void verifyDateTimeString(Step step, String dateStr) {
throw new UnexpectedException("Unsupported step " + step.name());
}
+ // Trim the startStr according to the TTL, for query
+ public long trimToStartTimeBucket(Step step, String startStr, boolean isRecord) {
+ if (configService == null) {
+ throw new UnexpectedException("ConfigService can not be null, should set ConfigService first.");
+ }
+ int ttl = isRecord ? configService.getRecordDataTTL() : configService.getMetricsDataTTL();
+ long startDate = convertToTimeBucket(step, startStr);
+ long timeFloor = Long.parseLong(new DateTime().plusDays(1 - ttl).toString("yyyyMMdd"));
+ switch (step) {
+ case DAY:
+ break;
+ case HOUR:
+ timeFloor = timeFloor * 100;
+ break;
+ case MINUTE:
+ timeFloor = timeFloor * 100 * 100;
+ break;
+ case SECOND:
+ timeFloor = timeFloor * 100 * 100 * 100;
+ break;
+ }
+
+ return Math.max(timeFloor, startDate);
+ }
+
+ // Trim the endStr according to the real date, for query
+ public long trimToEndTimeBucket(Step step, String endStr) {
+ long endDate = convertToTimeBucket(step, endStr);
+ long timeCeiling = Long.parseLong(new DateTime().toString("yyyyMMdd"));
Review Comment:
*[JodaConstructors](https://errorprone.info/bugpattern/JodaConstructors):* Use of certain JodaTime constructors are not allowed.
---
```suggestion
long timeCeiling = Long.parseLong(DateTime.now().toString("yyyyMMdd"));
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688211&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688211&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688211&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688211&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688211&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/ShardingRulesOperator.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseModelExtension;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule;
+import org.joda.time.DateTime;
+import org.joda.time.Days;
+
+public enum ShardingRulesOperator {
+ INSTANCE;
+
+ private final Map<String, ShardingRule> modelShardingRules = new HashMap<>();
+
+ private static final String TIME_RELATIVE_ID_SHARDING_EXPRESSION =
+ "${long time_bucket = Long.parseLong(id.substring(0,id.indexOf('_')));\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ private static final String TIME_RANGE_SHARDING_EXPRESSION =
+ "\"datetime-pattern\"=\"yyyyMMddHHmmss\"," +
+ "\"datetime-interval-unit\"=\"days\"," +
+ "\"datetime-interval-amount\"=\"1\"," +
+ "\"sharding-suffix-pattern\"=\"yyyyMMdd\"," +
+ "\"datetime-lower\"=\"20220101000000\"," +
+ "\"datetime-upper\"=\"20991201000000\"";
+
+ private static final String TIME_BUCKET_SHARDING_EXPRESSION =
+ "${\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ public void start(JDBCHikariCPClient client) throws IOException, SQLException, StorageException {
+ initShardingRules(client);
+ }
+
+ boolean createOrUpdateShardingRule(JDBCHikariCPClient client, Model model, Set<String> dataSources, int ttl) throws StorageException {
+ boolean isExecuted;
+ ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
+ builder.table(model.getName());
+ String tableName = model.getName();
+ SQLDatabaseModelExtension.Sharding sharding = model.getSqlDBModelExtension().getSharding();
+ isExecuted = executeShardingRule(
+ buildShardingRule(builder, tableName, dataSources, sharding.getShardingAlgorithm(),
+ sharding.getTableShardingColumn(),
+ sharding.getDsShardingColumn(),
+ ttl,
+ new DateTime()
+ ),
+ client,
+ tableName
+ );
+ // additional tables
+ for (String additionalTable : model.getSqlDBModelExtension().getAdditionalTables().keySet()) {
+ ShardingRule.ShardingRuleBuilder additionalBuilder = ShardingRule.builder();
+ additionalBuilder.table(additionalTable);
+ isExecuted = executeShardingRule(
+ buildShardingRule(additionalBuilder, additionalTable, dataSources, sharding.getShardingAlgorithm(),
+ sharding.getTableShardingColumn(),
+ sharding.getDsShardingColumn(), ttl, new DateTime()
Review Comment:
*[JodaConstructors](https://errorprone.info/bugpattern/JodaConstructors):* Use of certain JodaTime constructors are not allowed.
---
```suggestion
sharding.getDsShardingColumn(), ttl, DateTime.now()
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688312&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688312&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688312&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688312&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688312&lift_comment_rating=5) ]
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java:
##########
@@ -188,4 +191,49 @@ public void verifyDateTimeString(Step step, String dateStr) {
throw new UnexpectedException("Unsupported step " + step.name());
}
+ // Trim the startStr according to the TTL, for query
+ public long trimToStartTimeBucket(Step step, String startStr, boolean isRecord) {
+ if (configService == null) {
+ throw new UnexpectedException("ConfigService can not be null, should set ConfigService first.");
+ }
+ int ttl = isRecord ? configService.getRecordDataTTL() : configService.getMetricsDataTTL();
+ long startDate = convertToTimeBucket(step, startStr);
+ long timeFloor = Long.parseLong(new DateTime().plusDays(1 - ttl).toString("yyyyMMdd"));
Review Comment:
*[JodaConstructors](https://errorprone.info/bugpattern/JodaConstructors):* Use of certain JodaTime constructors are not allowed.
---
```suggestion
long timeFloor = Long.parseLong(DateTime.now().plusDays(1 - ttl).toString("yyyyMMdd"));
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688400&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688400&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688400&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688400&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688400&lift_comment_rating=5) ]
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/endpoint/EndpointRelationServerSideMetrics.java:
##########
@@ -27,17 +27,23 @@
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
+
@Stream(name = EndpointRelationServerSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.ENDPOINT_RELATION,
builder = EndpointRelationServerSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688113&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688113&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688113&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688113&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688113&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingStorageProvider.java:
##########
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.SQLException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.config.ConfigService;
+import org.apache.skywalking.oap.server.core.storage.IBatchDAO;
+import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageDAO;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.cache.INetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.core.storage.management.UITemplateManagementDAO;
+import org.apache.skywalking.oap.server.core.storage.model.ModelCreator;
+import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IEBPFProfilingDataDAO;
+import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IEBPFProfilingScheduleDAO;
+import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IEBPFProfilingTaskDAO;
+import org.apache.skywalking.oap.server.core.storage.profiling.ebpf.IServiceLabelDAO;
+import org.apache.skywalking.oap.server.core.storage.profiling.trace.IProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profiling.trace.IProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.profiling.trace.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAggregationQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IAlarmQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IEventQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ILogQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetadataQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IMetricsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITagAutoCompleteQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITopologyQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.IZipkinQueryDAO;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EBPFProfilingDataDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EBPFProfilingScheduleDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EBPFProfilingTaskDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2EventQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2HistoryDeleteDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2MetadataQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2NetworkAddressAliasDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ServiceLabelQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ProfileTaskLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ProfileTaskQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2StorageDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TagAutoCompleteQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TopNRecordsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2TopologyQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2UITemplateManagementDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLAggregationQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLAlarmQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLTraceQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MysqlBrowserLogQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao.MySQLMetricsQueryDAO;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.dao.MySQLZipkinQueryDAO;
+
+@Slf4j
+public class MySQLShardingStorageProvider extends ModuleProvider {
+
+ private MySQLShardingStorageConfig config;
+ private JDBCHikariCPClient mysqlClient;
+
+ public MySQLShardingStorageProvider() {
+ config = new MySQLShardingStorageConfig();
+ }
+
+ @Override
+ public String name() {
+ return "mysql-sharding";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return StorageModule.class;
+ }
+
+ @Override
+ public ModuleConfig createConfigBeanIfAbsent() {
+ return config;
+ }
+
+ @Override
+ public void prepare() throws ServiceNotProvidedException {
+ this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());
+
+ mysqlClient = new JDBCHikariCPClient(config.getProperties());
+
+ this.registerServiceImplementation(IBatchDAO.class, new H2BatchDAO(mysqlClient, config.getMaxSizeOfBatchSql(), config.getAsyncBatchPersistentPoolSize()));
+ this.registerServiceImplementation(
+ StorageDAO.class,
+ new H2StorageDAO(mysqlClient));
+ this.registerServiceImplementation(
+ INetworkAddressAliasDAO.class, new H2NetworkAddressAliasDAO(mysqlClient));
+
+ this.registerServiceImplementation(ITopologyQueryDAO.class, new H2TopologyQueryDAO(mysqlClient));
+ this.registerServiceImplementation(IMetricsQueryDAO.class, new MySQLMetricsQueryDAO(mysqlClient));
+ this.registerServiceImplementation(
+ ITraceQueryDAO.class,
+ new MySQLTraceQueryDAO(getManager(), mysqlClient)
+ );
+ this.registerServiceImplementation(IBrowserLogQueryDAO.class, new MysqlBrowserLogQueryDAO(mysqlClient));
+ this.registerServiceImplementation(
+ IMetadataQueryDAO.class, new H2MetadataQueryDAO(mysqlClient, config.getMetadataQueryMaxSize()));
+ this.registerServiceImplementation(IAggregationQueryDAO.class, new MySQLAggregationQueryDAO(mysqlClient));
+ this.registerServiceImplementation(IAlarmQueryDAO.class, new MySQLAlarmQueryDAO(mysqlClient, getManager()));
+ this.registerServiceImplementation(
+ IHistoryDeleteDAO.class, new H2HistoryDeleteDAO(mysqlClient));
+ this.registerServiceImplementation(ITopNRecordsQueryDAO.class, new H2TopNRecordsQueryDAO(mysqlClient));
+ this.registerServiceImplementation(
+ ILogQueryDAO.class,
+ new MySQLLogQueryDAO(mysqlClient, getManager()));
+
+ this.registerServiceImplementation(IProfileTaskQueryDAO.class, new H2ProfileTaskQueryDAO(mysqlClient));
+ this.registerServiceImplementation(IProfileTaskLogQueryDAO.class, new H2ProfileTaskLogQueryDAO(mysqlClient));
+ this.registerServiceImplementation(
+ IProfileThreadSnapshotQueryDAO.class, new H2ProfileThreadSnapshotQueryDAO(mysqlClient));
+ this.registerServiceImplementation(UITemplateManagementDAO.class, new H2UITemplateManagementDAO(mysqlClient));
+
+ this.registerServiceImplementation(
+ IHistoryDeleteDAO.class, new MySQLShardingHistoryDeleteDAO(mysqlClient, config, getManager()));
+
+ this.registerServiceImplementation(IEventQueryDAO.class, new H2EventQueryDAO(mysqlClient));
+
+ this.registerServiceImplementation(IEBPFProfilingTaskDAO.class, new H2EBPFProfilingTaskDAO(mysqlClient));
+ this.registerServiceImplementation(IEBPFProfilingScheduleDAO.class, new H2EBPFProfilingScheduleDAO(mysqlClient));
+ this.registerServiceImplementation(IEBPFProfilingDataDAO.class, new H2EBPFProfilingDataDAO(mysqlClient));
+ this.registerServiceImplementation(IServiceLabelDAO.class, new H2ServiceLabelQueryDAO(mysqlClient));
+ this.registerServiceImplementation(ITagAutoCompleteQueryDAO.class, new H2TagAutoCompleteQueryDAO(mysqlClient));
+ this.registerServiceImplementation(IZipkinQueryDAO.class, new MySQLZipkinQueryDAO(mysqlClient));
+ }
+
+ @Override
+ public void start() throws ServiceNotProvidedException, ModuleStartException {
+ final ConfigService configService = getManager().find(CoreModule.NAME)
Review Comment:
*[UnusedVariable](https://errorprone.info/bugpattern/UnusedVariable):* The local variable 'configService' is never read.
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337688261&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337688261&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688261&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337688261&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337688261&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r981048546
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/endpoint/EndpointTraffic.java:
##########
@@ -30,16 +30,22 @@
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
import org.apache.skywalking.oap.server.core.storage.annotation.ElasticSearch;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
+
@Stream(name = EndpointTraffic.INDEX_NAME, scopeId = DefaultScopeDefine.ENDPOINT,
builder = EndpointTraffic.Builder.class, processor = MetricsStreamProcessor.class)
@MetricsExtension(supportDownSampling = false, supportUpdate = false)
@EqualsAndHashCode
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=338353543&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=338353543&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338353543&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=338353543&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=338353543&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r983728376
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/service/ServiceRelationClientSideMetrics.java:
##########
@@ -27,16 +27,22 @@
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
+
@Stream(name = ServiceRelationClientSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_RELATION,
builder = ServiceRelationClientSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=339988575&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=339988575&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339988575&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=339988575&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=339988575&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980045913
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinSpanRecord.java:
##########
@@ -71,6 +75,7 @@ public class ZipkinSpanRecord extends Record {
@Setter
@Getter
@Column(columnName = TRACE_ID)
+ @SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_QUERY_TABLE}, reserveOriginalColumns = true)
private String traceId;
Review Comment:
Is this a bug fix? If so, we need a change log about this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980228293
##########
docs/en/setup/backend/configuration-vocabulary.md:
##########
@@ -130,6 +130,14 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | numOfSearchableValuesPerTag | In a trace segment, this includes multiple spans with multiple tags. Different spans may have same tag key, e.g. multiple HTTP exit spans all have their own `http.method` tags. This configuration sets the limit on the maximum number of values for the same tag key. | SW_STORAGE_NUM_OF_SEARCHABLE_VALUES_PER_TAG | 2 |
| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution | SW_STORAGE_MAX_SIZE_OF_BATCH_SQL | 2000 |
| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size | SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE | 4 |
+| - | mysql-sharding | - | Sharding-Proxy for MySQL properties. The MySQL JDBC Driver is not in the dist. Please copy it into the oap-lib folder manually. | - | - |
+| - | - | properties | Hikari connection pool configurations. | - | Listed in the `application.yaml`. |
+| - | - | metadataQueryMaxSize | The maximum size of metadata per query. | SW_STORAGE_MYSQL_QUERY_MAX_SIZE | 5000 |
+| - | - | maxSizeOfArrayColumn | Some entities (e.g. trace segments) include the logic column with multiple values. In MySQL, we use multiple physical columns to host the values, e.g. change column_a with values [1,2,3,4,5] to `column_a_0 = 1, column_a_1 = 2, column_a_2 = 3 , column_a_3 = 4, column_a_4 = 5`. | SW_STORAGE_MAX_SIZE_OF_ARRAY_COLUMN | 20 |
+| - | - | numOfSearchableValuesPerTag | In a trace segment, this includes multiple spans with multiple tags. Different spans may have same tag key, e.g. multiple HTTP exit spans all have their own `http.method` tags. This configuration sets the limit on the maximum number of values for the same tag key. | SW_STORAGE_NUM_OF_SEARCHABLE_VALUES_PER_TAG | 2 |
Review Comment:
These seem expired configurations from MySQL and PostgreSQL configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980079532
##########
test/e2e-v2/script/prepare/setup-ss-proxy/download-mysql.sh:
##########
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
Review Comment:
Consider reusing the existing `download-mysql.sh` instead of duplicating it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] sonatype-lift[bot] commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r979843656
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/ShardingRule.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere;
+
+import lombok.Builder;
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+
+@Getter
+@Setter
+@Builder(toBuilder = true)
Review Comment:
*[CanIgnoreReturnValueSuggester](https://errorprone.info/bugpattern/CanIgnoreReturnValueSuggester):* Methods that always 'return this' should be annotated with @CanIgnoreReturnValue
---
```suggestion
@CanIgnoreReturnValue
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337681221&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337681221&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681221&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681221&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337681221&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingHistoryDeleteDAO.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2HistoryDeleteDAO;
+import org.joda.time.DateTime;
+
+@Slf4j
+public class MySQLShardingHistoryDeleteDAO extends H2HistoryDeleteDAO {
+
+ private final JDBCHikariCPClient client;
+ private final MySQLShardingStorageConfig config;
+ private final ModuleManager manager;
+ private final Set<String> dataSources;
+ private final Map<String, Long> tableLatestSuccess;
+
+ public MySQLShardingHistoryDeleteDAO(JDBCHikariCPClient client, MySQLShardingStorageConfig config,
+ ModuleManager manager) {
+ super(client);
+ this.client = client;
+ this.config = config;
+ this.manager = manager;
+ this.dataSources = config.getDataSources();
+ this.tableLatestSuccess = new HashMap<>();
+ }
+
+ @Override
+ public void deleteHistory(Model model, String timeBucketColumnName, int ttl) throws IOException {
+ if (!model.isRecord()) {
+ if (!DownSampling.Minute.equals(model.getDownsampling())) {
+ return;
+ }
+ }
+
+ long deadline = Long.parseLong(new DateTime().plusDays(-ttl).toString("yyyyMMdd"));
Review Comment:
*[JodaConstructors](https://errorprone.info/bugpattern/JodaConstructors):* Use of certain JodaTime constructors are not allowed.
---
```suggestion
long deadline = Long.parseLong(DateTime.now().plusDays(-ttl).toString("yyyyMMdd"));
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337681200&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337681200&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681200&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681200&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337681200&lift_comment_rating=5) ]
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/relation/instance/ServiceInstanceRelationServerSideMetrics.java:
##########
@@ -27,17 +27,23 @@
import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor;
import org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData;
import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.core.storage.annotation.SQLDatabase;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Entity;
import org.apache.skywalking.oap.server.core.storage.type.Convert2Storage;
import org.apache.skywalking.oap.server.core.storage.type.StorageBuilder;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.ENTITY_ID;
+import static org.apache.skywalking.oap.server.core.analysis.metrics.Metrics.TIME_BUCKET;
+
@Stream(name = ServiceInstanceRelationServerSideMetrics.INDEX_NAME, scopeId = DefaultScopeDefine.SERVICE_INSTANCE_RELATION,
builder = ServiceInstanceRelationServerSideMetrics.Builder.class, processor = MetricsStreamProcessor.class)
@EqualsAndHashCode(of = {
Review Comment:
*[MissingOverride](https://errorprone.info/bugpattern/MissingOverride):* canEqual overrides method in Metrics; expected @Override
---
```suggestion
@Override @EqualsAndHashCode(of = {
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337681195&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337681195&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681195&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681195&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337681195&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBAggregationQueryDAO.java:
##########
@@ -50,14 +50,14 @@ public BanyanDBAggregationQueryDAO(BanyanDBStorageClient client) {
@Override
public List<SelectedRecord> sortMetrics(TopNCondition condition, String valueColumnName, Duration duration, List<KeyValue> additionalConditions) throws IOException {
Review Comment:
*[MixedMutabilityReturnType](https://errorprone.info/bugpattern/MixedMutabilityReturnType):* This method returns both mutable and immutable collections or maps from different paths. This may be confusing for users of the method.
---
```suggestion
return ImmutableList.of();
```
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337681100&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337681100&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681100&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681100&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337681100&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingHistoryDeleteDAO.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.analysis.DownSampling;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2HistoryDeleteDAO;
+import org.joda.time.DateTime;
+
+@Slf4j
+public class MySQLShardingHistoryDeleteDAO extends H2HistoryDeleteDAO {
+
+ private final JDBCHikariCPClient client;
+ private final MySQLShardingStorageConfig config;
+ private final ModuleManager manager;
+ private final Set<String> dataSources;
+ private final Map<String, Long> tableLatestSuccess;
+
+ public MySQLShardingHistoryDeleteDAO(JDBCHikariCPClient client, MySQLShardingStorageConfig config,
+ ModuleManager manager) {
+ super(client);
+ this.client = client;
+ this.config = config;
+ this.manager = manager;
+ this.dataSources = config.getDataSources();
+ this.tableLatestSuccess = new HashMap<>();
+ }
+
+ @Override
+ public void deleteHistory(Model model, String timeBucketColumnName, int ttl) throws IOException {
+ if (!model.isRecord()) {
+ if (!DownSampling.Minute.equals(model.getDownsampling())) {
+ return;
+ }
+ }
+
+ long deadline = Long.parseLong(new DateTime().plusDays(-ttl).toString("yyyyMMdd"));
+ //If it's a sharding table drop expired tables
+ if (model.getSqlDBModelExtension().isShardingTable()) {
+ boolean isRuleExecuted = false;
+ Long latestSuccessDeadline = this.tableLatestSuccess.get(model.getName());
+ if (latestSuccessDeadline != null && deadline <= latestSuccessDeadline) {
+ if (log.isDebugEnabled()) {
+ log.debug("Table = {} already deleted, skip, deadline = {}, ttl = {}", model.getName(), deadline, ttl);
+ }
+ return;
+ }
+ try {
+ //refresh sharding rules
+ isRuleExecuted = ShardingRulesOperator.INSTANCE.createOrUpdateShardingRule(client, model, this.dataSources, ttl);
+ if (isRuleExecuted) {
+ MySQLShardingTableInstaller installer = new MySQLShardingTableInstaller(client, manager, config);
+ installer.createTable(model);
+ }
+ } catch (StorageException e) {
+ throw new IOException(e.getMessage(), e);
+ }
+ List<String> realTables = new ArrayList<>();
+ try (Connection connection = client.getConnection()) {
+ ResultSet resultSet = connection.getMetaData()
+ .getTables(connection.getCatalog(), null, model.getName() + "_20%", null);
+ while (resultSet.next()) {
+ realTables.add(resultSet.getString("TABLE_NAME"));
+ }
+
+ //delete additional tables
+ for (String additionalTable : model.getSqlDBModelExtension().getAdditionalTables().keySet()) {
+ ResultSet additionalTableRS = connection.getMetaData()
+ .getTables(connection.getCatalog(), null,
+ additionalTable + "_20%", null);
+ while (additionalTableRS.next()) {
+ realTables.add(additionalTableRS.getString("TABLE_NAME"));
+ }
+ }
+ } catch (JDBCClientException | SQLException e) {
+ throw new IOException(e.getMessage(), e);
+ }
+
+ List<String> prepareDeleteTables = new ArrayList<>();
+ for (String table : realTables) {
+ long timeSeries = isolateTimeFromTableName(table);
+ if (deadline >= timeSeries) {
+ prepareDeleteTables.add(table);
+ }
+ }
+ if (log.isDebugEnabled()) {
+ log.debug("Tables to be dropped: {}", prepareDeleteTables);
Review Comment:
*[CRLF_INJECTION_LOGS](https://find-sec-bugs.github.io/bugs.htm#CRLF_INJECTION_LOGS):* This use of org/slf4j/Logger.debug(Ljava/lang/String;Ljava/lang/Object;)V might be used to include CRLF characters into log messages
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337681247&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337681247&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681247&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681247&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337681247&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/MySQLShardingTableInstaller.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.config.ConfigService;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.ModelColumn;
+import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseExtension;
+import org.apache.skywalking.oap.server.library.client.Client;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.TableMetaInfo;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.mysql.MySQLTableInstaller;
+
+@Slf4j
+public class MySQLShardingTableInstaller extends MySQLTableInstaller {
+ private final ConfigService configService;
+ private final Set<String> dataSources;
+
+ public MySQLShardingTableInstaller(Client client,
+ ModuleManager moduleManager,
+ MySQLShardingStorageConfig config) {
+ super(client, moduleManager);
+ this.configService = moduleManager.find(CoreModule.NAME).provider().getService(ConfigService.class);
+ this.dataSources = config.getDataSources();
+ }
+
+ @Override
+ protected boolean isExists(Model model) throws StorageException {
+ boolean isRuleExecuted = false;
+ boolean isTableExist = false;
+ if (configService == null) {
+ throw new UnexpectedException("ConfigService can not be null, should set ConfigService first.");
+ }
+ int ttl = model.isRecord() ? configService.getRecordDataTTL() : configService.getMetricsDataTTL();
+ TableMetaInfo.addModel(model);
+ JDBCHikariCPClient h2Client = (JDBCHikariCPClient) client;
+
+ try (Connection conn = h2Client.getConnection()) {
+ try (ResultSet rset = conn.getMetaData().getTables(conn.getCatalog(), null, model.getName(), null)) {
+ if (rset.next()) {
+ isTableExist = true;
+ }
+ }
+ } catch (SQLException | JDBCClientException e) {
+ throw new StorageException(e.getMessage(), e);
+ }
+ if (model.getSqlDBModelExtension().isShardingTable()) {
+ isRuleExecuted = ShardingRulesOperator.INSTANCE.createOrUpdateShardingRule(h2Client, model, this.dataSources, ttl);
+ }
+ return isTableExist && !isRuleExecuted;
+ }
+
+ @Override
+ protected void createTable(Model model) throws StorageException {
+ //todo: init
+ super.createTable(model);
+ }
+
+ @Override
+ protected void createTable(JDBCHikariCPClient client,
+ Connection connection,
+ String tableName,
+ List<ModelColumn> columns,
+ boolean additionalTable) throws JDBCClientException {
+ SQLBuilder tableCreateSQL = new SQLBuilder("CREATE TABLE IF NOT EXISTS " + tableName + " (");
+ tableCreateSQL.append(ID_COLUMN).append(" VARCHAR(512) ");
+ if (!additionalTable) {
+ tableCreateSQL.appendLine("PRIMARY KEY, ");
+ } else {
+ tableCreateSQL.appendLine(", ");
+ }
+ for (int i = 0; i < columns.size(); i++) {
+ ModelColumn column = columns.get(i);
+ tableCreateSQL.appendLine(
+ getColumn(column) + (i != columns.size() ? "," : ""));
+ }
+
+ int indexSeq = 0;
+
+ //Add indexes
+ List<String> columnList = columns.stream().map(column -> column.getColumnName().getStorageName()).collect(
+ Collectors.toList());
+ for (int i = 0; i < columns.size(); i++) {
+ ModelColumn column = columns.get(i);
+ for (final SQLDatabaseExtension.MultiColumnsIndex index : column.getSqlDatabaseExtension()
+ .getIndices()) {
+ final String[] multiColumns = index.getColumns();
+ //Create MultiColumnsIndex on the additional table only when it contains all need columns.
+ if (additionalTable && !columnList.containsAll(Arrays.asList(multiColumns))) {
+ continue;
+ }
+ tableCreateSQL.append("KEY K")
+ .append(String.valueOf(indexSeq++));
+ tableCreateSQL.append(" (");
+ for (int j = 0; j < multiColumns.length; j++) {
+ tableCreateSQL.append(multiColumns[j]);
+ if (j < multiColumns.length - 1) {
+ tableCreateSQL.append(",");
+ }
+ }
+ tableCreateSQL.appendLine("),");
+ }
+ if (column.shouldIndex() && column.getLength() < 513) {
+ tableCreateSQL.append("KEY K")
+ .append(String.valueOf(indexSeq++));
+ tableCreateSQL.append(" (").append(column.getColumnName().getStorageName()).append(")");
+ tableCreateSQL.appendLine(i != columns.size() - 1 ? "," : "");
+ }
+ }
+ tableCreateSQL.appendLine(")");
+
+ if (log.isDebugEnabled()) {
+ log.debug("creating table: " + tableCreateSQL.toStringInNewLine());
+ }
+
+ client.execute(connection, tableCreateSQL.toString());
Review Comment:
*[SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE](https://find-sec-bugs.github.io/bugs.htm#SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE):* org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql.MySQLShardingTableInstaller.createTable(JDBCHikariCPClient, Connection, String, List, boolean) passes a nonconstant String to an execute or addBatch method on an SQL statement
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337681245&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337681245&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681245&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681245&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337681245&lift_comment_rating=5) ]
##########
oap-server/server-storage-plugin/storage-shardingsphere-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/shardingsphere/mysql/ShardingRulesOperator.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.mysql;
+
+import java.io.IOException;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.storage.ShardingAlgorithm;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.model.Model;
+import org.apache.skywalking.oap.server.core.storage.model.SQLDatabaseModelExtension;
+import org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException;
+import org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLBuilder;
+import org.apache.skywalking.oap.server.storage.plugin.jdbc.shardingsphere.ShardingRule;
+import org.joda.time.DateTime;
+import org.joda.time.Days;
+
+public enum ShardingRulesOperator {
+ INSTANCE;
+
+ private final Map<String, ShardingRule> modelShardingRules = new HashMap<>();
+
+ private static final String TIME_RELATIVE_ID_SHARDING_EXPRESSION =
+ "${long time_bucket = Long.parseLong(id.substring(0,id.indexOf('_')));\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ private static final String TIME_RANGE_SHARDING_EXPRESSION =
+ "\"datetime-pattern\"=\"yyyyMMddHHmmss\"," +
+ "\"datetime-interval-unit\"=\"days\"," +
+ "\"datetime-interval-amount\"=\"1\"," +
+ "\"sharding-suffix-pattern\"=\"yyyyMMdd\"," +
+ "\"datetime-lower\"=\"20220101000000\"," +
+ "\"datetime-upper\"=\"20991201000000\"";
+
+ private static final String TIME_BUCKET_SHARDING_EXPRESSION =
+ "${\n" +
+ "if (10000000L < time_bucket && time_bucket < 99999999L) {return time_bucket;}\n" +
+ "if (1000000000L < time_bucket && time_bucket < 9999999999L) {return time_bucket.intdiv(100);}\n" +
+ "if (100000000000L < time_bucket && time_bucket < 999999999999L) {return time_bucket.intdiv(100*100);}\n" +
+ "if (10000000000000L < time_bucket && time_bucket < 99999999999999L) {return time_bucket.intdiv(100*100*100);}\n" +
+ "}";
+
+ public void start(JDBCHikariCPClient client) throws IOException, SQLException, StorageException {
+ initShardingRules(client);
+ }
+
+ boolean createOrUpdateShardingRule(JDBCHikariCPClient client, Model model, Set<String> dataSources, int ttl) throws StorageException {
+ boolean isExecuted;
+ ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
+ builder.table(model.getName());
+ String tableName = model.getName();
+ SQLDatabaseModelExtension.Sharding sharding = model.getSqlDBModelExtension().getSharding();
+ isExecuted = executeShardingRule(
+ buildShardingRule(builder, tableName, dataSources, sharding.getShardingAlgorithm(),
+ sharding.getTableShardingColumn(),
+ sharding.getDsShardingColumn(),
+ ttl,
+ new DateTime()
+ ),
+ client,
+ tableName
+ );
+ // additional tables
+ for (String additionalTable : model.getSqlDBModelExtension().getAdditionalTables().keySet()) {
+ ShardingRule.ShardingRuleBuilder additionalBuilder = ShardingRule.builder();
+ additionalBuilder.table(additionalTable);
+ isExecuted = executeShardingRule(
+ buildShardingRule(additionalBuilder, additionalTable, dataSources, sharding.getShardingAlgorithm(),
+ sharding.getTableShardingColumn(),
+ sharding.getDsShardingColumn(), ttl, new DateTime()
+ ), client, additionalTable
+ );
+ }
+
+ return isExecuted;
+ }
+
+ private void initShardingRules(JDBCHikariCPClient client) throws IOException {
+ SQLBuilder sql = new SQLBuilder("SHOW SHARDING TABLE RULES");
+ try (Connection connection = client.getConnection()) {
+ ResultSet resultSet = client.executeQuery(connection, sql.toString());
+ while (resultSet.next()) {
+ ShardingRule.ShardingRuleBuilder builder = ShardingRule.builder();
+ builder.table(resultSet.getString("TABLE"));
+ builder.actualDataNodes(resultSet.getString("ACTUAL_DATA_NODES"));
+ builder.actualDataSources(resultSet.getString("ACTUAL_DATA_SOURCES"));
+ // todo: shardingsphere-5.1.2 response DATABASE_STRATEGY and TABLE_STRATEGY type "inline" but should "standard"
+ builder.databaseStrategyType("standard");
+ builder.databaseShardingColumn(resultSet.getString("DATABASE_SHARDING_COLUMN"));
+ builder.databaseShardingAlgorithmType(resultSet.getString("DATABASE_SHARDING_ALGORITHM_TYPE"));
+ builder.databaseShardingAlgorithmProps(resultSet.getString("DATABASE_SHARDING_ALGORITHM_PROPS"));
+ // todo: shardingsphere-5.1.2 response DATABASE_STRATEGY and TABLE_STRATEGY type "inline" but should "standard"
+ builder.tableStrategyType("standard");
+ builder.tableShardingColumn(resultSet.getString("TABLE_SHARDING_COLUMN"));
+ builder.tableShardingAlgorithmType(resultSet.getString("TABLE_SHARDING_ALGORITHM_TYPE"));
+ builder.tableShardingAlgorithmProps(resultSet.getString("TABLE_SHARDING_ALGORITHM_PROPS"));
+ builder.keyGenerateColumn(resultSet.getString("KEY_GENERATE_COLUMN"));
+ builder.keyGeneratorType(resultSet.getString("KEY_GENERATOR_TYPE"));
+ builder.keyGeneratorProps(resultSet.getString("KEY_GENERATOR_PROPS"));
+
+ ShardingRule shardingRule = builder.build();
+ modelShardingRules.put(shardingRule.getTable(), shardingRule);
+ }
+ } catch (JDBCClientException | SQLException e) {
+ throw new IOException(e.getMessage(), e);
+ }
+ }
+
+ private void registerShardingRule(String tableName, ShardingRule rule) {
+ modelShardingRules.put(tableName, rule);
+ }
+
+ private ShardingRule.ShardingRuleBuilder buildShardingRule(ShardingRule.ShardingRuleBuilder builder,
+ String tableName,
+ Set<String> dataSources,
+ ShardingAlgorithm shardingAlgorithm,
+ String tableShardingColumn,
+ String dsShardingColumn,
+ int ttl,
+ DateTime currentDate) {
+ buildDataNodes(builder, tableName, dataSources, ttl, currentDate);
+ buildDatabaseStrategy(builder, dsShardingColumn, dataSources.size());
+
+ switch (shardingAlgorithm) {
+ case TIME_RANGE_SHARDING_ALGORITHM:
+ buildTimeRangeTableStrategy(builder, tableShardingColumn);
+ break;
+ case TIME_RELATIVE_ID_SHARDING_ALGORITHM:
+ buildExpressionTableStrategy(builder, tableName, tableShardingColumn,
+ TIME_RELATIVE_ID_SHARDING_EXPRESSION);
+ break;
+ case TIME_BUCKET_SHARDING_ALGORITHM:
+ buildExpressionTableStrategy(builder, tableName, tableShardingColumn, TIME_BUCKET_SHARDING_EXPRESSION);
+ break;
+ default:
+ throw new UnexpectedException("Unsupported sharding algorithm " + shardingAlgorithm);
+ }
+ return builder;
+ }
+
+ private void buildDataNodes(ShardingRule.ShardingRuleBuilder builder,
+ String tableName,
+ Set<String> dataSources,
+ int ttl,
+ DateTime currentDate) {
+ DateTime startDateTime = currentDate.plusDays(1 - ttl);
+ DateTime endDate = currentDate.plusDays(1);
+ int days = Days.daysBetween(startDateTime, endDate).getDays();
+ List<DateTime> timeRanges = new ArrayList<>();
+ StringBuilder nodesBuilder = new StringBuilder();
+ for (int i = 0; i <= days; i++) {
+ timeRanges.add(startDateTime.plusDays(i));
+ }
+ dataSources.forEach(dataSource -> {
+ timeRanges.forEach(dateTime -> {
+ nodesBuilder.append("\"")
+ .append(dataSource)
+ .append(".")
+ .append(tableName)
+ .append("_")
+ .append(dateTime.toString("yyyyMMdd"))
+ .append("\"")
+ .append(",");
+ });
+ });
+ builder.actualDataNodes(nodesBuilder.substring(0, nodesBuilder.length() - 1));
+ }
+
+ private void buildDatabaseStrategy(ShardingRule.ShardingRuleBuilder builder,
+ String shardingColumn,
+ int dsCount) {
+ StringBuilder propsBuilder = new StringBuilder();
+ propsBuilder.append("\"algorithm-expression\"=\"ds_${")
+ .append(shardingColumn)
+ .append(".hashCode()&Integer.MAX_VALUE%")
+ .append(dsCount)
+ .append("}\"");
+
+ builder.databaseStrategyType("\"standard\"")
+ .databaseShardingColumn(shardingColumn)
+ .databaseShardingAlgorithmType("\"inline\"")
+ .databaseShardingAlgorithmProps(propsBuilder.toString());
+
+ }
+
+ private void buildTimeRangeTableStrategy(ShardingRule.ShardingRuleBuilder builder,
+ String tableShardingColumn) {
+ builder.tableStrategyType("\"standard\"")
+ .tableShardingColumn(tableShardingColumn)
+ .tableShardingAlgorithmType("\"interval\"")
+ .tableShardingAlgorithmProps(TIME_RANGE_SHARDING_EXPRESSION);
+ }
+
+ private void buildExpressionTableStrategy(ShardingRule.ShardingRuleBuilder builder,
+ String tableName,
+ String tableShardingColumn,
+ String algorithmExpression
+ ) {
+ StringBuilder propsBuilder = new StringBuilder();
+ propsBuilder.append("\"allow-range-query-with-inline-sharding\"=\"true\",")
+ .append("\"algorithm-expression\"=\"")
+ .append(tableName)
+ .append("_")
+ .append(algorithmExpression)
+ .append("\"");
+
+ builder.tableStrategyType("\"standard\"")
+ .tableShardingColumn(tableShardingColumn)
+ .tableShardingAlgorithmType("\"inline\"")
+ .tableShardingAlgorithmProps(propsBuilder.toString());
+
+ }
+
+ private boolean executeShardingRule(ShardingRule.ShardingRuleBuilder builder,
+ JDBCHikariCPClient client,
+ String tableName) throws StorageException {
+ ShardingRule existRule = modelShardingRules.get(tableName);
+ ShardingRule shardingRule;
+ String shardingRuleSQL;
+ if (existRule == null) {
+ builder.operation("CREATE");
+ shardingRule = builder.build();
+ shardingRuleSQL = shardingRule.toShardingRuleSQL();
+ } else {
+ builder.operation("ALTER");
+ existRule.setOperation("ALTER");
+ shardingRule = builder.build();
+ shardingRuleSQL = shardingRule.toShardingRuleSQL();
+ if (existRule.toShardingRuleSQL().equals(shardingRuleSQL.replaceAll("\"", ""))) {
Review Comment:
*[UNSAFE_HASH_EQUALS](https://find-sec-bugs.github.io/bugs.htm#UNSAFE_HASH_EQUALS):* Unsafe comparison of hash that are susceptible to timing attack
---
<details><summary><b>âšī¸ Learn about @sonatype-lift commands</b></summary>
You can reply with the following commands. For example, reply with ***@sonatype-lift ignoreall*** to leave out all findings.
| **Command** | **Usage** |
| ------------- | ------------- |
| `@sonatype-lift ignore` | Leave out the above finding from this PR |
| `@sonatype-lift ignoreall` | Leave out all the existing findings from this PR |
| `@sonatype-lift exclude <file\|issue\|path\|tool>` | Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file |
**Note:** When talking to LiftBot, you need to **refresh** the page to see its response.
<sub>[Click here](https://github.com/apps/sonatype-lift/installations/new) to add LiftBot to another repo.</sub></details>
---
Was this a good recommendation?
[ [đ Not relevant](https://www.sonatype.com/lift-comment-rating?comment=337681253&lift_comment_rating=1) ] - [ [đ Won't fix](https://www.sonatype.com/lift-comment-rating?comment=337681253&lift_comment_rating=2) ] - [ [đ Not critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681253&lift_comment_rating=3) ] - [ [đ Critical, will fix](https://www.sonatype.com/lift-comment-rating?comment=337681253&lift_comment_rating=4) ] - [ [đ Critical, fixing now](https://www.sonatype.com/lift-comment-rating?comment=337681253&lift_comment_rating=5) ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980129406
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/input/Duration.java:
##########
@@ -27,45 +27,50 @@
@Getter
@Setter
+/**
+ * To optimize the query, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
+ */
public class Duration {
private String start;
private String end;
private Step step;
/**
- * See {@link DurationUtils#convertToTimeBucket(Step, String)}
+ * See {@link DurationUtils#trimToStartTimeBucket(Step, String, boolean)}
*/
- public long getStartTimeBucket() {
- return DurationUtils.INSTANCE.convertToTimeBucket(step, start);
+ public long getStartTimeBucket(boolean isRecord) {
Review Comment:
Or, we could have `Duration.forRecord.getStartTimeBucket()`. `forRecord` returns a `DurationCoverter`(new interface) implementation, which carries all methods. We could have two implementations for Record and Metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wankai123 commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wankai123 commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980049311
##########
docs/en/setup/backend/backend-storage.md:
##########
@@ -211,6 +211,34 @@ All connection-related settings, including URL link, username, and password, are
Only part of the settings is listed here. See the [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
To understand the function of the parameter `rewriteBatchedStatements=true` in MySQL, see the [MySQL official document](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-performance-extensions.html#cj-conn-prop_rewriteBatchedStatements).
+## MySQL-Sharding
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wankai123 commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wankai123 commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980048765
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/zipkin/ZipkinSpanRecord.java:
##########
@@ -71,6 +75,7 @@ public class ZipkinSpanRecord extends Record {
@Setter
@Getter
@Column(columnName = TRACE_ID)
+ @SQLDatabase.AdditionalEntity(additionalTables = {ADDITIONAL_QUERY_TABLE}, reserveOriginalColumns = true)
private String traceId;
Review Comment:
No, just for additional table data source sharding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r980090617
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/input/Duration.java:
##########
@@ -27,45 +27,50 @@
@Getter
@Setter
+/**
+ * To optimize the query, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
+ */
public class Duration {
private String start;
private String end;
private Step step;
/**
- * See {@link DurationUtils#convertToTimeBucket(Step, String)}
+ * See {@link DurationUtils#trimToStartTimeBucket(Step, String, boolean)}
*/
- public long getStartTimeBucket() {
- return DurationUtils.INSTANCE.convertToTimeBucket(step, start);
+ public long getStartTimeBucket(boolean isRecord) {
Review Comment:
> When I review and read this method, I am wondering, whether we separate this method as two, is that better for reading?
Aggree, when adding a `boolean` argument to a method, it's usually better in terms of readability to split the method into two, like `getRecordStartTimeBucket` and `getXXXStartTimeBucket`
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/annotation/SQLDatabase.java:
##########
@@ -135,4 +136,12 @@
@interface MultipleExtraColumn4AdditionalEntity {
ExtraColumn4AdditionalEntity[] value();
}
+
+ @Target({ElementType.TYPE})
+ @Retention(RetentionPolicy.RUNTIME)
+ @interface Sharding {
+ ShardingAlgorithm shardingAlgorithm();
+ String dsShardingColumn();
Review Comment:
In such case I'd rather write the full name, `dataSourceShardingColumn`, with IDE autocompletion it's the same key strokes but it is more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r979999637
##########
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/input/Duration.java:
##########
@@ -27,45 +27,50 @@
@Getter
@Setter
+/**
+ * To optimize the query, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
+ */
Review Comment:
```suggestion
/**
* @since 9.3.0 To optimize the query, the range of start and end times will be trimmed to [TTL_deadLine < time <= CurrentTime].
*/
@Getter
@Setter
```
This seems a better comment style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #9689: Support sharding MySQL database instances and tables.
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on code in PR #9689:
URL: https://github.com/apache/skywalking/pull/9689#discussion_r979815469
##########
docs/en/setup/backend/backend-storage.md:
##########
@@ -211,6 +211,34 @@ All connection-related settings, including URL link, username, and password, are
Only part of the settings is listed here. See the [HikariCP](https://github.com/brettwooldridge/HikariCP) connection pool document for full settings.
To understand the function of the parameter `rewriteBatchedStatements=true` in MySQL, see the [MySQL official document](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-connp-props-performance-extensions.html#cj-conn-prop_rewriteBatchedStatements).
+## MySQL-Sharding
Review Comment:
As a level two title, please update the `Natively supported storage:` part on the top
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org