You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/10 15:26:40 UTC
[GitHub] [dolphinscheduler] ruanwenjun opened a new pull request, #10406: Add mysql registry plugin
ruanwenjun opened a new pull request, #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406
## Purpose of the pull request
Add new registry plugin - mysql. If user don't have a zookeeper cluster, they can still use DolphinScheduler.
## Brief change log
- Add a new module `dolphinscheduler-registry-mysql`.
- Add dependency `dolphinscheduler-registry-mysql` to api/master/worker
## Verify this pull request
Change the registry.type to mysql and start a ds cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1152875704
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [5.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1152841730
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [15 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [5.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1153396652
Should we add this feature to the docs too? @ruanwenjun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894984687
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
+ PRIMARY KEY (`id`),
+ unique (`key`)
+) ENGINE = InnoDB
+ DEFAULT CHARSET = utf8;
+
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_lock`;
+CREATE TABLE `t_ds_mysql_registry_lock`
Review Comment:
The host can represent the lock owner.
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/pom.xml:
##########
@@ -0,0 +1,56 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to Apache Software Foundation (ASF) under one or more contributor
+ ~ license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright
+ ~ ownership. Apache Software Foundation (ASF) licenses this file to you under
+ ~ the Apache License, Version 2.0 (the "License"); you may
+ ~ not use this file except in compliance with the License.
+ ~ You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-plugins</artifactId>
+ <version>dev-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>dolphinscheduler-registry-mysql</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-common</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-dao</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1153407769
> Should we add this feature to the docs too? @ruanwenjun
Yes, right now, I have added a README.md under the `dolphinscheduler-registry-mysql`,
We may need to add a new label about registry under the docs. This will need to do some changes to about the website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1152872552
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [4.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894985607
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
+ PRIMARY KEY (`id`),
+ unique (`key`)
+) ENGINE = InnoDB
+ DEFAULT CHARSET = utf8;
+
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_lock`;
+CREATE TABLE `t_ds_mysql_registry_lock`
Review Comment:
> The host can represent the lock owner.
This is dangerous when users deploy multiple components in a single machine/host, like pseudo cluster mode, you at least need to add process id to the owner key?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894987247
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final HikariDataSource dataSource;
+
+ public MysqlOperator(MysqlRegistryProperties.MysqlDatasourceProperties datasourceProperties) {
+ HikariConfig hikariConfig = new HikariConfig();
+ hikariConfig.setDriverClassName(datasourceProperties.getDriverClassName());
+ hikariConfig.setJdbcUrl(datasourceProperties.getUrl());
+ hikariConfig.setUsername(datasourceProperties.getUsername());
+ hikariConfig.setPassword(datasourceProperties.getPassword());
+ hikariConfig.setMaximumPoolSize(datasourceProperties.getMaximumPoolSize());
+ hikariConfig.setConnectionTimeout(datasourceProperties.getConnectionTimeout());
+ hikariConfig.setIdleTimeout(datasourceProperties.getIdleTimeout());
+ hikariConfig.setPoolName("MysqlRegistryDataSourcePool");
+
+ this.dataSource = new HikariDataSource(hikariConfig);
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ // if no exception, the healthCheck success
+ preparedStatement.executeQuery();
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, create_time, last_update_time from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ ResultSet resultSet = preparedStatement.executeQuery();
Review Comment:
Also put `resultSet` in the try-with-resource block, (tho the result set will be closed when the statement/connection is closed)?
```suggestion
try (Connection connection = dataSource.getConnection();
PreparedStatement preparedStatement = connection.prepareStatement(sql);
ResultSet resultSet = preparedStatement.executeQuery()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1152828367
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [15 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [5.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894984719
##########
dolphinscheduler-api/pom.xml:
##########
@@ -64,6 +64,10 @@
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-zookeeper</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-mysql</artifactId>
+ </dependency>
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894971818
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/pom.xml:
##########
@@ -0,0 +1,56 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to Apache Software Foundation (ASF) under one or more contributor
+ ~ license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright
+ ~ ownership. Apache Software Foundation (ASF) licenses this file to you under
+ ~ the Apache License, Version 2.0 (the "License"); you may
+ ~ not use this file except in compliance with the License.
+ ~ You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-plugins</artifactId>
+ <version>dev-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>dolphinscheduler-registry-mysql</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-common</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-dao</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
Review Comment:
So glad to see ```lombok``` module is introduced. ^_^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894992149
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final HikariDataSource dataSource;
+ private final long expireTimeWindow;
+
+ public MysqlOperator(MysqlRegistryProperties registryProperties) {
+ this.expireTimeWindow = registryProperties.getTermExpireTimes() * registryProperties.getTermRefreshInterval().toMillis();
+
+ HikariConfig hikariConfig = registryProperties.getHikariConfig();
+ hikariConfig.setPoolName("MysqlRegistryDataSourcePool");
+
+ this.dataSource = new HikariDataSource(hikariConfig);
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql);
+ ResultSet resultSet = preparedStatement.executeQuery();) {
+ // if no exception, the healthCheck success
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, create_time, last_update_time from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql);
+ ResultSet resultSet = preparedStatement.executeQuery()) {
+ List<MysqlRegistryData> result = new ArrayList<>(resultSet.getFetchSize());
+ while (resultSet.next()) {
+ MysqlRegistryData mysqlRegistryData = MysqlRegistryData.builder()
+ .id(resultSet.getLong("id"))
+ .key(resultSet.getString("key"))
+ .data(resultSet.getString("data"))
+ .type(resultSet.getInt("type"))
+ .createTime(resultSet.getTimestamp("create_time"))
+ .lastUpdateTime(resultSet.getTimestamp("last_update_time"))
+ .build();
+ result.add(mysqlRegistryData);
+ }
+ return result;
+ }
+ }
+
+ public long insertOrUpdateEphemeralData(String key, String value) throws SQLException {
+ String sql = "INSERT INTO t_ds_mysql_registry_data (`key`, data, type, create_time, last_update_time) VALUES (?, ?, ?, current_timestamp, current_timestamp)" +
+ "ON DUPLICATE KEY UPDATE data=?, last_update_time=current_timestamp";
+ // put a ephemeralData
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setString(2, value);
+ preparedStatement.setInt(3, DataType.EPHEMERAL.getTypeValue());
+ preparedStatement.setString(4, value);
+ int insertCount = preparedStatement.executeUpdate();
+ ResultSet generatedKeys = preparedStatement.getGeneratedKeys();
+ if (insertCount < 1 || !generatedKeys.next()) {
+ throw new SQLException("Insert or update ephemeral data error");
+ }
+ return generatedKeys.getLong(1);
+ }
+ }
+
+ public long insertOrUpdatePersistentData(String key, String value) throws SQLException {
+ String sql = "INSERT INTO t_ds_mysql_registry_data (`key`, data, type, create_time, last_update_time) VALUES (?, ?, ?, current_timestamp, current_timestamp)" +
+ "ON DUPLICATE KEY UPDATE data=?, last_update_time=current_timestamp";
+ // put a persistent Data
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setString(2, value);
+ preparedStatement.setInt(3, DataType.PERSISTENT.getTypeValue());
+ preparedStatement.setString(4, value);
+ int insertCount = preparedStatement.executeUpdate();
+ ResultSet generatedKeys = preparedStatement.getGeneratedKeys();
+ if (insertCount < 1 || !generatedKeys.next()) {
+ throw new SQLException("Insert or update persistent data error");
+ }
+ return generatedKeys.getLong(1);
+ }
+ }
+
+ public void deleteEphemeralData(String key) throws SQLException {
+ String sql = "DELETE from t_ds_mysql_registry_data where `key` = ? and type = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setInt(2, DataType.EPHEMERAL.getTypeValue());
+ preparedStatement.execute();
+ }
+ }
+
+ public void deleteEphemeralData(long ephemeralNodeId) throws SQLException {
+ String sql = "DELETE from t_ds_mysql_registry_data where `id` = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setLong(1, ephemeralNodeId);
+ preparedStatement.execute();
+ }
+ }
+
+ public void deletePersistentData(String key) throws SQLException {
+ String sql = "DELETE from t_ds_mysql_registry_data where `key` = ? and type = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setInt(2, DataType.PERSISTENT.getTypeValue());
+ preparedStatement.execute();
+ }
+ }
+
+ public void clearExpireLock() {
+ String sql = "delete from t_ds_mysql_registry_lock where last_term < ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setTimestamp(1,
+ new Timestamp(System.currentTimeMillis() - expireTimeWindow));
+ int i = preparedStatement.executeUpdate();
+ if (i > 0) {
+ logger.info("Clear expire lock, size: {}", i);
+ }
+ } catch (Exception ex) {
+ logger.warn("Clear expire lock from mysql registry error", ex);
+ }
+ }
+
+ public void clearExpireEphemeralDate() {
Review Comment:
```suggestion
public void clearExpireEphemeralData() {
```
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final HikariDataSource dataSource;
+ private final long expireTimeWindow;
+
+ public MysqlOperator(MysqlRegistryProperties registryProperties) {
+ this.expireTimeWindow = registryProperties.getTermExpireTimes() * registryProperties.getTermRefreshInterval().toMillis();
+
+ HikariConfig hikariConfig = registryProperties.getHikariConfig();
+ hikariConfig.setPoolName("MysqlRegistryDataSourcePool");
+
+ this.dataSource = new HikariDataSource(hikariConfig);
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql);
+ ResultSet resultSet = preparedStatement.executeQuery();) {
+ // if no exception, the healthCheck success
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, create_time, last_update_time from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql);
+ ResultSet resultSet = preparedStatement.executeQuery()) {
+ List<MysqlRegistryData> result = new ArrayList<>(resultSet.getFetchSize());
+ while (resultSet.next()) {
+ MysqlRegistryData mysqlRegistryData = MysqlRegistryData.builder()
+ .id(resultSet.getLong("id"))
+ .key(resultSet.getString("key"))
+ .data(resultSet.getString("data"))
+ .type(resultSet.getInt("type"))
+ .createTime(resultSet.getTimestamp("create_time"))
+ .lastUpdateTime(resultSet.getTimestamp("last_update_time"))
+ .build();
+ result.add(mysqlRegistryData);
+ }
+ return result;
+ }
+ }
+
+ public long insertOrUpdateEphemeralData(String key, String value) throws SQLException {
+ String sql = "INSERT INTO t_ds_mysql_registry_data (`key`, data, type, create_time, last_update_time) VALUES (?, ?, ?, current_timestamp, current_timestamp)" +
+ "ON DUPLICATE KEY UPDATE data=?, last_update_time=current_timestamp";
+ // put a ephemeralData
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setString(2, value);
+ preparedStatement.setInt(3, DataType.EPHEMERAL.getTypeValue());
+ preparedStatement.setString(4, value);
+ int insertCount = preparedStatement.executeUpdate();
+ ResultSet generatedKeys = preparedStatement.getGeneratedKeys();
+ if (insertCount < 1 || !generatedKeys.next()) {
+ throw new SQLException("Insert or update ephemeral data error");
+ }
+ return generatedKeys.getLong(1);
+ }
+ }
+
+ public long insertOrUpdatePersistentData(String key, String value) throws SQLException {
+ String sql = "INSERT INTO t_ds_mysql_registry_data (`key`, data, type, create_time, last_update_time) VALUES (?, ?, ?, current_timestamp, current_timestamp)" +
+ "ON DUPLICATE KEY UPDATE data=?, last_update_time=current_timestamp";
+ // put a persistent Data
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setString(2, value);
+ preparedStatement.setInt(3, DataType.PERSISTENT.getTypeValue());
+ preparedStatement.setString(4, value);
+ int insertCount = preparedStatement.executeUpdate();
+ ResultSet generatedKeys = preparedStatement.getGeneratedKeys();
+ if (insertCount < 1 || !generatedKeys.next()) {
+ throw new SQLException("Insert or update persistent data error");
+ }
+ return generatedKeys.getLong(1);
+ }
+ }
+
+ public void deleteEphemeralData(String key) throws SQLException {
+ String sql = "DELETE from t_ds_mysql_registry_data where `key` = ? and type = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setInt(2, DataType.EPHEMERAL.getTypeValue());
+ preparedStatement.execute();
+ }
+ }
+
+ public void deleteEphemeralData(long ephemeralNodeId) throws SQLException {
+ String sql = "DELETE from t_ds_mysql_registry_data where `id` = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setLong(1, ephemeralNodeId);
+ preparedStatement.execute();
+ }
+ }
+
+ public void deletePersistentData(String key) throws SQLException {
+ String sql = "DELETE from t_ds_mysql_registry_data where `key` = ? and type = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setString(1, key);
+ preparedStatement.setInt(2, DataType.PERSISTENT.getTypeValue());
+ preparedStatement.execute();
+ }
+ }
+
+ public void clearExpireLock() {
+ String sql = "delete from t_ds_mysql_registry_lock where last_term < ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setTimestamp(1,
+ new Timestamp(System.currentTimeMillis() - expireTimeWindow));
+ int i = preparedStatement.executeUpdate();
+ if (i > 0) {
+ logger.info("Clear expire lock, size: {}", i);
+ }
+ } catch (Exception ex) {
+ logger.warn("Clear expire lock from mysql registry error", ex);
+ }
+ }
+
+ public void clearExpireEphemeralDate() {
+ String sql = "delete from t_ds_mysql_registry_data where last_update_time < ? and type = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setTimestamp(1, new Timestamp(System.currentTimeMillis() - expireTimeWindow));
+ preparedStatement.setInt(2, DataType.EPHEMERAL.getTypeValue());
+ int i = preparedStatement.executeUpdate();
+ if (i > 0) {
+ logger.info("clear expire ephemeral data, size:{}", i);
+ }
+ } catch (Exception ex) {
+ logger.warn("Clear expire ephemeral data from mysql registry error", ex);
+ }
+ }
+
+ public MysqlRegistryData getData(String key) throws SQLException {
+ String sql = "SELECT id, `key`, data, type, create_time, last_update_time FROM t_ds_mysql_registry_data WHERE `key` = " + key;
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql);
+ ResultSet resultSet = preparedStatement.executeQuery()) {
+ if (resultSet == null) {
+ return null;
+ }
+ return MysqlRegistryData.builder()
+ .id(resultSet.getLong("id"))
+ .key(resultSet.getString("key"))
+ .data(resultSet.getString("data"))
+ .type(resultSet.getInt("type"))
+ .createTime(resultSet.getTimestamp("create_time"))
+ .lastUpdateTime(resultSet.getTimestamp("last_update_time"))
+ .build();
+ }
+ }
+
+ public List<String> getChildren(String key) throws SQLException {
+ String sql = "SELECT `key` from t_ds_mysql_registry_data where `key` like ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setString(1, key + "%");
+ try (ResultSet resultSet = preparedStatement.executeQuery()) {
+ List<String> result = new ArrayList<>(resultSet.getFetchSize());
+ while (resultSet.next()) {
+ String fullPath = resultSet.getString("key");
+ if (fullPath.length() > key.length()) {
+ result.add(StringUtils.substringBefore(fullPath.substring(key.length() + 1), "/"));
+ }
+ }
+ return result;
+ }
+ }
+ }
+
+ public boolean existKey(String key) throws SQLException {
+ String sql = "SELECT 1 FROM t_ds_mysql_registry_data WHERE `key` = ?";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ preparedStatement.setString(1, key);
+ try (ResultSet resultSet = preparedStatement.executeQuery()) {
+ if (resultSet.next()) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Try to acquire the target Lock, if cannot acquire, return null.
+ */
+ public MysqlRegistryLock tryToAcquireLock(String key) throws SQLException {
+ String sql = "INSERT INTO t_ds_mysql_registry_lock (`key`, lock_owner, last_term, last_update_time, create_time) VALUES (?, ?, current_timestamp, current_timestamp, current_timestamp)";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql, Statement.RETURN_GENERATED_KEYS)) {
+ preparedStatement.setString(1, key);
+ // todo: if we start multiple master in one instance with the same ip,
+ // then only one master can get the lock at the same time.
Review Comment:
This is addressed now?
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894986537
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlRegistry.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.EphemeralDateManager;
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.RegistryLockManager;
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.SubscribeDataManager;
+import org.apache.dolphinscheduler.registry.api.ConnectionListener;
+import org.apache.dolphinscheduler.registry.api.Registry;
+import org.apache.dolphinscheduler.registry.api.RegistryException;
+import org.apache.dolphinscheduler.registry.api.SubscribeListener;
+
+import java.sql.SQLException;
+import java.time.Duration;
+import java.util.Collection;
+
+import javax.annotation.PostConstruct;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.stereotype.Component;
+
+
+/**
+ * This is one of the implementation of {@link Registry}, with this implementation, you need to rely on mysql database to
+ * store the DolphinScheduler master/worker's metadata and do the server registry/unRegistry.
+ */
+@Component
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "mysql")
+public class MysqlRegistry implements Registry {
+
+ private static Logger LOGGER = LoggerFactory.getLogger(MysqlRegistry.class);
+
+ private final EphemeralDateManager ephemeralDateManager;
+ private final SubscribeDataManager subscribeDataManager;
+ private final RegistryLockManager registryLockManager;
+ private final MysqlOperator mysqlOperator;
+
+ public MysqlRegistry(MysqlRegistryProperties mysqlRegistryProperties) {
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL = mysqlRegistryProperties.getTermRefreshInterval();
+ MysqlRegistryConstant.TERM_EXPIRE_TIMES = mysqlRegistryProperties.getTermExpireTimes();
Review Comment:
Hi, why do you choose to save the properties to a static class instead of injecting this `mysqlRegistryProperties` to where it's needed (as `mysqlRegistryProperties` is Spring-managed)? This is a bad practice to me as you are hard to keep `mysqlRegistryProperties` and `MysqlRegistryConstant.TERM_REFRESH_INTERVAL` in sync.
Also, as a default value of `MysqlRegistryProperties.termRefreshInterval` I don't think it necessary to add another constant if you can simply write
```java
public class MysqlRegistryProperties {
/**
* Used to schedule refresh the ephemeral data/ lock.
*/
private Duration termRefreshInterval = Duration.ofSeconds(2);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894984746
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.common.utils.NetUtils;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import javax.sql.DataSource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final DataSource dataSource;
+
+ public MysqlOperator(DataSource dataSource) {
+ this.dataSource = dataSource;
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ // if no exception, the healthCheck success
+ preparedStatement.executeQuery();
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, createTime, lastUpdateTime from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ ResultSet resultSet = preparedStatement.executeQuery();
+ List<MysqlRegistryData> result = new ArrayList<>(resultSet.getFetchSize());
+ while (resultSet.next()) {
+ MysqlRegistryData mysqlRegistryData = MysqlRegistryData.builder()
+ .id(resultSet.getLong(1))
+ .key(resultSet.getString(2))
+ .data(resultSet.getString(3))
+ .type(resultSet.getInt(4))
+ .createTime(resultSet.getTimestamp(5))
+ .lastUpdateTime(resultSet.getTimestamp(6))
Review Comment:
Good suggestion, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894986537
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlRegistry.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.EphemeralDateManager;
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.RegistryLockManager;
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.SubscribeDataManager;
+import org.apache.dolphinscheduler.registry.api.ConnectionListener;
+import org.apache.dolphinscheduler.registry.api.Registry;
+import org.apache.dolphinscheduler.registry.api.RegistryException;
+import org.apache.dolphinscheduler.registry.api.SubscribeListener;
+
+import java.sql.SQLException;
+import java.time.Duration;
+import java.util.Collection;
+
+import javax.annotation.PostConstruct;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.stereotype.Component;
+
+
+/**
+ * This is one of the implementation of {@link Registry}, with this implementation, you need to rely on mysql database to
+ * store the DolphinScheduler master/worker's metadata and do the server registry/unRegistry.
+ */
+@Component
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "mysql")
+public class MysqlRegistry implements Registry {
+
+ private static Logger LOGGER = LoggerFactory.getLogger(MysqlRegistry.class);
+
+ private final EphemeralDateManager ephemeralDateManager;
+ private final SubscribeDataManager subscribeDataManager;
+ private final RegistryLockManager registryLockManager;
+ private final MysqlOperator mysqlOperator;
+
+ public MysqlRegistry(MysqlRegistryProperties mysqlRegistryProperties) {
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL = mysqlRegistryProperties.getTermRefreshInterval();
+ MysqlRegistryConstant.TERM_EXPIRE_TIMES = mysqlRegistryProperties.getTermExpireTimes();
Review Comment:
Hi, why do you choose to save the properties to a static class instead of injecting this `mysqlRegistryProperties` to where it's needed? This is a bad practice to me as you are hard to keep `mysqlRegistryProperties` and `MysqlRegistryConstant.TERM_REFRESH_INTERVAL` in sync.
Also, as a default value of `MysqlRegistryProperties.termRefreshInterval` I don't think it necessary to add another constant if you can simply write
```java
public class MysqlRegistryProperties {
/**
* Used to schedule refresh the ephemeral data/ lock.
*/
private Duration termRefreshInterval = Duration.ofSeconds(2);
```
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/README.md:
##########
@@ -0,0 +1,33 @@
+# Introduction
+
+This module is the mysql registry plugin module, this plugin will use mysql as the registry center.
+
+# How to use
+
+If you want to set the registry center as mysql, you need to do the below two steps:
+
+1. Initialize the mysql table
+
+You can directly execute the sql script `src/main/resources/mysql_registry_init.sql`.
+
+2. Open the config
+
+You need to set the registry properties in master/worker/api's appplication.yml
+
+```yaml
+registry:
+ type: mysql
+ term-refresh-interval: 2000
Review Comment:
Use `Duration` type so users can set `2s`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894986644
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
+ PRIMARY KEY (`id`),
+ unique (`key`)
+) ENGINE = InnoDB
+ DEFAULT CHARSET = utf8;
+
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_lock`;
+CREATE TABLE `t_ds_mysql_registry_lock`
Review Comment:
> Make sense, I change the host to lock_owner, composited by host and processId.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894986931
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/README.md:
##########
@@ -0,0 +1,23 @@
+# Introduction
+
+This module is the mysql registry plugin module, this plugin will use mysql as the registry center.
+
+# How to use
+
+If you want to set the registry center as mysql, you need to do the below two steps:
+
+1. Initialize the mysql table
Review Comment:
> Yes, agree with you, I have added the datasource config, and remove the rely on dao module.
Hi, we still need to inform the users that they need to add MySQL driver manually as we can't bundle it in our distribution package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894989344
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlRegistryProperties.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Configuration;
+
+import lombok.Data;
+
+@Data
+@Configuration
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "mysql")
+@ConfigurationProperties(prefix = "registry")
+public class MysqlRegistryProperties {
+
+ /**
+ * Used to schedule refresh the ephemeral data/ lock.
+ */
+ private long termRefreshInterval = MysqlRegistryConstant.TERM_REFRESH_INTERVAL;
+ /**
+ * Used to calculate the expire time,
+ * e.g. if you set 2, and latest two refresh error, then the ephemeral data/lock will be expire.
+ */
+ private int termExpireTimes = MysqlRegistryConstant.TERM_EXPIRE_TIMES;
+ private MysqlDatasourceProperties mysqlDatasource;
Review Comment:
Good suggestion.
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlRegistryProperties.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Configuration;
+
+import lombok.Data;
+
+@Data
+@Configuration
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "mysql")
+@ConfigurationProperties(prefix = "registry")
+public class MysqlRegistryProperties {
+
+ /**
+ * Used to schedule refresh the ephemeral data/ lock.
+ */
+ private long termRefreshInterval = MysqlRegistryConstant.TERM_REFRESH_INTERVAL;
+ /**
+ * Used to calculate the expire time,
+ * e.g. if you set 2, and latest two refresh error, then the ephemeral data/lock will be expire.
+ */
+ private int termExpireTimes = MysqlRegistryConstant.TERM_EXPIRE_TIMES;
+ private MysqlDatasourceProperties mysqlDatasource;
Review Comment:
Good suggestion, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1153192632
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [4.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894986632
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
+ PRIMARY KEY (`id`),
+ unique (`key`)
+) ENGINE = InnoDB
+ DEFAULT CHARSET = utf8;
+
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_lock`;
+CREATE TABLE `t_ds_mysql_registry_lock`
Review Comment:
Make sense, I change the host to lock_owner, composite by host and processId.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894984616
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/EphemeralDateManager.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.registry.api.ConnectionListener;
+import org.apache.dolphinscheduler.registry.api.ConnectionState;
+
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This thread is used to check the connect state to mysql.
+ */
+public class EphemeralDateManager implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(EphemeralDateManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final List<ConnectionListener> connectionListeners = Collections.synchronizedList(new ArrayList<>());
+ private final Set<Long> ephemeralDateIds = Collections.synchronizedSet(new HashSet<>());
+ private final ScheduledExecutorService scheduledExecutorService;
+
+ public EphemeralDateManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ this.scheduledExecutorService = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("EphemeralDateTermRefreshThread").setDaemon(true).build());
+ mysqlOperator.clearExpireEphemeralDate();
+ }
+
+ public void start() {
+ this.scheduledExecutorService.scheduleWithFixedDelay(
+ new EphemeralDateTermRefreshTask(mysqlOperator, connectionListeners, ephemeralDateIds),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ public void addConnectionListener(ConnectionListener connectionListener) {
+ connectionListeners.add(connectionListener);
+ }
+
+ public void addEphemeralDateId(Long ephemeralDateId) {
+ ephemeralDateIds.add(ephemeralDateId);
+ }
+
+ @Override
+ public void close() throws SQLException {
+ ephemeralDateIds.clear();
+ connectionListeners.clear();
+ scheduledExecutorService.shutdownNow();
+ for (Long ephemeralDateId : ephemeralDateIds) {
+ mysqlOperator.deleteEphemeralData(ephemeralDateId);
+ }
+ }
+
+ // Use this task to refresh ephemeral term and check the connect state.
+ private static class EphemeralDateTermRefreshTask implements Runnable {
+ private final List<ConnectionListener> connectionListeners;
+ private final Set<Long> ephemeralDateIds;
+ private final MysqlOperator mysqlOperator;
+ private ConnectionState connectionState;
+
+ public EphemeralDateTermRefreshTask(MysqlOperator mysqlOperator,
+ List<ConnectionListener> connectionListeners,
+ Set<Long> ephemeralDateIds) {
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ this.connectionListeners = checkNotNull(connectionListeners);
+ this.ephemeralDateIds = checkNotNull(ephemeralDateIds);
+ }
+
+ @Override
+ public void run() {
+ try {
+ ConnectionState currentConnectionState = getConnectionState();
+ if (currentConnectionState == connectionState) {
+ // no state change
+ return;
+ }
+ if (connectionState == null) {
+ // first time connect
+ if (currentConnectionState == ConnectionState.CONNECTED) {
+ connectionState = ConnectionState.CONNECTED;
+ triggerListener(ConnectionState.CONNECTED);
+ }
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894985275
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/README.md:
##########
@@ -0,0 +1,23 @@
+# Introduction
+
+This module is the mysql registry plugin module, this plugin will use mysql as the registry center.
+
+# How to use
+
+If you want to set the registry center as mysql, you need to do the below two steps:
+
+1. Initialize the mysql table
Review Comment:
Yes, agree with you, I have added the datasource config, and remove the rely on dao module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1153197300
> > SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
> > [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
> > [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [5.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&v
iew=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
>
> @ruanwenjun please take a look of the Security Hotspots.
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894669035
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/README.md:
##########
@@ -0,0 +1,23 @@
+# Introduction
+
+This module is the mysql registry plugin module, this plugin will use mysql as the registry center.
+
+# How to use
+
+If you want to set the registry center as mysql, you need to do the below two steps:
+
+1. Initialize the mysql table
+
+You can directly execute the sql script `src/main/resources/mysql_registry_init.sql`.
+
+2. Open the config
+
+You need to set the registry type to mysql in master/worker/api's appplication.yml
+
+```yaml
+registry:
+ type: zookeeper
Review Comment:
```suggestion
type: mysql
```
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/pom.xml:
##########
@@ -0,0 +1,56 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to Apache Software Foundation (ASF) under one or more contributor
+ ~ license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright
+ ~ ownership. Apache Software Foundation (ASF) licenses this file to you under
+ ~ the Apache License, Version 2.0 (the "License"); you may
+ ~ not use this file except in compliance with the License.
+ ~ You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-plugins</artifactId>
+ <version>dev-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>dolphinscheduler-registry-mysql</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-common</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-dao</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
Review Comment:
If you want to use this. Add this to the root module (not in `<dependencyManagement>`, but in `<dependencies>`). We don't want to add it again and again in every module
##########
pom.xml:
##########
@@ -850,6 +856,13 @@
<version>${aws.sdk.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ <version>${lombok.version}</version>
+ <scope>provided</scope>
+ </dependency>
Review Comment:
Move this out from `<dependencyManagement` and move it into `<dependencies>`
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlRegistryConstant.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import lombok.NoArgsConstructor;
+
+@NoArgsConstructor
Review Comment:
I think you need `@UtilityClass`. `@NoArgsConstructor` means it will generate a constructor with no argument.
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
Review Comment:
column names are usually snake_case
##########
dolphinscheduler-api/pom.xml:
##########
@@ -64,6 +64,10 @@
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-registry-zookeeper</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-registry-mysql</artifactId>
+ </dependency>
Review Comment:
Can you do in the same way as how we did in the task plugins? Create a new module `dolphinscheduler-registry-all` to contain all these registry plugin, and we can only add `dolphinscheduler-registry-all` to these starter module, in the future when we add new registry module we can only add them in `dolphinscheduler-registry-all` once
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/RegistryLockManager.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.common.thread.ThreadUtils;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+import org.apache.dolphinscheduler.registry.api.RegistryException;
+
+import java.sql.SQLException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class RegistryLockManager implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(RegistryLockManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final Map<String, MysqlRegistryLock> lockHoldMap;
+ private final ScheduledExecutorService lockTermUpdateThreadPool;
+
+ public RegistryLockManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = mysqlOperator;
+ mysqlOperator.clearExpireLock();
+ this.lockHoldMap = new ConcurrentHashMap<>();
+ this.lockTermUpdateThreadPool = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("MysqlRegistryLockTermRefreshThread").setDaemon(true).build());
+ }
+
+ public void start() {
+ lockTermUpdateThreadPool.scheduleWithFixedDelay(
+ new LockTermRefreshTask(lockHoldMap, mysqlOperator),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Acquire the lock, if cannot get the lock will await.
+ */
+ public void acquireLock(String lockKey) throws RegistryException {
+ // maybe we can use the computeIf absent
+ lockHoldMap.computeIfAbsent(lockKey, key -> {
+ MysqlRegistryLock mysqlRegistryLock;
+ try {
+ while ((mysqlRegistryLock = mysqlOperator.tryToAcquireLock(lockKey)) == null) {
+ logger.debug("Acquire the lock {} failed try again", key);
+ // acquire failed, wait and try again
+ ThreadUtils.sleep(1_000L);
+ }
+ } catch (SQLException e) {
+ throw new RegistryException("Acquire the lock error", e);
+ }
+ return mysqlRegistryLock;
+ });
+ }
+
+ public void releaseLock(String lockKey) {
+ MysqlRegistryLock mysqlRegistryLock = lockHoldMap.get(lockKey);
+ if (mysqlRegistryLock != null) {
+ try {
+ // the lock is unExit
+ mysqlOperator.releaseLock(mysqlRegistryLock.getId());
+ lockHoldMap.remove(lockKey);
+ } catch (SQLException e) {
+ throw new RegistryException(String.format("Release lock: %s error", lockKey), e);
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ lockTermUpdateThreadPool.shutdownNow();
+ for (Map.Entry<String, MysqlRegistryLock> lockEntry : lockHoldMap.entrySet()) {
+ releaseLock(lockEntry.getKey());
+ }
+ }
+
+ /**
+ * This task is used to refresh the lock held by the current server.
+ */
+ static class LockTermRefreshTask implements Runnable {
+ private final Map<String, MysqlRegistryLock> lockHoldMap;
+ private final MysqlOperator mysqlOperator;
+
+ private LockTermRefreshTask(Map<String, MysqlRegistryLock> lockHoldMap, MysqlOperator mysqlOperator) {
+ this.lockHoldMap = checkNotNull(lockHoldMap);
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ }
+
Review Comment:
Replace these with `@RequiredArgsConstructor(access = AccessLevel.PRIVATE)` in the class
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/SubscribeDataManager.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.registry.api.Event;
+import org.apache.dolphinscheduler.registry.api.SubscribeListener;
+
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * Used to refresh if the subscribe path has been changed.
+ */
+public class SubscribeDataManager implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SubscribeDataManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final Map<String, List<SubscribeListener>> dataSubScribeMap = new ConcurrentHashMap<>();
+ private final ScheduledExecutorService dataSubscribeCheckThreadPool;
+ private final Map<String, MysqlRegistryData> mysqlRegistryDataMap = new ConcurrentHashMap<>();
+
+ public SubscribeDataManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = mysqlOperator;
+ this.dataSubscribeCheckThreadPool = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("MysqlRegistrySubscribeDataCheckThread").setDaemon(true).build());
+ }
+
+ public void start() {
+ dataSubscribeCheckThreadPool.scheduleWithFixedDelay(
+ new RegistrySubscribeDataCheckTask(mysqlRegistryDataMap, dataSubScribeMap, mysqlOperator),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ public void addListener(String path, SubscribeListener subscribeListener) {
+ dataSubScribeMap.computeIfAbsent(path, k -> new ArrayList<>()).add(subscribeListener);
+ }
+
+ public void removeListener(String path) {
+ dataSubScribeMap.remove(path);
+ }
+
+ public String getData(String path) throws SQLException {
+ MysqlRegistryData mysqlRegistryData = mysqlRegistryDataMap.get(path);
+ if (mysqlRegistryData == null) {
+ return null;
+ }
+ return mysqlRegistryData.getData();
+ }
+
+ @Override
+ public void close() {
+ dataSubscribeCheckThreadPool.shutdownNow();
+ dataSubScribeMap.clear();
+ }
+
+ static class RegistrySubscribeDataCheckTask implements Runnable {
+
+ private final Map<String, List<SubscribeListener>> dataSubScribeMap;
+ private final MysqlOperator mysqlOperator;
+ private final Map<String, MysqlRegistryData> mysqlRegistryDataMap;
+
+ public RegistrySubscribeDataCheckTask(
+ Map<String, MysqlRegistryData> mysqlRegistryDataMap,
+ Map<String, List<SubscribeListener>> dataSubScribeMap,
+ MysqlOperator mysqlOperator) {
+ this.mysqlRegistryDataMap = checkNotNull(mysqlRegistryDataMap);
+ this.dataSubScribeMap = checkNotNull(dataSubScribeMap);
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ }
Review Comment:
Replace this with `@RequiredArgsConstructor`
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.common.utils.NetUtils;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import javax.sql.DataSource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final DataSource dataSource;
+
+ public MysqlOperator(DataSource dataSource) {
+ this.dataSource = dataSource;
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ // if no exception, the healthCheck success
+ preparedStatement.executeQuery();
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, createTime, lastUpdateTime from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ ResultSet resultSet = preparedStatement.executeQuery();
+ List<MysqlRegistryData> result = new ArrayList<>(resultSet.getFetchSize());
+ while (resultSet.next()) {
+ MysqlRegistryData mysqlRegistryData = MysqlRegistryData.builder()
+ .id(resultSet.getLong(1))
+ .key(resultSet.getString(2))
+ .data(resultSet.getString(3))
+ .type(resultSet.getInt(4))
+ .createTime(resultSet.getTimestamp(5))
+ .lastUpdateTime(resultSet.getTimestamp(6))
Review Comment:
What about getting by column label instead of the column index?
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-zookeeper/src/main/java/org/apache/dolphinscheduler/plugin/registry/zookeeper/ZookeeperRegistryProperties.java:
##########
@@ -1,33 +1,32 @@
/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * http://www.apache.org/licenses/LICENSE-2.0
*
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
*/
-package org.apache.dolphinscheduler.registry.api;
+package org.apache.dolphinscheduler.plugin.registry.zookeeper;
import java.time.Duration;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Configuration
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "zookeeper")
Review Comment:
I think you can remove the `type` field in this class then
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/README.md:
##########
@@ -0,0 +1,23 @@
+# Introduction
+
+This module is the mysql registry plugin module, this plugin will use mysql as the registry center.
+
+# How to use
+
+If you want to set the registry center as mysql, you need to do the below two steps:
+
+1. Initialize the mysql table
Review Comment:
I think users need to add MySQL driver manually, configure the connection string, etc., if they are not already using MySQL as database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1152840844
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10406?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10406](https://codecov.io/gh/apache/dolphinscheduler/pull/10406?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4f9e262) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/b0d9d3f9ab20f822ce019eb73c893ad8811d14a9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b0d9d3f) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## dev #10406 +/- ##
=========================================
Coverage 40.56% 40.57%
- Complexity 4767 4768 +1
=========================================
Files 877 878 +1
Lines 35623 35671 +48
Branches 3945 3945
=========================================
+ Hits 14452 14474 +22
- Misses 19738 19761 +23
- Partials 1433 1436 +3
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10406?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/plugin/registry/zookeeper/ZookeeperRegistry.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS9kb2xwaGluc2NoZWR1bGVyLXJlZ2lzdHJ5LXBsdWdpbnMvZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS16b29rZWVwZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3JlZ2lzdHJ5L3pvb2tlZXBlci9ab29rZWVwZXJSZWdpc3RyeS5qYXZh) | `48.21% <100.00%> (ø)` | |
| [...egistry/zookeeper/ZookeeperRegistryProperties.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS9kb2xwaGluc2NoZWR1bGVyLXJlZ2lzdHJ5LXBsdWdpbnMvZG9scGhpbnNjaGVkdWxlci1yZWdpc3RyeS16b29rZWVwZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3JlZ2lzdHJ5L3pvb2tlZXBlci9ab29rZWVwZXJSZWdpc3RyeVByb3BlcnRpZXMuamF2YQ==) | `47.91% <100.00%> (ø)` | |
| [...dolphinscheduler/remote/future/ResponseFuture.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2Z1dHVyZS9SZXNwb25zZUZ1dHVyZS5qYXZh) | `81.96% <0.00%> (-1.64%)` | :arrow_down: |
| [...r/plugin/task/sqoop/parameter/SqoopParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Fvb3Avc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3Fvb3AvcGFyYW1ldGVyL1Nxb29wUGFyYW1ldGVycy5qYXZh) | `53.33% <0.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10406?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10406?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b0d9d3f...4f9e262](https://codecov.io/gh/apache/dolphinscheduler/pull/10406?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894985003
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.common.utils.NetUtils;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import javax.sql.DataSource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final DataSource dataSource;
+
+ public MysqlOperator(DataSource dataSource) {
+ this.dataSource = dataSource;
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ // if no exception, the healthCheck success
+ preparedStatement.executeQuery();
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, createTime, lastUpdateTime from t_ds_mysql_registry_data";
Review Comment:
This module will not rely on dao module, if we hope to use mybatis here may need to do other config like mapper scan.
This can be optimized latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894987247
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final HikariDataSource dataSource;
+
+ public MysqlOperator(MysqlRegistryProperties.MysqlDatasourceProperties datasourceProperties) {
+ HikariConfig hikariConfig = new HikariConfig();
+ hikariConfig.setDriverClassName(datasourceProperties.getDriverClassName());
+ hikariConfig.setJdbcUrl(datasourceProperties.getUrl());
+ hikariConfig.setUsername(datasourceProperties.getUsername());
+ hikariConfig.setPassword(datasourceProperties.getPassword());
+ hikariConfig.setMaximumPoolSize(datasourceProperties.getMaximumPoolSize());
+ hikariConfig.setConnectionTimeout(datasourceProperties.getConnectionTimeout());
+ hikariConfig.setIdleTimeout(datasourceProperties.getIdleTimeout());
+ hikariConfig.setPoolName("MysqlRegistryDataSourcePool");
+
+ this.dataSource = new HikariDataSource(hikariConfig);
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ // if no exception, the healthCheck success
+ preparedStatement.executeQuery();
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, create_time, last_update_time from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ ResultSet resultSet = preparedStatement.executeQuery();
Review Comment:
Also put `resultSet` in the try-with-resource block?
```suggestion
try (Connection connection = dataSource.getConnection();
PreparedStatement preparedStatement = connection.prepareStatement(sql);
ResultSet resultSet = preparedStatement.executeQuery()) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894989316
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlRegistry.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.EphemeralDateManager;
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.RegistryLockManager;
+import org.apache.dolphinscheduler.plugin.registry.mysql.task.SubscribeDataManager;
+import org.apache.dolphinscheduler.registry.api.ConnectionListener;
+import org.apache.dolphinscheduler.registry.api.Registry;
+import org.apache.dolphinscheduler.registry.api.RegistryException;
+import org.apache.dolphinscheduler.registry.api.SubscribeListener;
+
+import java.sql.SQLException;
+import java.time.Duration;
+import java.util.Collection;
+
+import javax.annotation.PostConstruct;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.stereotype.Component;
+
+
+/**
+ * This is one of the implementation of {@link Registry}, with this implementation, you need to rely on mysql database to
+ * store the DolphinScheduler master/worker's metadata and do the server registry/unRegistry.
+ */
+@Component
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "mysql")
+public class MysqlRegistry implements Registry {
+
+ private static Logger LOGGER = LoggerFactory.getLogger(MysqlRegistry.class);
+
+ private final EphemeralDateManager ephemeralDateManager;
+ private final SubscribeDataManager subscribeDataManager;
+ private final RegistryLockManager registryLockManager;
+ private final MysqlOperator mysqlOperator;
+
+ public MysqlRegistry(MysqlRegistryProperties mysqlRegistryProperties) {
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL = mysqlRegistryProperties.getTermRefreshInterval();
+ MysqlRegistryConstant.TERM_EXPIRE_TIMES = mysqlRegistryProperties.getTermExpireTimes();
Review Comment:
Yes, this is a bad practice, I updated as you suggest.
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/README.md:
##########
@@ -0,0 +1,33 @@
+# Introduction
+
+This module is the mysql registry plugin module, this plugin will use mysql as the registry center.
+
+# How to use
+
+If you want to set the registry center as mysql, you need to do the below two steps:
+
+1. Initialize the mysql table
+
+You can directly execute the sql script `src/main/resources/mysql_registry_init.sql`.
+
+2. Open the config
+
+You need to set the registry properties in master/worker/api's appplication.yml
+
+```yaml
+registry:
+ type: mysql
+ term-refresh-interval: 2000
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1152901130
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [5.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#issuecomment-1153168763
> SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10406)
>
> [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10406&resolved=false&types=CODE_SMELL)
>
> [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [5.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_coverage&view=list) [](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&vie
w=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10406&metric=new_duplicated_lines_density&view=list)
@ruanwenjun please take a look of the Security Hotspots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894971000
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.common.utils.NetUtils;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import javax.sql.DataSource;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final DataSource dataSource;
+
+ public MysqlOperator(DataSource dataSource) {
+ this.dataSource = dataSource;
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ // if no exception, the healthCheck success
+ preparedStatement.executeQuery();
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, createTime, lastUpdateTime from t_ds_mysql_registry_data";
Review Comment:
Can the style of mapper be used uniformly here?
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/RegistryLockManager.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.common.thread.ThreadUtils;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+import org.apache.dolphinscheduler.registry.api.RegistryException;
+
+import java.sql.SQLException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class RegistryLockManager implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(RegistryLockManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final Map<String, MysqlRegistryLock> lockHoldMap;
+ private final ScheduledExecutorService lockTermUpdateThreadPool;
+
+ public RegistryLockManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = mysqlOperator;
+ mysqlOperator.clearExpireLock();
+ this.lockHoldMap = new ConcurrentHashMap<>();
+ this.lockTermUpdateThreadPool = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("MysqlRegistryLockTermRefreshThread").setDaemon(true).build());
+ }
+
+ public void start() {
+ lockTermUpdateThreadPool.scheduleWithFixedDelay(
+ new LockTermRefreshTask(lockHoldMap, mysqlOperator),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Acquire the lock, if cannot get the lock will await.
+ */
+ public void acquireLock(String lockKey) throws RegistryException {
+ // maybe we can use the computeIf absent
+ lockHoldMap.computeIfAbsent(lockKey, key -> {
+ MysqlRegistryLock mysqlRegistryLock;
+ try {
+ while ((mysqlRegistryLock = mysqlOperator.tryToAcquireLock(lockKey)) == null) {
+ logger.debug("Acquire the lock {} failed try again", key);
+ // acquire failed, wait and try again
+ ThreadUtils.sleep(1_000L);
Review Comment:
It's better to use the constant.
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
+ PRIMARY KEY (`id`),
+ unique (`key`)
+) ENGINE = InnoDB
+ DEFAULT CHARSET = utf8;
+
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_lock`;
+CREATE TABLE `t_ds_mysql_registry_lock`
Review Comment:
should we add the lock owner to avoid deleting lock by mistake?
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/EphemeralDateManager.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.registry.api.ConnectionListener;
+import org.apache.dolphinscheduler.registry.api.ConnectionState;
+
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This thread is used to check the connect state to mysql.
+ */
+public class EphemeralDateManager implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(EphemeralDateManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final List<ConnectionListener> connectionListeners = Collections.synchronizedList(new ArrayList<>());
+ private final Set<Long> ephemeralDateIds = Collections.synchronizedSet(new HashSet<>());
+ private final ScheduledExecutorService scheduledExecutorService;
+
+ public EphemeralDateManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ this.scheduledExecutorService = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("EphemeralDateTermRefreshThread").setDaemon(true).build());
+ mysqlOperator.clearExpireEphemeralDate();
+ }
+
+ public void start() {
+ this.scheduledExecutorService.scheduleWithFixedDelay(
+ new EphemeralDateTermRefreshTask(mysqlOperator, connectionListeners, ephemeralDateIds),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ public void addConnectionListener(ConnectionListener connectionListener) {
+ connectionListeners.add(connectionListener);
+ }
+
+ public void addEphemeralDateId(Long ephemeralDateId) {
+ ephemeralDateIds.add(ephemeralDateId);
+ }
+
+ @Override
+ public void close() throws SQLException {
+ ephemeralDateIds.clear();
+ connectionListeners.clear();
+ scheduledExecutorService.shutdownNow();
+ for (Long ephemeralDateId : ephemeralDateIds) {
+ mysqlOperator.deleteEphemeralData(ephemeralDateId);
+ }
+ }
+
+ // Use this task to refresh ephemeral term and check the connect state.
+ private static class EphemeralDateTermRefreshTask implements Runnable {
+ private final List<ConnectionListener> connectionListeners;
+ private final Set<Long> ephemeralDateIds;
+ private final MysqlOperator mysqlOperator;
+ private ConnectionState connectionState;
+
+ public EphemeralDateTermRefreshTask(MysqlOperator mysqlOperator,
+ List<ConnectionListener> connectionListeners,
+ Set<Long> ephemeralDateIds) {
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ this.connectionListeners = checkNotNull(connectionListeners);
+ this.ephemeralDateIds = checkNotNull(ephemeralDateIds);
+ }
+
+ @Override
+ public void run() {
+ try {
+ ConnectionState currentConnectionState = getConnectionState();
+ if (currentConnectionState == connectionState) {
+ // no state change
+ return;
+ }
+ if (connectionState == null) {
+ // first time connect
+ if (currentConnectionState == ConnectionState.CONNECTED) {
+ connectionState = ConnectionState.CONNECTED;
+ triggerListener(ConnectionState.CONNECTED);
+ }
+ } else {
+ // already connect before
+ if (connectionState == ConnectionState.CONNECTED && currentConnectionState == ConnectionState.DISCONNECTED) {
+ connectionState = ConnectionState.DISCONNECTED;
+ triggerListener(ConnectionState.DISCONNECTED);
+ } else if (connectionState == ConnectionState.DISCONNECTED && currentConnectionState == ConnectionState.CONNECTED) {
+ connectionState = ConnectionState.CONNECTED;
+ triggerListener(ConnectionState.RECONNECTED);
+ }
Review Comment:
same here
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/EphemeralDateManager.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.registry.api.ConnectionListener;
+import org.apache.dolphinscheduler.registry.api.ConnectionState;
+
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This thread is used to check the connect state to mysql.
+ */
+public class EphemeralDateManager implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(EphemeralDateManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final List<ConnectionListener> connectionListeners = Collections.synchronizedList(new ArrayList<>());
+ private final Set<Long> ephemeralDateIds = Collections.synchronizedSet(new HashSet<>());
+ private final ScheduledExecutorService scheduledExecutorService;
+
+ public EphemeralDateManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ this.scheduledExecutorService = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("EphemeralDateTermRefreshThread").setDaemon(true).build());
+ mysqlOperator.clearExpireEphemeralDate();
+ }
+
+ public void start() {
+ this.scheduledExecutorService.scheduleWithFixedDelay(
+ new EphemeralDateTermRefreshTask(mysqlOperator, connectionListeners, ephemeralDateIds),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ public void addConnectionListener(ConnectionListener connectionListener) {
+ connectionListeners.add(connectionListener);
+ }
+
+ public void addEphemeralDateId(Long ephemeralDateId) {
+ ephemeralDateIds.add(ephemeralDateId);
+ }
+
+ @Override
+ public void close() throws SQLException {
+ ephemeralDateIds.clear();
+ connectionListeners.clear();
+ scheduledExecutorService.shutdownNow();
+ for (Long ephemeralDateId : ephemeralDateIds) {
+ mysqlOperator.deleteEphemeralData(ephemeralDateId);
+ }
+ }
+
+ // Use this task to refresh ephemeral term and check the connect state.
+ private static class EphemeralDateTermRefreshTask implements Runnable {
+ private final List<ConnectionListener> connectionListeners;
+ private final Set<Long> ephemeralDateIds;
+ private final MysqlOperator mysqlOperator;
+ private ConnectionState connectionState;
+
+ public EphemeralDateTermRefreshTask(MysqlOperator mysqlOperator,
+ List<ConnectionListener> connectionListeners,
+ Set<Long> ephemeralDateIds) {
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ this.connectionListeners = checkNotNull(connectionListeners);
+ this.ephemeralDateIds = checkNotNull(ephemeralDateIds);
+ }
+
+ @Override
+ public void run() {
+ try {
+ ConnectionState currentConnectionState = getConnectionState();
+ if (currentConnectionState == connectionState) {
+ // no state change
+ return;
+ }
+ if (connectionState == null) {
+ // first time connect
+ if (currentConnectionState == ConnectionState.CONNECTED) {
+ connectionState = ConnectionState.CONNECTED;
+ triggerListener(ConnectionState.CONNECTED);
+ }
Review Comment:
Should it add some else logic if `currentConnectState != ConnectionState.CONNECTED`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894984799
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/RegistryLockManager.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.common.thread.ThreadUtils;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+import org.apache.dolphinscheduler.registry.api.RegistryException;
+
+import java.sql.SQLException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class RegistryLockManager implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(RegistryLockManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final Map<String, MysqlRegistryLock> lockHoldMap;
+ private final ScheduledExecutorService lockTermUpdateThreadPool;
+
+ public RegistryLockManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = mysqlOperator;
+ mysqlOperator.clearExpireLock();
+ this.lockHoldMap = new ConcurrentHashMap<>();
+ this.lockTermUpdateThreadPool = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("MysqlRegistryLockTermRefreshThread").setDaemon(true).build());
+ }
+
+ public void start() {
+ lockTermUpdateThreadPool.scheduleWithFixedDelay(
+ new LockTermRefreshTask(lockHoldMap, mysqlOperator),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ /**
+ * Acquire the lock, if cannot get the lock will await.
+ */
+ public void acquireLock(String lockKey) throws RegistryException {
+ // maybe we can use the computeIf absent
+ lockHoldMap.computeIfAbsent(lockKey, key -> {
+ MysqlRegistryLock mysqlRegistryLock;
+ try {
+ while ((mysqlRegistryLock = mysqlOperator.tryToAcquireLock(lockKey)) == null) {
+ logger.debug("Acquire the lock {} failed try again", key);
+ // acquire failed, wait and try again
+ ThreadUtils.sleep(1_000L);
+ }
+ } catch (SQLException e) {
+ throw new RegistryException("Acquire the lock error", e);
+ }
+ return mysqlRegistryLock;
+ });
+ }
+
+ public void releaseLock(String lockKey) {
+ MysqlRegistryLock mysqlRegistryLock = lockHoldMap.get(lockKey);
+ if (mysqlRegistryLock != null) {
+ try {
+ // the lock is unExit
+ mysqlOperator.releaseLock(mysqlRegistryLock.getId());
+ lockHoldMap.remove(lockKey);
+ } catch (SQLException e) {
+ throw new RegistryException(String.format("Release lock: %s error", lockKey), e);
+ }
+ }
+ }
+
+ @Override
+ public void close() {
+ lockTermUpdateThreadPool.shutdownNow();
+ for (Map.Entry<String, MysqlRegistryLock> lockEntry : lockHoldMap.entrySet()) {
+ releaseLock(lockEntry.getKey());
+ }
+ }
+
+ /**
+ * This task is used to refresh the lock held by the current server.
+ */
+ static class LockTermRefreshTask implements Runnable {
+ private final Map<String, MysqlRegistryLock> lockHoldMap;
+ private final MysqlOperator mysqlOperator;
+
+ private LockTermRefreshTask(Map<String, MysqlRegistryLock> lockHoldMap, MysqlOperator mysqlOperator) {
+ this.lockHoldMap = checkNotNull(lockHoldMap);
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ }
+
Review Comment:
Done.
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894984811
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/task/SubscribeDataManager.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql.task;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlOperator;
+import org.apache.dolphinscheduler.plugin.registry.mysql.MysqlRegistryConstant;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.registry.api.Event;
+import org.apache.dolphinscheduler.registry.api.SubscribeListener;
+
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * Used to refresh if the subscribe path has been changed.
+ */
+public class SubscribeDataManager implements AutoCloseable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(SubscribeDataManager.class);
+
+ private final MysqlOperator mysqlOperator;
+ private final Map<String, List<SubscribeListener>> dataSubScribeMap = new ConcurrentHashMap<>();
+ private final ScheduledExecutorService dataSubscribeCheckThreadPool;
+ private final Map<String, MysqlRegistryData> mysqlRegistryDataMap = new ConcurrentHashMap<>();
+
+ public SubscribeDataManager(MysqlOperator mysqlOperator) {
+ this.mysqlOperator = mysqlOperator;
+ this.dataSubscribeCheckThreadPool = Executors.newScheduledThreadPool(
+ 1,
+ new ThreadFactoryBuilder().setNameFormat("MysqlRegistrySubscribeDataCheckThread").setDaemon(true).build());
+ }
+
+ public void start() {
+ dataSubscribeCheckThreadPool.scheduleWithFixedDelay(
+ new RegistrySubscribeDataCheckTask(mysqlRegistryDataMap, dataSubScribeMap, mysqlOperator),
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ MysqlRegistryConstant.TERM_REFRESH_INTERVAL,
+ TimeUnit.MILLISECONDS);
+ }
+
+ public void addListener(String path, SubscribeListener subscribeListener) {
+ dataSubScribeMap.computeIfAbsent(path, k -> new ArrayList<>()).add(subscribeListener);
+ }
+
+ public void removeListener(String path) {
+ dataSubScribeMap.remove(path);
+ }
+
+ public String getData(String path) throws SQLException {
+ MysqlRegistryData mysqlRegistryData = mysqlRegistryDataMap.get(path);
+ if (mysqlRegistryData == null) {
+ return null;
+ }
+ return mysqlRegistryData.getData();
+ }
+
+ @Override
+ public void close() {
+ dataSubscribeCheckThreadPool.shutdownNow();
+ dataSubScribeMap.clear();
+ }
+
+ static class RegistrySubscribeDataCheckTask implements Runnable {
+
+ private final Map<String, List<SubscribeListener>> dataSubScribeMap;
+ private final MysqlOperator mysqlOperator;
+ private final Map<String, MysqlRegistryData> mysqlRegistryDataMap;
+
+ public RegistrySubscribeDataCheckTask(
+ Map<String, MysqlRegistryData> mysqlRegistryDataMap,
+ Map<String, List<SubscribeListener>> dataSubScribeMap,
+ MysqlOperator mysqlOperator) {
+ this.mysqlRegistryDataMap = checkNotNull(mysqlRegistryDataMap);
+ this.dataSubScribeMap = checkNotNull(dataSubScribeMap);
+ this.mysqlOperator = checkNotNull(mysqlOperator);
+ }
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894984781
##########
pom.xml:
##########
@@ -850,6 +856,13 @@
<version>${aws.sdk.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ <version>${lombok.version}</version>
+ <scope>provided</scope>
+ </dependency>
Review Comment:
Done.
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-zookeeper/src/main/java/org/apache/dolphinscheduler/plugin/registry/zookeeper/ZookeeperRegistryProperties.java:
##########
@@ -1,33 +1,32 @@
/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
+ * http://www.apache.org/licenses/LICENSE-2.0
*
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
*/
-package org.apache.dolphinscheduler.registry.api;
+package org.apache.dolphinscheduler.plugin.registry.zookeeper;
import java.time.Duration;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Configuration
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "zookeeper")
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894987111
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlRegistryProperties.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Configuration;
+
+import lombok.Data;
+
+@Data
+@Configuration
+@ConditionalOnProperty(prefix = "registry", name = "type", havingValue = "mysql")
+@ConfigurationProperties(prefix = "registry")
+public class MysqlRegistryProperties {
+
+ /**
+ * Used to schedule refresh the ephemeral data/ lock.
+ */
+ private long termRefreshInterval = MysqlRegistryConstant.TERM_REFRESH_INTERVAL;
+ /**
+ * Used to calculate the expire time,
+ * e.g. if you set 2, and latest two refresh error, then the ephemeral data/lock will be expire.
+ */
+ private int termExpireTimes = MysqlRegistryConstant.TERM_EXPIRE_TIMES;
+ private MysqlDatasourceProperties mysqlDatasource;
Review Comment:
What about simply using `HikariConfig` so we don't need to copy over the properties again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894986644
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/resources/mysql_registry_init.sql:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+SET FOREIGN_KEY_CHECKS = 0;
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_data`;
+CREATE TABLE `t_ds_mysql_registry_data`
+(
+ `id` bigint(11) NOT NULL AUTO_INCREMENT COMMENT 'primary key',
+ `key` varchar(200) NOT NULL COMMENT 'key, like zookeeper node path',
+ `data` varchar(200) NOT NULL COMMENT 'data, like zookeeper node value',
+ `type` tinyint(4) NOT NULL COMMENT '1: ephemeral node, 2: persistent node',
+ `lastUpdateTime` timestamp NULL COMMENT 'last update time',
+ `createTime` timestamp NULL COMMENT 'create time',
+ PRIMARY KEY (`id`),
+ unique (`key`)
+) ENGINE = InnoDB
+ DEFAULT CHARSET = utf8;
+
+
+DROP TABLE IF EXISTS `t_ds_mysql_registry_lock`;
+CREATE TABLE `t_ds_mysql_registry_lock`
Review Comment:
> Make sense, I change the host to lock_owner, composited by host and processId.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun merged pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun merged PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10406: Add mysql registry plugin
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #10406:
URL: https://github.com/apache/dolphinscheduler/pull/10406#discussion_r894989357
##########
dolphinscheduler-registry/dolphinscheduler-registry-plugins/dolphinscheduler-registry-mysql/src/main/java/org/apache/dolphinscheduler/plugin/registry/mysql/MysqlOperator.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.registry.mysql;
+
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.DataType;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryData;
+import org.apache.dolphinscheduler.plugin.registry.mysql.model.MysqlRegistryLock;
+
+import org.apache.commons.lang3.StringUtils;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.SQLIntegrityConstraintViolationException;
+import java.sql.Statement;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.zaxxer.hikari.HikariConfig;
+import com.zaxxer.hikari.HikariDataSource;
+
+/**
+ * Used to CRUD from mysql
+ */
+public class MysqlOperator implements AutoCloseable {
+
+ private static final Logger logger = LoggerFactory.getLogger(MysqlOperator.class);
+
+ private final HikariDataSource dataSource;
+
+ public MysqlOperator(MysqlRegistryProperties.MysqlDatasourceProperties datasourceProperties) {
+ HikariConfig hikariConfig = new HikariConfig();
+ hikariConfig.setDriverClassName(datasourceProperties.getDriverClassName());
+ hikariConfig.setJdbcUrl(datasourceProperties.getUrl());
+ hikariConfig.setUsername(datasourceProperties.getUsername());
+ hikariConfig.setPassword(datasourceProperties.getPassword());
+ hikariConfig.setMaximumPoolSize(datasourceProperties.getMaximumPoolSize());
+ hikariConfig.setConnectionTimeout(datasourceProperties.getConnectionTimeout());
+ hikariConfig.setIdleTimeout(datasourceProperties.getIdleTimeout());
+ hikariConfig.setPoolName("MysqlRegistryDataSourcePool");
+
+ this.dataSource = new HikariDataSource(hikariConfig);
+ }
+
+ public void healthCheck() throws SQLException {
+ String sql = "select 1 from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ // if no exception, the healthCheck success
+ preparedStatement.executeQuery();
+ }
+ }
+
+ public List<MysqlRegistryData> queryAllMysqlRegistryData() throws SQLException {
+ String sql = "select id, `key`, data, type, create_time, last_update_time from t_ds_mysql_registry_data";
+ try (Connection connection = dataSource.getConnection();
+ PreparedStatement preparedStatement = connection.prepareStatement(sql)) {
+ ResultSet resultSet = preparedStatement.executeQuery();
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org