You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/05/29 07:18:52 UTC

[GitHub] [dolphinscheduler] Tianqi-Dotes opened a new pull request, #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Tianqi-Dotes opened a new pull request, #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276

   <!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
   
   
   ## Purpose of the pull request
   
   replace DataSourceController API with string JSON and cast to DTO
   
   ## Brief change log
   
   replace DataSourceController API with string JSON and cast to DTO
   
   ## Verify this pull request
   
   test locally


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276#issuecomment-1140684400

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10276)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10276&resolved=false&types=CODE_SMELL)
   
   [![65.1%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60-16px.png '65.1%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10276&metric=new_coverage&view=list) [65.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10276&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10276&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10276&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276#issuecomment-1140679658

   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10276?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10276](https://codecov.io/gh/apache/dolphinscheduler/pull/10276?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (00a5a8b) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/4c1ef0a89a81e3e7464300a7d010a4786abd3e7b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4c1ef0a) will **decrease** coverage by `0.10%`.
   > The diff coverage is `48.76%`.
   
   > :exclamation: Current head 00a5a8b differs from pull request most recent head 7b0019f. Consider uploading reports for the commit 7b0019f to get more accurate results
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev   #10276      +/-   ##
   ============================================
   - Coverage     41.25%   41.15%   -0.11%     
   - Complexity     4760     4772      +12     
   ============================================
     Files           854      857       +3     
     Lines         34579    34659      +80     
     Branches       3821     3828       +7     
   ============================================
   - Hits          14265    14263       -2     
   - Misses        18944    19019      +75     
   - Partials       1370     1377       +7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10276?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/dolphinscheduler/api/ApiApplicationServer.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL0FwaUFwcGxpY2F0aW9uU2VydmVyLmphdmE=) | `60.00% <ø> (ø)` | |
   | [...scheduler/api/controller/DataSourceController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvRGF0YVNvdXJjZUNvbnRyb2xsZXIuamF2YQ==) | `38.70% <0.00%> (-4.15%)` | :arrow_down: |
   | [...lphinscheduler/common/enums/AuthorizationType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2VudW1zL0F1dGhvcml6YXRpb25UeXBlLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...asource/api/datasource/BaseDataSourceParamDTO.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYXRhc291cmNlLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLWRhdGFzb3VyY2UtYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi9kYXRhc291cmNlL2FwaS9kYXRhc291cmNlL0Jhc2VEYXRhU291cmNlUGFyYW1EVE8uamF2YQ==) | `78.57% <ø> (ø)` | |
   | [.../org/apache/dolphinscheduler/spi/enums/DbType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc3BpL2VudW1zL0RiVHlwZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...permission/ResourcePermissionCheckServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2UvcGVybWlzc2lvbi9SZXNvdXJjZVBlcm1pc3Npb25DaGVja1NlcnZpY2VJbXBsLmphdmE=) | `24.00% <24.00%> (ø)` | |
   | [...r/plugin/datasource/api/utils/DataSourceUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYXRhc291cmNlLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLWRhdGFzb3VyY2UtYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi9kYXRhc291cmNlL2FwaS91dGlscy9EYXRhU291cmNlVXRpbHMuamF2YQ==) | `48.00% <25.00%> (-28.32%)` | :arrow_down: |
   | [...hinscheduler/api/service/impl/BaseServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9CYXNlU2VydmljZUltcGwuamF2YQ==) | `90.69% <50.00%> (-4.18%)` | :arrow_down: |
   | [...urce/clickhouse/ClickHouseDataSourceProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYXRhc291cmNlLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLWRhdGFzb3VyY2UtYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi9kYXRhc291cmNlL2FwaS9kYXRhc291cmNlL2NsaWNraG91c2UvQ2xpY2tIb3VzZURhdGFTb3VyY2VQcm9jZXNzb3IuamF2YQ==) | `5.76% <50.00%> (+3.76%)` | :arrow_up: |
   | [...rce/api/datasource/db2/Db2DataSourceProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYXRhc291cmNlLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLWRhdGFzb3VyY2UtYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi9kYXRhc291cmNlL2FwaS9kYXRhc291cmNlL2RiMi9EYjJEYXRhU291cmNlUHJvY2Vzc29yLmphdmE=) | `50.98% <50.00%> (-0.05%)` | :arrow_down: |
   | ... and [19 more](https://codecov.io/gh/apache/dolphinscheduler/pull/10276/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10276?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10276?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [4c1ef0a...7b0019f](https://codecov.io/gh/apache/dolphinscheduler/pull/10276?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng merged pull request #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Posted by GitBox <gi...@apache.org>.
caishunfeng merged PR #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] Tianqi-Dotes commented on a diff in pull request #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Posted by GitBox <gi...@apache.org>.
Tianqi-Dotes commented on code in PR #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276#discussion_r884401074


##########
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/utils/DataSourceUtils.java:
##########
@@ -17,43 +17,55 @@
 
 package org.apache.dolphinscheduler.plugin.datasource.api.utils;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.clickhouse.ClickHouseDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.db2.Db2DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.hive.HiveDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.mysql.MySQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.oracle.OracleDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.postgresql.PostgreSQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.presto.PrestoDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift.RedshiftDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.spark.SparkDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.sqlserver.SQLServerDataSourceProcessor;
 import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
 import org.apache.dolphinscheduler.spi.enums.DbType;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.sql.Connection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import static java.lang.String.format;
 
 public class DataSourceUtils {
 
-    private DataSourceUtils() {
+    public DataSourceUtils() {
     }
 
     private static final Logger logger = LoggerFactory.getLogger(DataSourceUtils.class);
 
-    private static final DataSourceProcessor mysqlProcessor = new MySQLDataSourceProcessor();
-    private static final DataSourceProcessor postgreSqlProcessor = new PostgreSQLDataSourceProcessor();
-    private static final DataSourceProcessor hiveProcessor = new HiveDataSourceProcessor();
-    private static final DataSourceProcessor sparkProcessor = new SparkDataSourceProcessor();
-    private static final DataSourceProcessor clickhouseProcessor = new ClickHouseDataSourceProcessor();
-    private static final DataSourceProcessor oracleProcessor = new OracleDataSourceProcessor();
-    private static final DataSourceProcessor sqlServerProcessor = new SQLServerDataSourceProcessor();
-    private static final DataSourceProcessor db2PROCESSOR = new Db2DataSourceProcessor();
-    private static final DataSourceProcessor prestoPROCESSOR = new PrestoDataSourceProcessor();
-    private static final DataSourceProcessor redshiftProcessor = new RedshiftDataSourceProcessor();
+    private static final Map<String, DataSourceProcessor> dataSourceProcessorMap = new ConcurrentHashMap<>();
+
+    public void installProcessor() {
+        final Set<String> names = new HashSet<>();
+
+        ServiceLoader.load(DataSourceProcessor.class).forEach(factory -> {
+            final String name = factory.getDbType().name();
+
+            logger.info("start register processor: " + name);

Review Comment:
   done



##########
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/utils/DataSourceUtils.java:
##########
@@ -17,43 +17,55 @@
 
 package org.apache.dolphinscheduler.plugin.datasource.api.utils;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.clickhouse.ClickHouseDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.db2.Db2DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.hive.HiveDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.mysql.MySQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.oracle.OracleDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.postgresql.PostgreSQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.presto.PrestoDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift.RedshiftDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.spark.SparkDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.sqlserver.SQLServerDataSourceProcessor;
 import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
 import org.apache.dolphinscheduler.spi.enums.DbType;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.sql.Connection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import static java.lang.String.format;
 
 public class DataSourceUtils {
 
-    private DataSourceUtils() {
+    public DataSourceUtils() {
     }
 
     private static final Logger logger = LoggerFactory.getLogger(DataSourceUtils.class);
 
-    private static final DataSourceProcessor mysqlProcessor = new MySQLDataSourceProcessor();
-    private static final DataSourceProcessor postgreSqlProcessor = new PostgreSQLDataSourceProcessor();
-    private static final DataSourceProcessor hiveProcessor = new HiveDataSourceProcessor();
-    private static final DataSourceProcessor sparkProcessor = new SparkDataSourceProcessor();
-    private static final DataSourceProcessor clickhouseProcessor = new ClickHouseDataSourceProcessor();
-    private static final DataSourceProcessor oracleProcessor = new OracleDataSourceProcessor();
-    private static final DataSourceProcessor sqlServerProcessor = new SQLServerDataSourceProcessor();
-    private static final DataSourceProcessor db2PROCESSOR = new Db2DataSourceProcessor();
-    private static final DataSourceProcessor prestoPROCESSOR = new PrestoDataSourceProcessor();
-    private static final DataSourceProcessor redshiftProcessor = new RedshiftDataSourceProcessor();
+    private static final Map<String, DataSourceProcessor> dataSourceProcessorMap = new ConcurrentHashMap<>();
+
+    public void installProcessor() {
+        final Set<String> names = new HashSet<>();
+
+        ServiceLoader.load(DataSourceProcessor.class).forEach(factory -> {
+            final String name = factory.getDbType().name();
+
+            logger.info("start register processor: " + name);
+            if (!names.add(name)) {
+                throw new IllegalStateException(format("Duplicate datasource plugins named '%s'", name));
+            }
+            loadDatasourceClient(factory);
+
+            logger.info("done register processor: " + name);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276#discussion_r884236520


##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/ApiApplicationServer.java:
##########
@@ -43,5 +43,7 @@ public static void main(String[] args) {
     public void run(ApplicationReadyEvent readyEvent) {
         // install task plugin
         taskPluginManager.installPlugin();
+        DataSourceUtils dataSourceUtils = new DataSourceUtils();

Review Comment:
   It‘s better to use sprint bean, and maybe `DataSourceManager` is better, WDYT?



##########
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/utils/DataSourceUtils.java:
##########
@@ -17,43 +17,55 @@
 
 package org.apache.dolphinscheduler.plugin.datasource.api.utils;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.clickhouse.ClickHouseDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.db2.Db2DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.hive.HiveDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.mysql.MySQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.oracle.OracleDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.postgresql.PostgreSQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.presto.PrestoDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift.RedshiftDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.spark.SparkDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.sqlserver.SQLServerDataSourceProcessor;
 import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
 import org.apache.dolphinscheduler.spi.enums.DbType;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.sql.Connection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import static java.lang.String.format;
 
 public class DataSourceUtils {
 
-    private DataSourceUtils() {
+    public DataSourceUtils() {
     }
 
     private static final Logger logger = LoggerFactory.getLogger(DataSourceUtils.class);
 
-    private static final DataSourceProcessor mysqlProcessor = new MySQLDataSourceProcessor();
-    private static final DataSourceProcessor postgreSqlProcessor = new PostgreSQLDataSourceProcessor();
-    private static final DataSourceProcessor hiveProcessor = new HiveDataSourceProcessor();
-    private static final DataSourceProcessor sparkProcessor = new SparkDataSourceProcessor();
-    private static final DataSourceProcessor clickhouseProcessor = new ClickHouseDataSourceProcessor();
-    private static final DataSourceProcessor oracleProcessor = new OracleDataSourceProcessor();
-    private static final DataSourceProcessor sqlServerProcessor = new SQLServerDataSourceProcessor();
-    private static final DataSourceProcessor db2PROCESSOR = new Db2DataSourceProcessor();
-    private static final DataSourceProcessor prestoPROCESSOR = new PrestoDataSourceProcessor();
-    private static final DataSourceProcessor redshiftProcessor = new RedshiftDataSourceProcessor();
+    private static final Map<String, DataSourceProcessor> dataSourceProcessorMap = new ConcurrentHashMap<>();
+
+    public void installProcessor() {
+        final Set<String> names = new HashSet<>();
+
+        ServiceLoader.load(DataSourceProcessor.class).forEach(factory -> {
+            final String name = factory.getDbType().name();
+
+            logger.info("start register processor: " + name);
+            if (!names.add(name)) {
+                throw new IllegalStateException(format("Duplicate datasource plugins named '%s'", name));
+            }
+            loadDatasourceClient(factory);
+
+            logger.info("done register processor: " + name);

Review Comment:
   ```suggestion
               logger.info("done register processor: {}", name);
   ```



##########
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/utils/DataSourceUtils.java:
##########
@@ -17,43 +17,55 @@
 
 package org.apache.dolphinscheduler.plugin.datasource.api.utils;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.clickhouse.ClickHouseDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.db2.Db2DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.hive.HiveDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.mysql.MySQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.oracle.OracleDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.postgresql.PostgreSQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.presto.PrestoDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift.RedshiftDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.spark.SparkDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.sqlserver.SQLServerDataSourceProcessor;
 import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
 import org.apache.dolphinscheduler.spi.enums.DbType;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.sql.Connection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import static java.lang.String.format;
 
 public class DataSourceUtils {
 
-    private DataSourceUtils() {
+    public DataSourceUtils() {
     }
 
     private static final Logger logger = LoggerFactory.getLogger(DataSourceUtils.class);
 
-    private static final DataSourceProcessor mysqlProcessor = new MySQLDataSourceProcessor();
-    private static final DataSourceProcessor postgreSqlProcessor = new PostgreSQLDataSourceProcessor();
-    private static final DataSourceProcessor hiveProcessor = new HiveDataSourceProcessor();
-    private static final DataSourceProcessor sparkProcessor = new SparkDataSourceProcessor();
-    private static final DataSourceProcessor clickhouseProcessor = new ClickHouseDataSourceProcessor();
-    private static final DataSourceProcessor oracleProcessor = new OracleDataSourceProcessor();
-    private static final DataSourceProcessor sqlServerProcessor = new SQLServerDataSourceProcessor();
-    private static final DataSourceProcessor db2PROCESSOR = new Db2DataSourceProcessor();
-    private static final DataSourceProcessor prestoPROCESSOR = new PrestoDataSourceProcessor();
-    private static final DataSourceProcessor redshiftProcessor = new RedshiftDataSourceProcessor();
+    private static final Map<String, DataSourceProcessor> dataSourceProcessorMap = new ConcurrentHashMap<>();
+
+    public void installProcessor() {
+        final Set<String> names = new HashSet<>();
+
+        ServiceLoader.load(DataSourceProcessor.class).forEach(factory -> {
+            final String name = factory.getDbType().name();
+
+            logger.info("start register processor: " + name);
+            if (!names.add(name)) {
+                throw new IllegalStateException(format("Duplicate datasource plugins named '%s'", name));
+            }
+            loadDatasourceClient(factory);
+
+            logger.info("done register processor: " + name);
+
+        });
+        int i = 1;

Review Comment:
   remove if not use.



##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataSourceController.java:
##########
@@ -92,7 +90,8 @@ public class DataSourceController extends BaseController {
     @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
     public Result createDataSource(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
                                    @ApiParam(name = "dataSourceParam", value = "DATA_SOURCE_PARAM", required = true)
-                                   @RequestBody BaseDataSourceParamDTO dataSourceParam) {
+                                   @RequestBody String jsonStr) {

Review Comment:
   It's better to provide json example.



##########
dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/utils/DataSourceUtils.java:
##########
@@ -17,43 +17,55 @@
 
 package org.apache.dolphinscheduler.plugin.datasource.api.utils;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
 import org.apache.dolphinscheduler.plugin.datasource.api.datasource.DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.clickhouse.ClickHouseDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.db2.Db2DataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.hive.HiveDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.mysql.MySQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.oracle.OracleDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.postgresql.PostgreSQLDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.presto.PrestoDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.redshift.RedshiftDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.spark.SparkDataSourceProcessor;
-import org.apache.dolphinscheduler.plugin.datasource.api.datasource.sqlserver.SQLServerDataSourceProcessor;
 import org.apache.dolphinscheduler.spi.datasource.ConnectionParam;
 import org.apache.dolphinscheduler.spi.enums.DbType;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.sql.Connection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import static java.lang.String.format;
 
 public class DataSourceUtils {
 
-    private DataSourceUtils() {
+    public DataSourceUtils() {
     }
 
     private static final Logger logger = LoggerFactory.getLogger(DataSourceUtils.class);
 
-    private static final DataSourceProcessor mysqlProcessor = new MySQLDataSourceProcessor();
-    private static final DataSourceProcessor postgreSqlProcessor = new PostgreSQLDataSourceProcessor();
-    private static final DataSourceProcessor hiveProcessor = new HiveDataSourceProcessor();
-    private static final DataSourceProcessor sparkProcessor = new SparkDataSourceProcessor();
-    private static final DataSourceProcessor clickhouseProcessor = new ClickHouseDataSourceProcessor();
-    private static final DataSourceProcessor oracleProcessor = new OracleDataSourceProcessor();
-    private static final DataSourceProcessor sqlServerProcessor = new SQLServerDataSourceProcessor();
-    private static final DataSourceProcessor db2PROCESSOR = new Db2DataSourceProcessor();
-    private static final DataSourceProcessor prestoPROCESSOR = new PrestoDataSourceProcessor();
-    private static final DataSourceProcessor redshiftProcessor = new RedshiftDataSourceProcessor();
+    private static final Map<String, DataSourceProcessor> dataSourceProcessorMap = new ConcurrentHashMap<>();
+
+    public void installProcessor() {
+        final Set<String> names = new HashSet<>();
+
+        ServiceLoader.load(DataSourceProcessor.class).forEach(factory -> {
+            final String name = factory.getDbType().name();
+
+            logger.info("start register processor: " + name);

Review Comment:
   ```suggestion
               logger.info("start register processor: {}", name);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] Tianqi-Dotes commented on a diff in pull request #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Posted by GitBox <gi...@apache.org>.
Tianqi-Dotes commented on code in PR #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276#discussion_r884447134


##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataSourceController.java:
##########
@@ -92,7 +90,8 @@ public class DataSourceController extends BaseController {
     @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
     public Result createDataSource(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
                                    @ApiParam(name = "dataSourceParam", value = "DATA_SOURCE_PARAM", required = true)
-                                   @RequestBody BaseDataSourceParamDTO dataSourceParam) {
+                                   @RequestBody String jsonStr) {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] Tianqi-Dotes commented on a diff in pull request #10276: [Feature][Datasource]replace DataSourceController API with string JSON and cast to DTO

Posted by GitBox <gi...@apache.org>.
Tianqi-Dotes commented on code in PR #10276:
URL: https://github.com/apache/dolphinscheduler/pull/10276#discussion_r884400689


##########
dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/ApiApplicationServer.java:
##########
@@ -43,5 +43,7 @@ public static void main(String[] args) {
     public void run(ApplicationReadyEvent readyEvent) {
         // install task plugin
         taskPluginManager.installPlugin();
+        DataSourceUtils dataSourceUtils = new DataSourceUtils();

Review Comment:
   done change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org