You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "xinxingi (via GitHub)" <gi...@apache.org> on 2024/04/15 11:58:06 UTC
[PR] Fix seatunnel 15706 improvement [dolphinscheduler]
xinxingi opened a new pull request, #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852
<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! -->
## Purpose of the pull request
#15706
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
(or)
If your pull request contain incompatible change, you should also add it to `docs/docs/en/guide/upgrede/incompatible.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "rickchengx (via GitHub)" <gi...@apache.org>.
rickchengx commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1566730765
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/java/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTask.java:
##########
@@ -184,8 +188,17 @@ private String buildCustomConfigContent() {
}
private String buildConfigFilePath() {
- return String.format("%s/seatunnel_%s.conf", taskExecutionContext.getExecutePath(),
- taskExecutionContext.getTaskAppId());
+ return String.format("%s/seatunnel_%s.%s", taskExecutionContext.getExecutePath(),
+ taskExecutionContext.getTaskAppId(), formatDetector());
+ }
+
+ private String formatDetector() {
+ try {
+ new Gson().fromJson(seatunnelParameters.getRawScript(), JsonElement.class);
+ return "json";
+ } catch (JsonSyntaxException e) {
Review Comment:
If it is not `json`, are there any other format besides `conf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1567002243
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/test/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTaskTest.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.dolphinscheduler.plugin.task.seatunnel;
+
Review Comment:
Delete blank line
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/test/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTaskTest.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.dolphinscheduler.plugin.task.seatunnel;
+
+
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.junit.Test;
+import org.junit.jupiter.api.Assertions;
+
+public class SeatunnelTaskTest {
+ private static final String EXECUTE_PATH = "/home";
+ private static final String TASK_APPID = "9527";
+
+ @Test
+ public void formatDetector() throws Exception{
+ SeatunnelParameters seatunnelParameters = new SeatunnelParameters();
+ seatunnelParameters.setRawScript(RAW_SCRIPT);
+
+ TaskExecutionContext taskExecutionContext = new TaskExecutionContext();
+ taskExecutionContext.setExecutePath(EXECUTE_PATH);
+ taskExecutionContext.setTaskAppId(TASK_APPID);
+ taskExecutionContext.setTaskParams(JSONUtils.toJsonString(seatunnelParameters));
+
+ SeatunnelTask seatunnelTask = new SeatunnelTask(taskExecutionContext);
+ seatunnelTask.setSeatunnelParameters(seatunnelParameters);
+ Assertions.assertEquals("/home/seatunnel_9527.conf", seatunnelTask.buildCustomConfigCommand());
+
+ seatunnelParameters.setRawScript(RAW_SCRIPT_2);
+ seatunnelTask.setSeatunnelParameters(seatunnelParameters);
+ Assertions.assertEquals("/home/seatunnel_9527.json", seatunnelTask.buildCustomConfigCommand());
+
+
+
+
+
+ }
+
+ private static final String RAW_SCRIPT = "env {\n" +
+ " execution.parallelism = 2\n" +
+ " job.mode = \"BATCH\"\n" +
+ " checkpoint.interval = 10000\n" +
+ "}\n" +
+ "\n" +
+ "source {\n" +
+ " FakeSource {\n" +
+ " parallelism = 2\n" +
+ " result_table_name = \"fake\"\n" +
+ " row.num = 16\n" +
+ " schema = {\n" +
+ " fields {\n" +
+ " name = \"string\"\n" +
+ " age = \"int\"\n" +
+ " }\n" +
+ " }\n" +
+ " }\n" +
+ "}\n" +
+ "\n" +
+ "sink {\n" +
+ " Console {\n" +
+ " }\n" +
+ "}";
+
Review Comment:
Ditto
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/test/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTaskTest.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.dolphinscheduler.plugin.task.seatunnel;
+
+
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.junit.Test;
+import org.junit.jupiter.api.Assertions;
+
+public class SeatunnelTaskTest {
+ private static final String EXECUTE_PATH = "/home";
+ private static final String TASK_APPID = "9527";
+
+ @Test
+ public void formatDetector() throws Exception{
+ SeatunnelParameters seatunnelParameters = new SeatunnelParameters();
+ seatunnelParameters.setRawScript(RAW_SCRIPT);
+
+ TaskExecutionContext taskExecutionContext = new TaskExecutionContext();
+ taskExecutionContext.setExecutePath(EXECUTE_PATH);
+ taskExecutionContext.setTaskAppId(TASK_APPID);
+ taskExecutionContext.setTaskParams(JSONUtils.toJsonString(seatunnelParameters));
+
+ SeatunnelTask seatunnelTask = new SeatunnelTask(taskExecutionContext);
+ seatunnelTask.setSeatunnelParameters(seatunnelParameters);
+ Assertions.assertEquals("/home/seatunnel_9527.conf", seatunnelTask.buildCustomConfigCommand());
+
+ seatunnelParameters.setRawScript(RAW_SCRIPT_2);
+ seatunnelTask.setSeatunnelParameters(seatunnelParameters);
+ Assertions.assertEquals("/home/seatunnel_9527.json", seatunnelTask.buildCustomConfigCommand());
+
+
+
+
+
Review Comment:
Ditto
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/test/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTaskTest.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.dolphinscheduler.plugin.task.seatunnel;
+
+
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.junit.Test;
+import org.junit.jupiter.api.Assertions;
+
+public class SeatunnelTaskTest {
+ private static final String EXECUTE_PATH = "/home";
+ private static final String TASK_APPID = "9527";
+
+ @Test
+ public void formatDetector() throws Exception{
+ SeatunnelParameters seatunnelParameters = new SeatunnelParameters();
+ seatunnelParameters.setRawScript(RAW_SCRIPT);
+
+ TaskExecutionContext taskExecutionContext = new TaskExecutionContext();
+ taskExecutionContext.setExecutePath(EXECUTE_PATH);
+ taskExecutionContext.setTaskAppId(TASK_APPID);
+ taskExecutionContext.setTaskParams(JSONUtils.toJsonString(seatunnelParameters));
+
+ SeatunnelTask seatunnelTask = new SeatunnelTask(taskExecutionContext);
+ seatunnelTask.setSeatunnelParameters(seatunnelParameters);
+ Assertions.assertEquals("/home/seatunnel_9527.conf", seatunnelTask.buildCustomConfigCommand());
+
+ seatunnelParameters.setRawScript(RAW_SCRIPT_2);
+ seatunnelTask.setSeatunnelParameters(seatunnelParameters);
+ Assertions.assertEquals("/home/seatunnel_9527.json", seatunnelTask.buildCustomConfigCommand());
+
+
+
+
+
+ }
+
+ private static final String RAW_SCRIPT = "env {\n" +
+ " execution.parallelism = 2\n" +
+ " job.mode = \"BATCH\"\n" +
+ " checkpoint.interval = 10000\n" +
+ "}\n" +
+ "\n" +
+ "source {\n" +
+ " FakeSource {\n" +
+ " parallelism = 2\n" +
+ " result_table_name = \"fake\"\n" +
+ " row.num = 16\n" +
+ " schema = {\n" +
+ " fields {\n" +
+ " name = \"string\"\n" +
+ " age = \"int\"\n" +
+ " }\n" +
+ " }\n" +
+ " }\n" +
+ "}\n" +
+ "\n" +
+ "sink {\n" +
+ " Console {\n" +
+ " }\n" +
+ "}";
+
+
+ private static final String RAW_SCRIPT_2 = "{\n" +
+ " \"env\": {\n" +
+ " \"execution.parallelism\": 2,\n" +
+ " \"job.mode\": \"BATCH\",\n" +
+ " \"checkpoint.interval\": 10000\n" +
+ " },\n" +
+ " \"source\": {\n" +
+ " \"FakeSource\": {\n" +
+ " \"parallelism\": 2,\n" +
+ " \"result_table_name\": \"fake\",\n" +
+ " \"row.num\": 16,\n" +
+ " \"schema\": {\n" +
+ " \"fields\": {\n" +
+ " \"name\": \"string\",\n" +
+ " \"age\": \"int\"\n" +
+ " }\n" +
+ " }\n" +
+ " }\n" +
+ " },\n" +
+ " \"sink\": {\n" +
+ " \"Console\": {}\n" +
+ " }\n" +
+ "}";
+
Review Comment:
Ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "xinxingi (via GitHub)" <gi...@apache.org>.
xinxingi commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1566954294
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/java/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTask.java:
##########
@@ -184,8 +188,17 @@ private String buildCustomConfigContent() {
}
private String buildConfigFilePath() {
- return String.format("%s/seatunnel_%s.conf", taskExecutionContext.getExecutePath(),
- taskExecutionContext.getTaskAppId());
+ return String.format("%s/seatunnel_%s.%s", taskExecutionContext.getExecutePath(),
+ taskExecutionContext.getTaskAppId(), formatDetector());
+ }
+
+ private String formatDetector() {
+ try {
+ new Gson().fromJson(seatunnelParameters.getRawScript(), JsonElement.class);
+ return "json";
+ } catch (JsonSyntaxException e) {
Review Comment:
No, currently only these two formats are available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2060271775
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "caishunfeng (via GitHub)" <gi...@apache.org>.
caishunfeng commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1567024612
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/JSONUtils.java:
##########
@@ -196,7 +196,10 @@ public static <T> List<T> toList(String json, Class<T> clazz) {
* @return true if valid
*/
public static boolean checkJsonValid(String json) {
+ return checkJsonValid(json, true);
+ }
+ public static boolean checkJsonValid(String json, Boolean logFlag) {
Review Comment:
Why not log default if parse wrong? I think it's not need to add this log flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2058727734
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2058726539
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1566885625
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/java/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTask.java:
##########
@@ -184,8 +188,17 @@ private String buildCustomConfigContent() {
}
private String buildConfigFilePath() {
- return String.format("%s/seatunnel_%s.conf", taskExecutionContext.getExecutePath(),
- taskExecutionContext.getTaskAppId());
+ return String.format("%s/seatunnel_%s.%s", taskExecutionContext.getExecutePath(),
+ taskExecutionContext.getTaskAppId(), formatDetector());
+ }
+
+ private String formatDetector() {
+ try {
+ new Gson().fromJson(seatunnelParameters.getRawScript(), JsonElement.class);
Review Comment:
It's better to use `JSONUtils` instead of `Gson`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2056732359
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1566994653
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/java/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTask.java:
##########
@@ -184,8 +184,12 @@ private String buildCustomConfigContent() {
}
private String buildConfigFilePath() {
- return String.format("%s/seatunnel_%s.conf", taskExecutionContext.getExecutePath(),
- taskExecutionContext.getTaskAppId());
+ return String.format("%s/seatunnel_%s.%s", taskExecutionContext.getExecutePath(),
+ taskExecutionContext.getTaskAppId(), formatDetector());
+ }
+
+ private String formatDetector() {
+ return JSONUtils.checkJsonValid(seatunnelParameters.getRawScript(), false) ? "json" : "conf";
Review Comment:
Could you out `json` and `conf` to constants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "xinxingi (via GitHub)" <gi...@apache.org>.
xinxingi commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1566994405
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-seatunnel/src/main/java/org/apache/dolphinscheduler/plugin/task/seatunnel/SeatunnelTask.java:
##########
@@ -184,8 +188,17 @@ private String buildCustomConfigContent() {
}
private String buildConfigFilePath() {
- return String.format("%s/seatunnel_%s.conf", taskExecutionContext.getExecutePath(),
- taskExecutionContext.getTaskAppId());
+ return String.format("%s/seatunnel_%s.%s", taskExecutionContext.getExecutePath(),
+ taskExecutionContext.getTaskAppId(), formatDetector());
+ }
+
+ private String formatDetector() {
+ try {
+ new Gson().fromJson(seatunnelParameters.getRawScript(), JsonElement.class);
+ return "json";
+ } catch (JsonSyntaxException e) {
Review Comment:
https://seatunnel.apache.org/docs/2.3.4/concept/config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1567034242
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/JSONUtils.java:
##########
@@ -196,7 +196,10 @@ public static <T> List<T> toList(String json, Class<T> clazz) {
* @return true if valid
*/
public static boolean checkJsonValid(String json) {
+ return checkJsonValid(json, true);
+ }
+ public static boolean checkJsonValid(String json, Boolean logFlag) {
Review Comment:
Looks like use this method to check a string is a json or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2062883642
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli merged PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2058632698
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2062883872
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2056733290
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2060272170
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2058629889
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_accepted_issues&view=list)
Measures
 [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15852&resolved=false&inNewCodePeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15852&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2056721061
## [Codecov](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15852?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 39.52%. Comparing base [(`f99d3f1`)](https://app.codecov.io/gh/apache/dolphinscheduler/commit/f99d3f1ed37f1f36a51cfa2063d298f1bed5de49?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`933baa7`)](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15852?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> :exclamation: Current head 933baa7 differs from pull request most recent head 267cfb8. Consider uploading reports for the commit 267cfb8 to get more accurate results
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## dev #15852 +/- ##
============================================
- Coverage 39.59% 39.52% -0.07%
+ Complexity 5017 4996 -21
============================================
Files 1347 1347
Lines 45646 45611 -35
Branches 4892 4869 -23
============================================
- Hits 18073 18028 -45
- Misses 25649 25659 +10
Partials 1924 1924
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15852?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "xinxingi (via GitHub)" <gi...@apache.org>.
xinxingi commented on code in PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#discussion_r1567067909
##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/JSONUtils.java:
##########
@@ -196,7 +196,10 @@ public static <T> List<T> toList(String json, Class<T> clazz) {
* @return true if valid
*/
public static boolean checkJsonValid(String json) {
+ return checkJsonValid(json, true);
+ }
+ public static boolean checkJsonValid(String json, Boolean logFlag) {
Review Comment:
@caishunfeng The original method has not changed, only enhancements have been made to the original method. For the adaptation of the generated file extension, no log output is required. Adding this log parsing is of little significance and may cause some confusion to users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix-15706] Seatunnel improvement [dolphinscheduler]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15852:
URL: https://github.com/apache/dolphinscheduler/pull/15852#issuecomment-2059075428
Please retry analysis of this Pull-Request directly on SonarCloud
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org