You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/05/23 07:52:54 UTC
[GitHub] [dolphinscheduler] vagetablechicken opened a new pull request, #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
vagetablechicken opened a new pull request, #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198
## Purpose of the pull request
Add the task OpenMLDB
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
This change added tests and can be verified as follows:
- *Execute command test in module*
- *Manually verified the change by testing locally.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1138267777
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [49.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r879995512
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/main/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTask.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.google.common.base.Preconditions;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+/**
+ * openmldb task
+ */
+public class OpenmldbTask extends AbstractTaskExecutor {
Review Comment:
But still have many differences. If we extend `PythonTask`, it'll only reduce ~ 50 lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1138084311
ping @songjianet and @labbomb to take a looking the frontend code if you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r880046049
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/main/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTask.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.google.common.base.Preconditions;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+/**
+ * openmldb task
+ */
+public class OpenmldbTask extends AbstractTaskExecutor {
+
+ /**
+ * openmldb parameters
+ */
+ private OpenmldbParameters openmldbParameters;
+
+ /**
+ * shell command executor
+ */
+ private final ShellCommandExecutor shellCommandExecutor;
+
+ private final TaskExecutionContext taskRequest;
+
+ private static final String PYTHON_HOME = "PYTHON_HOME";
+
+ /**
+ * python process(openmldb only supports version 3 by default)
+ */
+ private static final String OPENMLDB_PYTHON = "python3";
+ private static final Pattern PYTHON_PATH_PATTERN = Pattern.compile("/bin/python[\\d.]*$");
+
+ /**
+ * constructor
+ *
+ * @param taskRequest taskRequest
+ */
+ public OpenmldbTask(TaskExecutionContext taskRequest) {
+ super(taskRequest);
+ this.taskRequest = taskRequest;
+
+ this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle,
+ taskRequest,
+ logger);
+ }
+
+ @Override
+ public void init() {
+ logger.info("openmldb task params {}", taskRequest.getTaskParams());
+
+ openmldbParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), OpenmldbParameters.class);
+
+ assert openmldbParameters != null;
+ if (!openmldbParameters.checkParameters()) {
+ throw new TaskException("openmldb task params is not valid");
+ }
+ }
+
+ @Override
+ public void handle() throws Exception {
+ try {
+ // generate the content of this python script
+ String pythonScriptContent = buildPythonScriptContent();
+ // generate the file path of this python script
+ String pythonScriptFile = buildPythonCommandFilePath();
+
+ // create this file
+ createPythonCommandFileIfNotExists(pythonScriptContent, pythonScriptFile);
+ String command = buildPythonExecuteCommand(pythonScriptFile);
+
+ TaskResponse taskResponse = shellCommandExecutor.run(command);
+ setExitStatusCode(taskResponse.getExitStatusCode());
+ setAppIds(taskResponse.getAppIds());
+ setProcessId(taskResponse.getProcessId());
+ setVarPool(shellCommandExecutor.getVarPool());
+ } catch (Exception e) {
+ logger.error("openmldb task failure", e);
+ setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);
+ throw new TaskException("run openmldb task error", e);
+ }
+ }
+
+ @Override
+ public void cancelApplication(boolean cancelApplication) throws Exception {
+ // cancel process
+ shellCommandExecutor.cancelApplication();
+ }
+
+ @Override
+ public AbstractParameters getParameters() {
+ return openmldbParameters;
+ }
+
+ /**
+ * create python command file if not exists
+ *
+ * @param pythonScript exec python script
+ * @param pythonScriptFile python script file
+ * @throws IOException io exception
+ */
+ protected void createPythonCommandFileIfNotExists(String pythonScript, String pythonScriptFile) throws IOException {
+ logger.info("tenantCode :{}, task dir:{}", taskRequest.getTenantCode(), taskRequest.getExecutePath());
+
+ if (!Files.exists(Paths.get(pythonScriptFile))) {
+ logger.info("generate python script file:{}", pythonScriptFile);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("#-*- encoding=utf8 -*-\n");
+
+ sb.append("\n\n");
+ sb.append(pythonScript);
+ logger.info(sb.toString());
+
+ // write data to file
+ FileUtils.writeStringToFile(new File(pythonScriptFile),
+ sb.toString(),
+ StandardCharsets.UTF_8);
+ }
+ }
+
+ /**
+ * build python command file path
+ *
+ * @return python command file path
+ */
+ protected String buildPythonCommandFilePath() {
+ return String.format("%s/openmldb_%s.py", taskRequest.getExecutePath(), taskRequest.getTaskAppId());
+ }
+
+ /**
+ * build python script content
+ *
+ * @return raw python script
+ */
+ private String buildPythonScriptContent() {
+ // sqls doesn't need \n, use ; to split
+ String rawSqlScript = openmldbParameters.getSql().replaceAll("[\\r]?\\n", " ");
Review Comment:
Hi, I don't think you can simply replace all line breaks with spaces, if there are comments in the sql scripts, you will totally change the logic of it, for example, if the original sql is
```sql
select * from users
-- some comment
inner join order on users.order_id = order.id;
```
will become
```sql
select * from users -- some comment inner join order on users.order_id = order.id;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r883356739
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/test/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTaskTest.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.powermock.reflect.Whitebox;
+
+public class OpenmldbTaskTest {
+ static class MockOpenmldbTask extends OpenmldbTask {
+ /**
+ * constructor
+ *
+ * @param taskRequest taskRequest
+ */
+ public MockOpenmldbTask(TaskExecutionContext taskRequest) {
+ super(taskRequest);
+ }
+
+ @Override
+ protected Map<String, Property> mergeParamsWithContext(AbstractParameters parameters) {
+ return new HashMap<>();
+ }
+ }
+
+ private OpenmldbTask createOpenmldbTask() {
+ return new MockOpenmldbTask(null);
+ }
+
+ @Test
+ public void buildPythonExecuteCommand() throws Exception {
+ OpenmldbTask openmldbTask = createOpenmldbTask();
+ String pythonFile = "test.py";
+ String result1 = openmldbTask.buildPythonExecuteCommand(pythonFile);
+ Assert.assertEquals("python3 test.py", result1);
+ }
+
+ @Test
+ public void buildSQLWithComment() throws Exception {
+ OpenmldbTask openmldbTask = createOpenmldbTask();
+ OpenmldbParameters openmldbParameters = new OpenmldbParameters();
+ openmldbParameters.setExecuteMode("offline");
+ String rawSQLScript = "select * from users\r\n"
+ + "-- some comment\n"
+ + "inner join order on users.order_id = order.id; \n\n;"
+ + "select * from users;";
Review Comment:
Got conflicts, so I'v merged dev branch. Sorry to dismiss your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r880110395
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/main/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTask.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.google.common.base.Preconditions;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+/**
+ * openmldb task
+ */
+public class OpenmldbTask extends AbstractTaskExecutor {
+
+ /**
+ * openmldb parameters
+ */
+ private OpenmldbParameters openmldbParameters;
+
+ /**
+ * shell command executor
+ */
+ private final ShellCommandExecutor shellCommandExecutor;
+
+ private final TaskExecutionContext taskRequest;
+
+ private static final String PYTHON_HOME = "PYTHON_HOME";
+
+ /**
+ * python process(openmldb only supports version 3 by default)
+ */
+ private static final String OPENMLDB_PYTHON = "python3";
+ private static final Pattern PYTHON_PATH_PATTERN = Pattern.compile("/bin/python[\\d.]*$");
+
+ /**
+ * constructor
+ *
+ * @param taskRequest taskRequest
+ */
+ public OpenmldbTask(TaskExecutionContext taskRequest) {
+ super(taskRequest);
+ this.taskRequest = taskRequest;
+
+ this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle,
+ taskRequest,
+ logger);
+ }
+
+ @Override
+ public void init() {
+ logger.info("openmldb task params {}", taskRequest.getTaskParams());
+
+ openmldbParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), OpenmldbParameters.class);
+
+ assert openmldbParameters != null;
+ if (!openmldbParameters.checkParameters()) {
+ throw new TaskException("openmldb task params is not valid");
+ }
+ }
+
+ @Override
+ public void handle() throws Exception {
+ try {
+ // generate the content of this python script
+ String pythonScriptContent = buildPythonScriptContent();
+ // generate the file path of this python script
+ String pythonScriptFile = buildPythonCommandFilePath();
+
+ // create this file
+ createPythonCommandFileIfNotExists(pythonScriptContent, pythonScriptFile);
+ String command = buildPythonExecuteCommand(pythonScriptFile);
+
+ TaskResponse taskResponse = shellCommandExecutor.run(command);
+ setExitStatusCode(taskResponse.getExitStatusCode());
+ setAppIds(taskResponse.getAppIds());
+ setProcessId(taskResponse.getProcessId());
+ setVarPool(shellCommandExecutor.getVarPool());
+ } catch (Exception e) {
+ logger.error("openmldb task failure", e);
+ setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);
+ throw new TaskException("run openmldb task error", e);
+ }
+ }
+
+ @Override
+ public void cancelApplication(boolean cancelApplication) throws Exception {
+ // cancel process
+ shellCommandExecutor.cancelApplication();
+ }
+
+ @Override
+ public AbstractParameters getParameters() {
+ return openmldbParameters;
+ }
+
+ /**
+ * create python command file if not exists
+ *
+ * @param pythonScript exec python script
+ * @param pythonScriptFile python script file
+ * @throws IOException io exception
+ */
+ protected void createPythonCommandFileIfNotExists(String pythonScript, String pythonScriptFile) throws IOException {
+ logger.info("tenantCode :{}, task dir:{}", taskRequest.getTenantCode(), taskRequest.getExecutePath());
+
+ if (!Files.exists(Paths.get(pythonScriptFile))) {
+ logger.info("generate python script file:{}", pythonScriptFile);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("#-*- encoding=utf8 -*-\n");
+
+ sb.append("\n\n");
+ sb.append(pythonScript);
+ logger.info(sb.toString());
+
+ // write data to file
+ FileUtils.writeStringToFile(new File(pythonScriptFile),
+ sb.toString(),
+ StandardCharsets.UTF_8);
+ }
+ }
+
+ /**
+ * build python command file path
+ *
+ * @return python command file path
+ */
+ protected String buildPythonCommandFilePath() {
+ return String.format("%s/openmldb_%s.py", taskRequest.getExecutePath(), taskRequest.getTaskAppId());
+ }
+
+ /**
+ * build python script content
+ *
+ * @return raw python script
+ */
+ private String buildPythonScriptContent() {
+ // sqls doesn't need \n, use ; to split
+ String rawSqlScript = openmldbParameters.getSql().replaceAll("[\\r]?\\n", " ");
Review Comment:
Yes, I should replace \r\n to '/\n' in string, not the space. I'll fix it and add ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r880110395
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/main/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTask.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.google.common.base.Preconditions;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+/**
+ * openmldb task
+ */
+public class OpenmldbTask extends AbstractTaskExecutor {
+
+ /**
+ * openmldb parameters
+ */
+ private OpenmldbParameters openmldbParameters;
+
+ /**
+ * shell command executor
+ */
+ private final ShellCommandExecutor shellCommandExecutor;
+
+ private final TaskExecutionContext taskRequest;
+
+ private static final String PYTHON_HOME = "PYTHON_HOME";
+
+ /**
+ * python process(openmldb only supports version 3 by default)
+ */
+ private static final String OPENMLDB_PYTHON = "python3";
+ private static final Pattern PYTHON_PATH_PATTERN = Pattern.compile("/bin/python[\\d.]*$");
+
+ /**
+ * constructor
+ *
+ * @param taskRequest taskRequest
+ */
+ public OpenmldbTask(TaskExecutionContext taskRequest) {
+ super(taskRequest);
+ this.taskRequest = taskRequest;
+
+ this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle,
+ taskRequest,
+ logger);
+ }
+
+ @Override
+ public void init() {
+ logger.info("openmldb task params {}", taskRequest.getTaskParams());
+
+ openmldbParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), OpenmldbParameters.class);
+
+ assert openmldbParameters != null;
+ if (!openmldbParameters.checkParameters()) {
+ throw new TaskException("openmldb task params is not valid");
+ }
+ }
+
+ @Override
+ public void handle() throws Exception {
+ try {
+ // generate the content of this python script
+ String pythonScriptContent = buildPythonScriptContent();
+ // generate the file path of this python script
+ String pythonScriptFile = buildPythonCommandFilePath();
+
+ // create this file
+ createPythonCommandFileIfNotExists(pythonScriptContent, pythonScriptFile);
+ String command = buildPythonExecuteCommand(pythonScriptFile);
+
+ TaskResponse taskResponse = shellCommandExecutor.run(command);
+ setExitStatusCode(taskResponse.getExitStatusCode());
+ setAppIds(taskResponse.getAppIds());
+ setProcessId(taskResponse.getProcessId());
+ setVarPool(shellCommandExecutor.getVarPool());
+ } catch (Exception e) {
+ logger.error("openmldb task failure", e);
+ setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);
+ throw new TaskException("run openmldb task error", e);
+ }
+ }
+
+ @Override
+ public void cancelApplication(boolean cancelApplication) throws Exception {
+ // cancel process
+ shellCommandExecutor.cancelApplication();
+ }
+
+ @Override
+ public AbstractParameters getParameters() {
+ return openmldbParameters;
+ }
+
+ /**
+ * create python command file if not exists
+ *
+ * @param pythonScript exec python script
+ * @param pythonScriptFile python script file
+ * @throws IOException io exception
+ */
+ protected void createPythonCommandFileIfNotExists(String pythonScript, String pythonScriptFile) throws IOException {
+ logger.info("tenantCode :{}, task dir:{}", taskRequest.getTenantCode(), taskRequest.getExecutePath());
+
+ if (!Files.exists(Paths.get(pythonScriptFile))) {
+ logger.info("generate python script file:{}", pythonScriptFile);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("#-*- encoding=utf8 -*-\n");
+
+ sb.append("\n\n");
+ sb.append(pythonScript);
+ logger.info(sb.toString());
+
+ // write data to file
+ FileUtils.writeStringToFile(new File(pythonScriptFile),
+ sb.toString(),
+ StandardCharsets.UTF_8);
+ }
+ }
+
+ /**
+ * build python command file path
+ *
+ * @return python command file path
+ */
+ protected String buildPythonCommandFilePath() {
+ return String.format("%s/openmldb_%s.py", taskRequest.getExecutePath(), taskRequest.getTaskAppId());
+ }
+
+ /**
+ * build python script content
+ *
+ * @return raw python script
+ */
+ private String buildPythonScriptContent() {
+ // sqls doesn't need \n, use ; to split
+ String rawSqlScript = openmldbParameters.getSql().replaceAll("[\\r]?\\n", " ");
Review Comment:
Yes, I should replace \\r\\n to '\\n' in string, not the space. Comments and '\n' can be solved by sql engine. I'll fix it and add ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r880114749
##########
docs/docs/zh/guide/task/openmldb.md:
##########
@@ -0,0 +1,71 @@
+# MLflow节点
+
+## 综述
+
+[OpenMLDB](https://openmldb.ai/) 是一个优秀的开源机器学习数据库，提供生产级数据及特征开发全栈解决方案。
+
+OpenMLDB任务组件可以连接OpenMLDB集群执行任务。
+
+## 创建任务
+
+- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面；
+- 拖动工具栏的 <img src="/img/tasks/icons/openmldb.png" width="15"/> 任务节点到画板中。
+
+
+## 任务样例
+
+首先介绍一些DS通用参数：
+
+- **节点名称** ：设置任务的名称。一个工作流定义中的节点名称是唯一的。
+- **运行标志** ：标识这个节点是否能正常调度,如果不需要执行，可以打开禁止执行开关。
+- **描述** ：描述该节点的功能。
+- **任务优先级** ：worker 线程数不足时，根据优先级从高到低依次执行，优先级一样时根据先进先出原则执行。
+- **Worker 分组** ：任务分配给 worker 组的机器执行，选择 Default，会随机选择一台 worker 机执行。
+- **环境名称** ：配置运行脚本的环境。
+- **失败重试次数** ：任务失败重新提交的次数。
+- **失败重试间隔** ：任务失败重新提交任务的时间间隔，以分钟为单位。
+- **延迟执行时间** ：任务延迟执行的时间，以分钟为单位。
+- **超时告警** ：勾选超时告警、超时失败，当任务超过"超时时长"后，会发送告警邮件并且任务执行失败。
+- **前置任务** ：选择当前任务的前置任务，会将被选择的前置任务设置为当前任务的上游。
+
+### OpenMLDB 参数
+
+#### BasicAlgorithm
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1139379936
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [49.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1134503156
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [10.3% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [7.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r879287076
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/pom.xml:
##########
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>dolphinscheduler-task-plugin</artifactId>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <version>dev-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>dolphinscheduler-task-openmldb</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-spi</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-task-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ <version>1.18.20</version>
+ </dependency>
Review Comment:
> Actually I can remove this dependency in this pr. Maybe we could discuss it later?
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1134363868
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [8.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [8.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1141606305
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [49.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1140645757
> ping @songjianet and @labbomb to take a looking the frontend code if you have time
ping @songjianet @labbomb again, PTAL if you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r879192295
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/pom.xml:
##########
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>dolphinscheduler-task-plugin</artifactId>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <version>dev-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>dolphinscheduler-task-openmldb</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-spi</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-task-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ <version>1.18.20</version>
+ </dependency>
Review Comment:
Actually I can remove this dependency in this pr. Maybe we could discuss it later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie merged pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie merged PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1134333478
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [8.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [8.3% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r882263553
##########
docs/docs/zh/guide/task/openmldb.md:
##########
@@ -0,0 +1,69 @@
+# OpenMLDB 节点
+
+## 综述
+
+[OpenMLDB](https://openmldb.ai/) 是一个优秀的开源机器学习数据库，提供生产级数据及特征开发全栈解决方案。
+
+OpenMLDB任务组件可以连接OpenMLDB集群执行任务。
+
+## 创建任务
+
+- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面；
+- 拖动工具栏的 <img src="/img/tasks/icons/openmldb.png" width="15"/> 任务节点到画板中。
+
+
Review Comment:
```suggestion
```
##########
docs/docs/en/guide/task/openmldb.md:
##########
@@ -0,0 +1,76 @@
+# OpenMLDB Node
+
+## Overview
+
+[OpenMLDB](https://openmldb.ai/) is an excellent open source machine learning database, providing a full-stack
+FeatureOps solution for production.
+
+OpenMLDB task plugin used to execute tasks on OpenMLDB cluster.
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the
+ DAG editing page.
+- Drag from the toolbar <img src="/img/tasks/icons/openmldb.png" width="15"/> task node to canvas.
+
+## Task Example
+
+First, introduce some general parameters of DolphinScheduler
+
+- **Node name**: The node name in a workflow definition is unique.
+- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select
+ the `prohibition execution`.
+- **Descriptive information**: Describe the function of the node.
+- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high
+ to low, and tasks with the same priority will execute in a first-in first-out order.
+- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected,
+ randomly select a worker machine for execution.
+- **Environment Name**: Configure the environment name in which run the script.
+- **Times of failed retry attempts**: The number of times the task failed to resubmit.
+- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task.
+- **Delayed execution time**: The time (unit minute) that a task delays in execution.
+- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm
+ email will send and the task execution will fail.
+- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as
+ upstream of the current task.
+
+### OpenMLDB Parameters
+
+**Task Parameter**
+
+- **zookeeper** ：OpenMLDB cluster zookeeper address, e.g. 127.0.0.1:2181.
+- **zookeeper path** : OpenMLDB cluster zookeeper path, e.g. /openmldb.
+- **Execute Mode** ：determine the init mode, offline or online. You can switch it in sql statement.
+- **SQL statement** ：SQL statement.
+- Custom parameters: It is the user-defined parameters of Python, which will replace the content with \${variable} in the script.
+
+Here are some examples:
+
+#### Load data
+
+
+
+We use `LOAD DATA` to load data into OpenMLDB cluster. We select `offline` here, so it will load to offline storage.
+
+#### Feature extraction
+
+
+
+We use `SELECT INTO` to do feature extraction. We select `offline` here, so it will run sql on offline engine.
+
+## Environment to prepare
+
+### Start the OpenMLDB cluster
+
+You should create an OpenMLDB cluster first. If in production env, please check [deploy OpenMLDB](https://openmldb.ai/docs/en/v0.5/deploy/install_deploy.html).
+
+You can follow [run OpenMLDB in docker](https://openmldb.ai/docs/zh/v0.5/quickstart/openmldb_quickstart.html#id11)
+to a quick start.
+
+### Python env
+
+The OpenMLDB task will use OpenMLDB Python SDK to connect OpenMLDB cluster. So you should have the Python env.
+
+We will use `python3` by default. You can set `PYTHON_HOME` to use your custom python env.
+
+Make sure you have installed OpenMLDB Python SDK, using `pip install openmldb`.
Review Comment:
```suggestion
Make sure you have installed OpenMLDB Python SDK in the host where the worker server running, using `pip install openmldb`.
```
##########
docs/configs/docsdev.js:
##########
@@ -161,6 +161,10 @@ export default {
title: 'MLflow',
link: '/en-us/docs/dev/user_doc/guide/task/mlflow.html',
},
+ {
+ title: 'Openmldb',
+ link: '/en-us/docs/dev/user_doc/guide/task/openmldb.html',
+ },
Review Comment:
You have to add zh docs into https://github.com/apache/dolphinscheduler/blob/dev/docs/configs/docsdev.js#L528 you could search `MLflow` in file https://github.com/apache/dolphinscheduler/blob/dev/docs/configs/docsdev.js and there are four match item here
##########
dolphinscheduler-ui/src/locales/modules/en_US.ts:
##########
@@ -997,6 +997,14 @@ const project = {
mlflowProjectRepository_tips: 'github respository or path on worker',
mlflowProjectVersion: 'Project Version',
mlflowProjectVersion_tips: 'git version',
+ openmldb_zk_address: 'zookeeper address',
Review Comment:
we should modify system screenshots at the same time when we change the keyword from `zk` to `zookeeper`
##########
docs/docs/en/guide/task/openmldb.md:
##########
@@ -0,0 +1,76 @@
+# OpenMLDB Node
+
+## Overview
+
+[OpenMLDB](https://openmldb.ai/) is an excellent open source machine learning database, providing a full-stack
+FeatureOps solution for production.
+
+OpenMLDB task plugin used to execute tasks on OpenMLDB cluster.
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the
+ DAG editing page.
+- Drag from the toolbar <img src="/img/tasks/icons/openmldb.png" width="15"/> task node to canvas.
+
+## Task Example
+
+First, introduce some general parameters of DolphinScheduler
+
+- **Node name**: The node name in a workflow definition is unique.
+- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select
+ the `prohibition execution`.
+- **Descriptive information**: Describe the function of the node.
+- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high
+ to low, and tasks with the same priority will execute in a first-in first-out order.
+- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected,
+ randomly select a worker machine for execution.
+- **Environment Name**: Configure the environment name in which run the script.
+- **Times of failed retry attempts**: The number of times the task failed to resubmit.
+- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task.
+- **Delayed execution time**: The time (unit minute) that a task delays in execution.
+- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm
+ email will send and the task execution will fail.
+- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as
+ upstream of the current task.
+
+### OpenMLDB Parameters
+
+**Task Parameter**
+
+- **zookeeper** ：OpenMLDB cluster zookeeper address, e.g. 127.0.0.1:2181.
+- **zookeeper path** : OpenMLDB cluster zookeeper path, e.g. /openmldb.
+- **Execute Mode** ：determine the init mode, offline or online. You can switch it in sql statement.
+- **SQL statement** ：SQL statement.
+- Custom parameters: It is the user-defined parameters of Python, which will replace the content with \${variable} in the script.
+
+Here are some examples:
+
+#### Load data
+
+
+
+We use `LOAD DATA` to load data into OpenMLDB cluster. We select `offline` here, so it will load to offline storage.
+
+#### Feature extraction
+
+
+
+We use `SELECT INTO` to do feature extraction. We select `offline` here, so it will run sql on offline engine.
+
+## Environment to prepare
+
+### Start the OpenMLDB cluster
+
+You should create an OpenMLDB cluster first. If in production env, please check [deploy OpenMLDB](https://openmldb.ai/docs/en/v0.5/deploy/install_deploy.html).
+
+You can follow [run OpenMLDB in docker](https://openmldb.ai/docs/zh/v0.5/quickstart/openmldb_quickstart.html#id11)
+to a quick start.
+
+### Python env
+
+The OpenMLDB task will use OpenMLDB Python SDK to connect OpenMLDB cluster. So you should have the Python env.
+
+We will use `python3` by default. You can set `PYTHON_HOME` to use your custom python env.
+
+Make sure you have installed OpenMLDB Python SDK, using `pip install openmldb`.
Review Comment:
we should mention users install deps in the worker server to run the task. Same as ZH doc
##########
docs/docs/zh/guide/task/openmldb.md:
##########
@@ -0,0 +1,69 @@
+# OpenMLDB 节点
+
+## 综述
+
+[OpenMLDB](https://openmldb.ai/) 是一个优秀的开源机器学习数据库，提供生产级数据及特征开发全栈解决方案。
+
+OpenMLDB任务组件可以连接OpenMLDB集群执行任务。
+
+## 创建任务
+
+- 点击项目管理-项目名称-工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面；
+- 拖动工具栏的 <img src="/img/tasks/icons/openmldb.png" width="15"/> 任务节点到画板中。
+
+
Review Comment:
remove unused adding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1135750362
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [46.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1134326038
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10198?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10198](https://codecov.io/gh/apache/dolphinscheduler/pull/10198?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2e535b9) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/7a8f3f1810ec283fc5acd66077d46db88b0400af?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7a8f3f1) will **decrease** coverage by `0.10%`.
> The diff coverage is `10.30%`.
> :exclamation: Current head 2e535b9 differs from pull request most recent head 1340398. Consider uploading reports for the commit 1340398 to get more accurate results
```diff
@@ Coverage Diff @@
## dev #10198 +/- ##
============================================
- Coverage 40.94% 40.84% -0.11%
- Complexity 4732 4733 +1
============================================
Files 854 858 +4
Lines 34553 34650 +97
Branches 3818 3827 +9
============================================
+ Hits 14148 14153 +5
- Misses 19051 19141 +90
- Partials 1354 1356 +2
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10198?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...duler/plugin/task/openmldb/OpenmldbParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stb3Blbm1sZGIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svb3Blbm1sZGIvT3Blbm1sZGJQYXJhbWV0ZXJzLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...uler/plugin/task/openmldb/OpenmldbTaskChannel.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stb3Blbm1sZGIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svb3Blbm1sZGIvT3Blbm1sZGJUYXNrQ2hhbm5lbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ugin/task/openmldb/OpenmldbTaskChannelFactory.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stb3Blbm1sZGIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svb3Blbm1sZGIvT3Blbm1sZGJUYXNrQ2hhbm5lbEZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...inscheduler/plugin/task/openmldb/OpenmldbTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stb3Blbm1sZGIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svb3Blbm1sZGIvT3Blbm1sZGJUYXNrLmphdmE=) | `12.65% <12.65%> (ø)` | |
| [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `77.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...org/apache/dolphinscheduler/remote/utils/Host.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL3V0aWxzL0hvc3QuamF2YQ==) | `37.77% <0.00%> (-2.23%)` | :arrow_down: |
| [...dolphinscheduler/remote/future/ResponseFuture.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2Z1dHVyZS9SZXNwb25zZUZ1dHVyZS5qYXZh) | `81.96% <0.00%> (-1.64%)` | :arrow_down: |
| [...e/dolphinscheduler/remote/NettyRemotingClient.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10198/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL05ldHR5UmVtb3RpbmdDbGllbnQuamF2YQ==) | `52.11% <0.00%> (-1.41%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10198?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10198?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [7a8f3f1...1340398](https://codecov.io/gh/apache/dolphinscheduler/pull/10198?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1135928605
Hi there, I've fixed all. Plz help me to check again, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r879435574
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/main/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTask.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.google.common.base.Preconditions;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+/**
+ * openmldb task
+ */
+public class OpenmldbTask extends AbstractTaskExecutor {
Review Comment:
Found out this task class have many similar to `PythonTask`, could we extends from python task instead of abstract?
##########
docs/docs/en/guide/task/openmldb.md:
##########
@@ -0,0 +1,76 @@
+# OpenMLDB Node
+
+## Overview
+
+[OpenMLDB](https://openmldb.ai/) is an excellent open source machine learning database, providing a full-stack
+FeatureOps solution for production.
+
+OpenMLDB task plugin used to execute tasks on OpenMLDB cluster.
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the
+ DAG editing page.
+- Drag from the toolbar <img src="/img/tasks/icons/openmldb.png" width="15"/> task node to canvas.
+
+## Task Example
+
+First, introduce some general parameters of DolphinScheduler
+
+- **Node name**: The node name in a workflow definition is unique.
+- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select
+ the `prohibition execution`.
+- **Descriptive information**: Describe the function of the node.
+- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high
+ to low, and tasks with the same priority will execute in a first-in first-out order.
+- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected,
+ randomly select a worker machine for execution.
+- **Environment Name**: Configure the environment name in which run the script.
+- **Times of failed retry attempts**: The number of times the task failed to resubmit.
+- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task.
+- **Delayed execution time**: The time (unit minute) that a task delays in execution.
+- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm
+ email will send and the task execution will fail.
+- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as
+ upstream of the current task.
+
+### OpenMLDB Parameters
+
+**Task Parameter**
+
+- **zk** ：OpenMLDB cluster address, e.g. 127.0.0.1:2181.
+- **zk path** : OpenMLDB cluster address, e.g. /openmldb.
Review Comment:
I think we should better use the keyword `Zookeeper` in both document and UI
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-python/src/test/java/org/apache/dolphinscheduler/plugin/task/python/PythonTaskTest.java:
##########
@@ -35,4 +35,4 @@ public void buildPythonExecuteCommand() throws Exception {
private PythonTask createPythonTask() {
return new PythonTask(null);
}
-}
\ No newline at end of file
+}
Review Comment:
I think it is unrelated change here
##########
docs/docs/en/guide/task/openmldb.md:
##########
@@ -0,0 +1,76 @@
+# OpenMLDB Node
+
+## Overview
+
+[OpenMLDB](https://openmldb.ai/) is an excellent open source machine learning database, providing a full-stack
+FeatureOps solution for production.
+
+OpenMLDB task plugin used to execute tasks on OpenMLDB cluster.
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the "Create Workflow" button to enter the
+ DAG editing page.
+- Drag from the toolbar <img src="/img/tasks/icons/openmldb.png" width="15"/> task node to canvas.
+
+## Task Example
+
+First, introduce some general parameters of DolphinScheduler
+
+- **Node name**: The node name in a workflow definition is unique.
+- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select
+ the `prohibition execution`.
+- **Descriptive information**: Describe the function of the node.
+- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high
+ to low, and tasks with the same priority will execute in a first-in first-out order.
+- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected,
+ randomly select a worker machine for execution.
+- **Environment Name**: Configure the environment name in which run the script.
+- **Times of failed retry attempts**: The number of times the task failed to resubmit.
+- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task.
+- **Delayed execution time**: The time (unit minute) that a task delays in execution.
+- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm
+ email will send and the task execution will fail.
+- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as
+ upstream of the current task.
+
+### OpenMLDB Parameters
+
+**Task Parameter**
+
+- **zk** ：OpenMLDB cluster address, e.g. 127.0.0.1:2181.
+- **zk path** : OpenMLDB cluster address, e.g. /openmldb.
+- **Execute Mode** ：determine the init mode, offline or online. You can switch it in sql statement.
+- **SQL statement** ：SQL statement.
+- Custom parameters: It is the user-defined parameters of Python, which will replace the content with \${variable} in the script.
+
+Here are some examples:
+
+#### Load data
+
+
+
+We use `LOAD DATA` to load data into OpenMLDB cluster. We select `offline` here, so it will load to offline storage.
+
+#### Feature extraction
+
+
+
+We use `SELECT INTO` to do feature extraction. We select `offline` here, so it will run sql on offline engine.
+
+## Environment to prepare
+
+### Start the OpenMLDB cluster
+
+You should create an OpenMLDB cluster first. If in production env, please check [deploy OpenMLDB](https://openmldb.ai/docs/en/v0.5/deploy/install_deploy.html).
+
+You can follow [run OpenMLDB in docker](https://openmldb.ai/docs/zh/v0.5/quickstart/openmldb_quickstart.html#id11)
+to a quick start.
+
+### Python env
+
+The OpenMLDB task will use OpenMLDB Python SDK to connect OpenMLDB cluster. So you should have the Python env.
+
+We will use `python3` by default. You can set `PYTHON_HOME` to use your custom python env.
+
+Make sure you have installed OpenMLDB Python SDK, using 'pip install openmldb'.
Review Comment:
```suggestion
Make sure you have installed OpenMLDB Python SDK, using `pip install openmldb`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r885149777
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-openmldb.ts:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { useI18n } from 'vue-i18n'
+import { useCustomParams, useResources } from '.'
+import type { IJsonItem } from '../types'
+
+export function useOpenmldb(model: { [field: string]: any }): IJsonItem[] {
+ const { t } = useI18n()
+ const options = [
+ {
+ label: t('project.node.openmldb_execute_mode_offline'),
+ value: 'offline'
+ },
+ {
+ label: t('project.node.openmldb_execute_mode_online'),
+ value: 'online'
+ }
+ ]
+ return [
+ {
+ type: 'input',
+ field: 'zk',
+ name: t('project.node.openmldb_zk_address'),
+ props: {
+ placeholder: t('project.node.openmldb_zk_address_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ required: true,
+ validator(validate: any, value: string) {
+ if (!value) {
+ return new Error(t('project.node.openmldb_zk_address_tips'))
+ }
+ }
+ }
+ },
+ {
+ type: 'input',
+ field: 'zkPath',
+ name: t('project.node.openmldb_zk_path'),
+ props: {
+ placeholder: t('project.node.openmldb_zk_path_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ required: true,
+ validator(validate: any, value: string) {
+ if (!value) {
+ return new Error(t('project.node.openmldb_zk_path_tips'))
+ }
+ }
+ }
+ },
+ {
+ type: 'radio',
+ field: 'executeMode',
+ name: t('project.node.openmldb_execute_mode'),
+ options: options
+ },
+ {
+ type: 'editor',
+ field: 'sql',
+ name: t('project.node.sql_statement'),
+ validate: {
+ trigger: ['input', 'trigger'],
+ required: true,
+ message: t('project.node.sql_empty_tips')
+ }
+ },
+ useResources(),
+ ...useCustomParams({ model, field: 'localParams', isSimple: false })
+ ]
+}
Review Comment:
ok, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1135469862
> Please associate the corresponding issue. @vagetablechicken
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1135365903
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [10.3% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [7.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r879156501
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/pom.xml:
##########
@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>dolphinscheduler-task-plugin</artifactId>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <version>dev-SNAPSHOT</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>dolphinscheduler-task-openmldb</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-spi</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <artifactId>dolphinscheduler-task-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ <version>1.18.20</version>
+ </dependency>
Review Comment:
Hi, this effects developers' IDE installation, please open a discussion before adding this, (though I like to use this too)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r880096191
##########
docs/docs/zh/guide/task/openmldb.md:
##########
@@ -0,0 +1,71 @@
+# MLflow节点
Review Comment:
```suggestion
# OpenMLDB 节点
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Amy0104 commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
Amy0104 commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r884782376
##########
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-openmldb.ts:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { reactive } from 'vue'
+import * as Fields from '../fields/index'
+import type { IJsonItem, INodeData } from '../types'
+import { ITaskData } from '../types'
+
+export function useOpenmldb({
+ projectCode,
+ from = 0,
+ readonly,
+ data
+}: {
+ projectCode: number
+ from?: number
+ readonly?: boolean
+ data?: ITaskData
+}) {
+ const model = reactive({
+ name: '',
+ taskType: 'OPENMLDB',
+ flag: 'YES',
+ description: '',
+ timeoutFlag: false,
+ localParams: [],
+ environmentCode: null,
+ failRetryInterval: 1,
+ failRetryTimes: 0,
+ workerGroup: 'default',
+ delayTime: 0,
+ timeout: 30,
+ zk: '',
+ zkPath: '',
+ executeMode: 'offline'
+ } as INodeData)
Review Comment:
Here you should set the default value of timeoutNotifyStrategy to ['WARN'].
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-openmldb.ts:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { useI18n } from 'vue-i18n'
+import { useCustomParams, useResources } from '.'
+import type { IJsonItem } from '../types'
+
+export function useOpenmldb(model: { [field: string]: any }): IJsonItem[] {
+ const { t } = useI18n()
+ const options = [
+ {
+ label: t('project.node.openmldb_execute_mode_offline'),
+ value: 'offline'
+ },
+ {
+ label: t('project.node.openmldb_execute_mode_online'),
+ value: 'online'
+ }
+ ]
+ return [
+ {
+ type: 'input',
+ field: 'zk',
+ name: t('project.node.openmldb_zk_address'),
+ props: {
+ placeholder: t('project.node.openmldb_zk_address_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ required: true,
+ validator(validate: any, value: string) {
+ if (!value) {
+ return new Error(t('project.node.openmldb_zk_address_tips'))
+ }
+ }
+ }
+ },
+ {
+ type: 'input',
+ field: 'zkPath',
+ name: t('project.node.openmldb_zk_path'),
+ props: {
+ placeholder: t('project.node.openmldb_zk_path_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ required: true,
+ validator(validate: any, value: string) {
+ if (!value) {
+ return new Error(t('project.node.openmldb_zk_path_tips'))
+ }
+ }
+ }
+ },
+ {
+ type: 'radio',
+ field: 'executeMode',
+ name: t('project.node.openmldb_execute_mode'),
+ options: options
+ },
+ {
+ type: 'editor',
+ field: 'sql',
+ name: t('project.node.sql_statement'),
+ validate: {
+ trigger: ['input', 'trigger'],
+ required: true,
+ message: t('project.node.sql_empty_tips')
+ }
+ },
+ useResources(),
+ ...useCustomParams({ model, field: 'localParams', isSimple: false })
+ ]
+}
Review Comment:
The code should be formatted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] songjianet commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
songjianet commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r884835723
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-openmldb.ts:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { useI18n } from 'vue-i18n'
+import { useCustomParams, useResources } from '.'
+import type { IJsonItem } from '../types'
+
+export function useOpenmldb(model: { [field: string]: any }): IJsonItem[] {
+ const { t } = useI18n()
+ const options = [
+ {
+ label: t('project.node.openmldb_execute_mode_offline'),
+ value: 'offline'
+ },
+ {
+ label: t('project.node.openmldb_execute_mode_online'),
+ value: 'online'
+ }
+ ]
+ return [
+ {
+ type: 'input',
+ field: 'zk',
+ name: t('project.node.openmldb_zk_address'),
+ props: {
+ placeholder: t('project.node.openmldb_zk_address_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ required: true,
+ validator(validate: any, value: string) {
+ if (!value) {
+ return new Error(t('project.node.openmldb_zk_address_tips'))
+ }
+ }
+ }
+ },
+ {
+ type: 'input',
+ field: 'zkPath',
+ name: t('project.node.openmldb_zk_path'),
+ props: {
+ placeholder: t('project.node.openmldb_zk_path_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ required: true,
+ validator(validate: any, value: string) {
+ if (!value) {
+ return new Error(t('project.node.openmldb_zk_path_tips'))
+ }
+ }
+ }
+ },
+ {
+ type: 'radio',
+ field: 'executeMode',
+ name: t('project.node.openmldb_execute_mode'),
+ options: options
+ },
+ {
+ type: 'editor',
+ field: 'sql',
+ name: t('project.node.sql_statement'),
+ validate: {
+ trigger: ['input', 'trigger'],
+ required: true,
+ message: t('project.node.sql_empty_tips')
+ }
+ },
+ useResources(),
+ ...useCustomParams({ model, field: 'localParams', isSimple: false })
+ ]
+}
Review Comment:
For the formatting issue, you can leave it alone, I will do the formatting later. @vagetablechicken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r882599221
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/test/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTaskTest.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.powermock.reflect.Whitebox;
+
+public class OpenmldbTaskTest {
+ static class MockOpenmldbTask extends OpenmldbTask {
+ /**
+ * constructor
+ *
+ * @param taskRequest taskRequest
+ */
+ public MockOpenmldbTask(TaskExecutionContext taskRequest) {
+ super(taskRequest);
+ }
+
+ @Override
+ protected Map<String, Property> mergeParamsWithContext(AbstractParameters parameters) {
+ return new HashMap<>();
+ }
+ }
+
+ private OpenmldbTask createOpenmldbTask() {
+ return new MockOpenmldbTask(null);
+ }
+
+ @Test
+ public void buildPythonExecuteCommand() throws Exception {
+ OpenmldbTask openmldbTask = createOpenmldbTask();
+ String pythonFile = "test.py";
+ String result1 = openmldbTask.buildPythonExecuteCommand(pythonFile);
+ Assert.assertEquals("python3 test.py", result1);
+ }
+
+ @Test
+ public void buildSQLWithComment() throws Exception {
+ OpenmldbTask openmldbTask = createOpenmldbTask();
+ OpenmldbParameters openmldbParameters = new OpenmldbParameters();
+ openmldbParameters.setExecuteMode("offline");
+ String rawSQLScript = "select * from users\r\n"
+ + "-- some comment\n"
+ + "inner join order on users.order_id = order.id; \n\n;"
+ + "select * from users;";
Review Comment:
👍 thanks for adding this test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1141618573
Merging... Thank @vagetablechicken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] vagetablechicken commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r880116513
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-openmldb/src/main/java/org/apache/dolphinscheduler/plugin/task/openmldb/OpenmldbTask.java:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.openmldb;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.MapUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.google.common.base.Preconditions;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+/**
+ * openmldb task
+ */
+public class OpenmldbTask extends AbstractTaskExecutor {
Review Comment:
After the fix in `buildPythonScriptContent`, it's more similar. I'll try to extend from `PythonTask`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] songjianet commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
songjianet commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1135462366
Please associate the corresponding issue. @vagetablechicken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on a diff in pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on code in PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#discussion_r880096191
##########
docs/docs/zh/guide/task/openmldb.md:
##########
@@ -0,0 +1,71 @@
+# MLflow节点
Review Comment:
```suggestion
# OpenMLDB 节点
```
A small mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10198: [Feature][Task-Plugin]Add OpenMLDB task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10198:
URL: https://github.com/apache/dolphinscheduler/pull/10198#issuecomment-1135946276
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10198)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10198&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list) [49.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10198&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org