You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/08/22 03:43:19 UTC
[GitHub] [dolphinscheduler] baisui1981 opened a new pull request #6015: Enhance using experience of DataX in DS by integeration with TIS
baisui1981 opened a new pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
implemention of [https://github.com/apache/dolphinscheduler/issues/5992](https://github.com/apache/dolphinscheduler/issues/5992)
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-906016540
> @kezhenxu94 Hi, can you help to see the problem here? It seems that the task-plugin module does not have license detection?
Hi, @CalvinKirs why wasn't module `dolphinscheduler-task-plugin` added in root `pom.xml`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r694445228
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/pom.xml
##########
@@ -0,0 +1,88 @@
+<?xml version="1.0" encoding="UTF-8"?>
Review comment:
Still missing license header
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/pom.xml
##########
@@ -0,0 +1,88 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>dolphinscheduler-task-plugin</artifactId>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <version>1.3.6-SNAPSHOT</version>
+ </parent>
+
+
+ <modelVersion>4.0.0</modelVersion>
+
+<!-- <packaging>dolphinscheduler-plugin</packaging>-->
Review comment:
remove this
##########
File path: dolphinscheduler-dist/release-docs/LICENSE
##########
@@ -492,6 +492,8 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
slf4j-api 1.7.5: https://mvnrepository.com/artifact/org.slf4j/slf4j-api/1.7.5, MIT
animal-sniffer-annotations 1.14 https://mvnrepository.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.14, MIT
checker-compat-qual 2.0.0 https://mvnrepository.com/artifact/org.checkerframework/checker-compat-qual/2.0.0, MIT + GPLv2
+ moco-core 1.2.0: https://mvnrepository.com/artifact/com.github.dreamhead/moco-core/1.2.0, MIT
+ moco-runner 1.2.0: https://mvnrepository.com/artifact/com.github.dreamhead/moco-runner/1.2.0, MIT
Review comment:
Why should we add these, they are `test` scope dependencies
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/test/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskTest.java
##########
@@ -0,0 +1,151 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.dreamhead.moco.HttpServer;
+import org.apache.commons.io.IOUtils;
+import org.apache.dolphinscheduler.server.worker.task.TaskProps;
+import org.apache.dolphinscheduler.spi.task.ExecutionStatus;
+import org.apache.dolphinscheduler.spi.task.TaskRequest;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.powermock.api.mockito.PowerMockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+
+import static com.github.dreamhead.moco.Moco.pathResource;
+import static com.github.dreamhead.moco.MocoJsonRunner.jsonHttpServer;
+import static com.github.dreamhead.moco.Runner.running;
+
+
+public class TISTaskTest {
+ private static final Logger logger = LoggerFactory.getLogger(TISTaskTest.class);
+ private TISTask tisTask;
+
+ private TaskRequest taskExecutionContext;
+
+ @Before
+ public void before() throws Exception {
+
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ props.setEnvFile(".dolphinscheduler_env.sh");
+ props.setTaskStartTime(new Date());
+ props.setTaskTimeout(0);
+ props.setTaskParams("{\"targetJobName\":\"mysql_elastic\"}");
+
+
+ taskExecutionContext = Mockito.mock(TaskRequest.class);
+ Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(props.getTaskParams());
+ Mockito.when(taskExecutionContext.getExecutePath()).thenReturn("/tmp");
+ Mockito.when(taskExecutionContext.getTaskAppId()).thenReturn(UUID.randomUUID().toString());
+ Mockito.when(taskExecutionContext.getTenantCode()).thenReturn("root");
+ Mockito.when(taskExecutionContext.getStartTime()).thenReturn(new Date());
+ Mockito.when(taskExecutionContext.getTaskTimeout()).thenReturn(10000);
+ Mockito.when(taskExecutionContext.getLogPath()).thenReturn("/tmp/dx");
+// Mockito.when(taskExecutionContext.getVarPool())
+// .thenReturn("[{\"direct\":\"IN\",\"prop\":\"" + TISTask.KEY_POOL_VAR_TIS_HOST + "\",\"type\":\"VARCHAR\",\"value\":\"127.0.0.1:8080\"}]");
+ Map<String, String> gloabParams = Collections.singletonMap(TISTask.KEY_POOL_VAR_TIS_HOST, "127.0.0.1:8080");
+ Mockito.when(taskExecutionContext.getDefinedParams()).thenReturn(gloabParams);
+
+ tisTask = PowerMockito.spy(new TISTask(taskExecutionContext, logger));
+ tisTask.init();
+
+ }
+
+ /**
+ * Method: DataxTask()
+ */
+ @Test
+ public void testDataxTask()
+ throws Exception {
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ Assert.assertNotNull(new TISTask(null, logger));
+ }
+
+ /**
+ * Method: init
+ */
+ @Test
+ public void testInit()
+ throws Exception {
+ try {
+ tisTask.init();
+ } catch (Exception e) {
+ Assert.fail(e.getMessage());
+ }
+ }
+
+ /**
+ * Method: handle()
+ */
+ @Test
+ public void testHandle()
+ throws Exception {
+ String taskName = "mysql_elastic";
+ int taskid = 1087;
+
+ // https://github.com/dreamhead/moco/blob/master/moco-doc/websocket-apis.md
+ // https://github.com/dreamhead/moco/blob/master/moco-doc/socket-apis.md#json-text
+ // https://blog.csdn.net/qq_36450484/article/details/99655976
Review comment:
Remove this kind of links
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskChannel.java
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.tis;
+
+import org.apache.dolphinscheduler.spi.task.AbstractTask;
+import org.apache.dolphinscheduler.spi.task.TaskChannel;
+import org.apache.dolphinscheduler.spi.task.TaskRequest;
+
+/**
+ *
+ **/
Review comment:
remove
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/test/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskTest.java
##########
@@ -0,0 +1,151 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.dreamhead.moco.HttpServer;
+import org.apache.commons.io.IOUtils;
+import org.apache.dolphinscheduler.server.worker.task.TaskProps;
+import org.apache.dolphinscheduler.spi.task.ExecutionStatus;
+import org.apache.dolphinscheduler.spi.task.TaskRequest;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.powermock.api.mockito.PowerMockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+
+import static com.github.dreamhead.moco.Moco.pathResource;
+import static com.github.dreamhead.moco.MocoJsonRunner.jsonHttpServer;
+import static com.github.dreamhead.moco.Runner.running;
+
+
+public class TISTaskTest {
+ private static final Logger logger = LoggerFactory.getLogger(TISTaskTest.class);
+ private TISTask tisTask;
+
+ private TaskRequest taskExecutionContext;
+
+ @Before
+ public void before() throws Exception {
+
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ props.setEnvFile(".dolphinscheduler_env.sh");
+ props.setTaskStartTime(new Date());
+ props.setTaskTimeout(0);
+ props.setTaskParams("{\"targetJobName\":\"mysql_elastic\"}");
+
+
+ taskExecutionContext = Mockito.mock(TaskRequest.class);
+ Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(props.getTaskParams());
+ Mockito.when(taskExecutionContext.getExecutePath()).thenReturn("/tmp");
+ Mockito.when(taskExecutionContext.getTaskAppId()).thenReturn(UUID.randomUUID().toString());
+ Mockito.when(taskExecutionContext.getTenantCode()).thenReturn("root");
+ Mockito.when(taskExecutionContext.getStartTime()).thenReturn(new Date());
+ Mockito.when(taskExecutionContext.getTaskTimeout()).thenReturn(10000);
+ Mockito.when(taskExecutionContext.getLogPath()).thenReturn("/tmp/dx");
+// Mockito.when(taskExecutionContext.getVarPool())
+// .thenReturn("[{\"direct\":\"IN\",\"prop\":\"" + TISTask.KEY_POOL_VAR_TIS_HOST + "\",\"type\":\"VARCHAR\",\"value\":\"127.0.0.1:8080\"}]");
+ Map<String, String> gloabParams = Collections.singletonMap(TISTask.KEY_POOL_VAR_TIS_HOST, "127.0.0.1:8080");
+ Mockito.when(taskExecutionContext.getDefinedParams()).thenReturn(gloabParams);
+
+ tisTask = PowerMockito.spy(new TISTask(taskExecutionContext, logger));
+ tisTask.init();
+
+ }
+
+ /**
+ * Method: DataxTask()
+ */
+ @Test
+ public void testDataxTask()
+ throws Exception {
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ Assert.assertNotNull(new TISTask(null, logger));
+ }
+
+ /**
+ * Method: init
+ */
+ @Test
+ public void testInit()
+ throws Exception {
+ try {
+ tisTask.init();
+ } catch (Exception e) {
+ Assert.fail(e.getMessage());
+ }
+ }
+
+ /**
+ * Method: handle()
+ */
Review comment:
Remove meaningless comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r695559049
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/test/resources/org/apache/dolphinscheduler/plugin/task/tis/TISTaskTest.json
##########
@@ -0,0 +1,60 @@
+[
+ {
+ "description": "trigger TIS task execute",
+ "request": {
+ "uri": "/tjs/coredefine/coredefine.ajax",
+ "method": "post",
+ "headers": {
Review comment:
sorry,I made a mistake, thank you for your reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] baisui1981 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
baisui1981 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r694533683
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/pom.xml
##########
@@ -0,0 +1,88 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>dolphinscheduler-task-plugin</artifactId>
+ <groupId>org.apache.dolphinscheduler</groupId>
+ <version>1.3.6-SNAPSHOT</version>
+ </parent>
+
+
+ <modelVersion>4.0.0</modelVersion>
+
+<!-- <packaging>dolphinscheduler-plugin</packaging>-->
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a change in pull request #6015: Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r693444627
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISParameters.java
##########
@@ -0,0 +1,59 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
Review comment:
```suggestion
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.tis;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r696231720
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISTask.java
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.tis;
+
+import org.apache.dolphinscheduler.common.utils.CollectionUtils;
+import org.apache.dolphinscheduler.spi.task.AbstractParameters;
+import org.apache.dolphinscheduler.spi.task.AbstractTask;
+import org.apache.dolphinscheduler.spi.task.TaskConstants;
+import org.apache.dolphinscheduler.spi.task.TaskRequest;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import org.apache.http.HttpEntity;
+import org.apache.http.StatusLine;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+
+import java.net.HttpURLConnection;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import org.asynchttpclient.Dsl;
+import org.asynchttpclient.ws.WebSocket;
+import org.asynchttpclient.ws.WebSocketListener;
+import org.asynchttpclient.ws.WebSocketUpgradeHandler;
+import org.slf4j.Logger;
+
+/**
+ * TIS DataX Task
+ **/
+public class TISTask extends AbstractTask {
+
+ public static final String WS_REQUEST_PATH = "/tjs/download/logfeedback";
+ public static final String KEY_POOL_VAR_TIS_HOST = "tisHost";
+ private final TaskRequest taskExecutionContext;
+
+ private TISParameters tisParameters;
+
+ public TISTask(TaskRequest taskExecutionContext, Logger logger) {
+ super(taskExecutionContext, logger);
+ this.taskExecutionContext = taskExecutionContext;
+ }
+
+ @Override
+ public void init() {
+ super.init();
+ logger.info("tis task params {}", taskExecutionContext.getTaskParams());
+ tisParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), TISParameters.class);
+ if (!tisParameters.checkParameters()) {
+ throw new RuntimeException("datax task params is not valid");
+ }
+ }
+
+ @Override
+ public void handle() throws Exception {
+ // Trigger TIS DataX pipeline
+ logger.info("start execute TIS task");
+ long startTime = System.currentTimeMillis();
+ String targetJobName = this.tisParameters.getTargetJobName();
+ final String tisHost = taskExecutionContext.getDefinedParams().get(KEY_POOL_VAR_TIS_HOST);
+ if (StringUtils.isEmpty(tisHost)) {
+ throw new IllegalStateException("global var '" + KEY_POOL_VAR_TIS_HOST + "' can not be empty");
+ }
+ try {
+ final String triggerUrl = String.format("http://%s/tjs/coredefine/coredefine.ajax", tisHost);
+ final String getStatusUrl = String.format("http://%s/tjs/config/config.ajax?action=collection_action&emethod=get_task_status", tisHost);
+ HttpPost post = new HttpPost(triggerUrl);
Review comment:
hi, it seems that the parameter substitution step is missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a change in pull request #6015: Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r693444786
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskChannelFactory.java
##########
@@ -0,0 +1,49 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
Review comment:
```suggestion
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.tis;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905182819
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-906017791
> > @kezhenxu94 Hi, can you help to see the problem here? It seems that the task-plugin module does not have license detection?
>
> Hi, @CalvinKirs why wasn't module `dolphinscheduler-task-plugin` added in root `pom.xml`?
thanks for your remind,I left this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] baisui1981 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
baisui1981 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r694547237
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/test/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskTest.java
##########
@@ -0,0 +1,151 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.dreamhead.moco.HttpServer;
+import org.apache.commons.io.IOUtils;
+import org.apache.dolphinscheduler.server.worker.task.TaskProps;
+import org.apache.dolphinscheduler.spi.task.ExecutionStatus;
+import org.apache.dolphinscheduler.spi.task.TaskRequest;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.powermock.api.mockito.PowerMockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+
+import static com.github.dreamhead.moco.Moco.pathResource;
+import static com.github.dreamhead.moco.MocoJsonRunner.jsonHttpServer;
+import static com.github.dreamhead.moco.Runner.running;
+
+
+public class TISTaskTest {
+ private static final Logger logger = LoggerFactory.getLogger(TISTaskTest.class);
+ private TISTask tisTask;
+
+ private TaskRequest taskExecutionContext;
+
+ @Before
+ public void before() throws Exception {
+
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ props.setEnvFile(".dolphinscheduler_env.sh");
+ props.setTaskStartTime(new Date());
+ props.setTaskTimeout(0);
+ props.setTaskParams("{\"targetJobName\":\"mysql_elastic\"}");
+
+
+ taskExecutionContext = Mockito.mock(TaskRequest.class);
+ Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(props.getTaskParams());
+ Mockito.when(taskExecutionContext.getExecutePath()).thenReturn("/tmp");
+ Mockito.when(taskExecutionContext.getTaskAppId()).thenReturn(UUID.randomUUID().toString());
+ Mockito.when(taskExecutionContext.getTenantCode()).thenReturn("root");
+ Mockito.when(taskExecutionContext.getStartTime()).thenReturn(new Date());
+ Mockito.when(taskExecutionContext.getTaskTimeout()).thenReturn(10000);
+ Mockito.when(taskExecutionContext.getLogPath()).thenReturn("/tmp/dx");
+// Mockito.when(taskExecutionContext.getVarPool())
+// .thenReturn("[{\"direct\":\"IN\",\"prop\":\"" + TISTask.KEY_POOL_VAR_TIS_HOST + "\",\"type\":\"VARCHAR\",\"value\":\"127.0.0.1:8080\"}]");
+ Map<String, String> gloabParams = Collections.singletonMap(TISTask.KEY_POOL_VAR_TIS_HOST, "127.0.0.1:8080");
+ Mockito.when(taskExecutionContext.getDefinedParams()).thenReturn(gloabParams);
+
+ tisTask = PowerMockito.spy(new TISTask(taskExecutionContext, logger));
+ tisTask.init();
+
+ }
+
+ /**
+ * Method: DataxTask()
+ */
+ @Test
+ public void testDataxTask()
+ throws Exception {
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ Assert.assertNotNull(new TISTask(null, logger));
+ }
+
+ /**
+ * Method: init
+ */
+ @Test
+ public void testInit()
+ throws Exception {
+ try {
+ tisTask.init();
+ } catch (Exception e) {
+ Assert.fail(e.getMessage());
+ }
+ }
+
+ /**
+ * Method: handle()
+ */
+ @Test
+ public void testHandle()
+ throws Exception {
+ String taskName = "mysql_elastic";
+ int taskid = 1087;
+
+ // https://github.com/dreamhead/moco/blob/master/moco-doc/websocket-apis.md
+ // https://github.com/dreamhead/moco/blob/master/moco-doc/socket-apis.md#json-text
+ // https://blog.csdn.net/qq_36450484/article/details/99655976
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] baisui1981 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
baisui1981 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r694532919
##########
File path: dolphinscheduler-dist/release-docs/LICENSE
##########
@@ -492,6 +492,8 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
slf4j-api 1.7.5: https://mvnrepository.com/artifact/org.slf4j/slf4j-api/1.7.5, MIT
animal-sniffer-annotations 1.14 https://mvnrepository.com/artifact/org.codehaus.mojo/animal-sniffer-annotations/1.14, MIT
checker-compat-qual 2.0.0 https://mvnrepository.com/artifact/org.checkerframework/checker-compat-qual/2.0.0, MIT + GPLv2
+ moco-core 1.2.0: https://mvnrepository.com/artifact/com.github.dreamhead/moco-core/1.2.0, MIT
+ moco-runner 1.2.0: https://mvnrepository.com/artifact/com.github.dreamhead/moco-runner/1.2.0, MIT
Review comment:
thanks for your remind, i will remote the moco 2 line registion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905402961
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a change in pull request #6015: Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r693444765
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskChannel.java
##########
@@ -0,0 +1,35 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
Review comment:
```suggestion
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.tis;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a change in pull request #6015: Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r693444656
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISParamsConstants.java
##########
@@ -0,0 +1,25 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
Review comment:
```suggestion
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.tis;
```
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISParameters.java
##########
@@ -0,0 +1,59 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
Review comment:
```suggestion
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.tis;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905335194
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] removed a comment on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905402961
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905182819
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905335194
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] baisui1981 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
baisui1981 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r695549821
##########
File path: tools/dependencies/known-dependencies.txt
##########
@@ -253,3 +253,6 @@ xmlbeans-3.1.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.14.jar
+moco-core-1.2.0.jar
+moco-runner-1.2.0.jar
Review comment:
tks，i forgot to remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] baisui1981 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
baisui1981 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r694547488
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/test/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskTest.java
##########
@@ -0,0 +1,151 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.github.dreamhead.moco.HttpServer;
+import org.apache.commons.io.IOUtils;
+import org.apache.dolphinscheduler.server.worker.task.TaskProps;
+import org.apache.dolphinscheduler.spi.task.ExecutionStatus;
+import org.apache.dolphinscheduler.spi.task.TaskRequest;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+import org.powermock.api.mockito.PowerMockito;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+
+import static com.github.dreamhead.moco.Moco.pathResource;
+import static com.github.dreamhead.moco.MocoJsonRunner.jsonHttpServer;
+import static com.github.dreamhead.moco.Runner.running;
+
+
+public class TISTaskTest {
+ private static final Logger logger = LoggerFactory.getLogger(TISTaskTest.class);
+ private TISTask tisTask;
+
+ private TaskRequest taskExecutionContext;
+
+ @Before
+ public void before() throws Exception {
+
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ props.setEnvFile(".dolphinscheduler_env.sh");
+ props.setTaskStartTime(new Date());
+ props.setTaskTimeout(0);
+ props.setTaskParams("{\"targetJobName\":\"mysql_elastic\"}");
+
+
+ taskExecutionContext = Mockito.mock(TaskRequest.class);
+ Mockito.when(taskExecutionContext.getTaskParams()).thenReturn(props.getTaskParams());
+ Mockito.when(taskExecutionContext.getExecutePath()).thenReturn("/tmp");
+ Mockito.when(taskExecutionContext.getTaskAppId()).thenReturn(UUID.randomUUID().toString());
+ Mockito.when(taskExecutionContext.getTenantCode()).thenReturn("root");
+ Mockito.when(taskExecutionContext.getStartTime()).thenReturn(new Date());
+ Mockito.when(taskExecutionContext.getTaskTimeout()).thenReturn(10000);
+ Mockito.when(taskExecutionContext.getLogPath()).thenReturn("/tmp/dx");
+// Mockito.when(taskExecutionContext.getVarPool())
+// .thenReturn("[{\"direct\":\"IN\",\"prop\":\"" + TISTask.KEY_POOL_VAR_TIS_HOST + "\",\"type\":\"VARCHAR\",\"value\":\"127.0.0.1:8080\"}]");
+ Map<String, String> gloabParams = Collections.singletonMap(TISTask.KEY_POOL_VAR_TIS_HOST, "127.0.0.1:8080");
+ Mockito.when(taskExecutionContext.getDefinedParams()).thenReturn(gloabParams);
+
+ tisTask = PowerMockito.spy(new TISTask(taskExecutionContext, logger));
+ tisTask.init();
+
+ }
+
+ /**
+ * Method: DataxTask()
+ */
+ @Test
+ public void testDataxTask()
+ throws Exception {
+ TaskProps props = new TaskProps();
+ props.setExecutePath("/tmp");
+ props.setTaskAppId(String.valueOf(System.currentTimeMillis()));
+ props.setTaskInstanceId(1);
+ props.setTenantCode("1");
+ Assert.assertNotNull(new TISTask(null, logger));
+ }
+
+ /**
+ * Method: init
+ */
+ @Test
+ public void testInit()
+ throws Exception {
+ try {
+ tisTask.init();
+ } catch (Exception e) {
+ Assert.fail(e.getMessage());
+ }
+ }
+
+ /**
+ * Method: handle()
+ */
Review comment:
ok
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISTaskChannel.java
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.tis;
+
+import org.apache.dolphinscheduler.spi.task.AbstractTask;
+import org.apache.dolphinscheduler.spi.task.TaskChannel;
+import org.apache.dolphinscheduler.spi.task.TaskRequest;
+
+/**
+ *
+ **/
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905181195
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> :exclamation: No coverage uploaded for pull request base (`task-plugin@79952d9`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## task-plugin #6015 +/- ##
==============================================
Coverage ? 46.30%
Complexity ? 3744
==============================================
Files ? 613
Lines ? 25088
Branches ? 2843
==============================================
Hits ? 11616
Misses ? 12329
Partials ? 1143
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [79952d9...2f636b5](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905525957
Kudos, SonarCloud Quality Gate passed! 
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=6015&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=6015&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-906013268
@kezhenxu94 Hi, can you help to see the problem here? It seems that the task-plugin module does not have license detection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs merged pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
CalvinKirs merged pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] baisui1981 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
baisui1981 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r695543677
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/test/resources/org/apache/dolphinscheduler/plugin/task/tis/TISTaskTest.json
##########
@@ -0,0 +1,60 @@
+[
+ {
+ "description": "trigger TIS task execute",
+ "request": {
+ "uri": "/tjs/coredefine/coredefine.ajax",
+ "method": "post",
+ "headers": {
Review comment:
how to add license in json format file, as i have review some existing json format file in project DS have not found license annotation ever
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a change in pull request #6015: Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r693444722
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/main/java/org/apache/dolphinscheduler/plugin/task/tis/TISTask.java
##########
@@ -0,0 +1,344 @@
+package org.apache.dolphinscheduler.plugin.task.tis;/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
Review comment:
```suggestion
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dolphinscheduler.plugin.task.tis;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] kezhenxu94 commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r695459304
##########
File path: tools/dependencies/known-dependencies.txt
##########
@@ -253,3 +253,6 @@ xmlbeans-3.1.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.14.jar
+moco-core-1.2.0.jar
+moco-runner-1.2.0.jar
Review comment:
Hi @baisui1981 , you need to check why these 2 testing frameworks are included in the distributed binary tar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on a change in pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on a change in pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#discussion_r695386652
##########
File path: dolphinscheduler-task-plugin/dolphinscheduler-task-tis/src/test/resources/org/apache/dolphinscheduler/plugin/task/tis/TISTaskTest.json
##########
@@ -0,0 +1,60 @@
+[
+ {
+ "description": "trigger TIS task execute",
+ "request": {
+ "uri": "/tjs/coredefine/coredefine.ajax",
+ "method": "post",
+ "headers": {
Review comment:
Seems to have lost the license head
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter edited a comment on pull request #6015: [Feature-5992] Enhance using experience of DataX in DS by integeration with TIS
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6015:
URL: https://github.com/apache/dolphinscheduler/pull/6015#issuecomment-905181195
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> :exclamation: No coverage uploaded for pull request base (`task-plugin@79952d9`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## task-plugin #6015 +/- ##
==============================================
Coverage ? 46.30%
Complexity ? 3744
==============================================
Files ? 613
Lines ? 25088
Branches ? 2843
==============================================
Hits ? 11617
Misses ? 12328
Partials ? 1143
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [79952d9...fd73692](https://codecov.io/gh/apache/dolphinscheduler/pull/6015?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org