You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "HxpSerein (via GitHub)" <gi...@apache.org> on 2024/09/06 14:52:30 UTC
[PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
HxpSerein opened a new pull request, #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040
<!--
Thank you for contributing to StreamPark! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
## Contribution Checklist
- If this is your first time, please read our contributor guidelines: [Submit Code](https://streampark.apache.org/community/submit_guide/submit_code).
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-streampark/issues).
- Name the pull request in the form "[Feature] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- If the PR is unfinished, add `[WIP]` in your PR title, e.g., `[WIP][Feature] Title of the pull request`.
-->
## What changes were proposed in this pull request
Issue Number: close https://github.com/apache/incubator-streampark/issues/3960
<!--(For example: This pull request proposed to add checkstyle plugin).-->
## Brief change log
Add task distribution for streampark
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verifying this change
<!--*(Please pick either of the following options)*-->
This change added tests and can be verified as follows:
- ConsistentHashTest
<!--*(example:)*
- *Added integration tests for end-to-end.*
- *Added *Test to verify the change.*
- *Manually verified the change by testing locally.* -->
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2335053267
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [38 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "HxpSerein (via GitHub)" <gi...@apache.org>.
HxpSerein commented on code in PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#discussion_r1753082237
##########
streampark-console/streampark-console-service/src/main/resources/mapper/core/DistributionTaskMapper.xml:
##########
@@ -0,0 +1,28 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<!DOCTYPE mapper
+ PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
+ "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
+<mapper namespace="org.apache.streampark.console.core.mapper.DistributionTaskMapper">
+ <resultMap id="BaseResultMap" type="org.apache.streampark.console.core.entity.DistributionTask">
Review Comment:
Resolved.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/DistributionTaskServiceImpl.java:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.console.core.service.impl;
+
+import org.apache.streampark.console.base.util.ConsistentHash;
+import org.apache.streampark.console.base.util.JacksonUtils;
+import org.apache.streampark.console.core.bean.FlinkTaskItem;
+import org.apache.streampark.console.core.entity.Application;
+import org.apache.streampark.console.core.entity.DistributionTask;
+import org.apache.streampark.console.core.enums.DistributionTaskEnum;
+import org.apache.streampark.console.core.enums.EngineTypeEnum;
+import org.apache.streampark.console.core.mapper.DistributionTaskMapper;
+import org.apache.streampark.console.core.service.DistributionTaskService;
+import org.apache.streampark.console.core.service.application.ApplicationActionService;
+
+import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.beans.factory.annotation.Qualifier;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Service;
+import org.springframework.transaction.annotation.Propagation;
+import org.springframework.transaction.annotation.Transactional;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
+import java.util.stream.Collectors;
+
+@Slf4j
+@Service
+@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
+public class DistributionTaskServiceImpl extends ServiceImpl<DistributionTaskMapper, DistributionTask>
+ implements
+ DistributionTaskService {
+
+ /**
+ * Server name
+ */
+ private String serverName;
Review Comment:
Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2336569674
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [38 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys merged PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2338634785
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [38 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2339495168
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [44 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2341348455
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [44 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "Mrart (via GitHub)" <gi...@apache.org>.
Mrart commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2336597245
> Since the resource center is not yet developed and HA is incomplete, the current testing support is somewhat limited. To ensure code stability, I propose to temporarily support only Flink tasks, with support for Kubernetes and Spark jobs added once the resource center is completed. Do you have any suggestions on this approach? @Mrart @wolfboys
I agree it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "Mrart (via GitHub)" <gi...@apache.org>.
Mrart commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2338225655
Another suggestion is that we merge our commits before we merge them and change the commit to match what we changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#discussion_r1752294390
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -240,6 +240,19 @@ create table `t_flink_sql` (
) engine=innodb auto_increment=100000 default charset=utf8mb4 collate=utf8mb4_general_ci;
+-- ----------------------------
+-- Table structure for t_distribution_task
+-- ----------------------------
+drop table if exists `t_distribution_task`;
Review Comment:
Would it be better to rename table "t_distributed_task"?
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -240,6 +240,19 @@ create table `t_flink_sql` (
) engine=innodb auto_increment=100000 default charset=utf8mb4 collate=utf8mb4_general_ci;
+-- ----------------------------
+-- Table structure for t_ha_task
+-- ----------------------------
+drop table if exists `t_ha_task`;
+create table `t_ha_task` (
Review Comment:
I suggest the table name rename to `t_console_ha_task`
##########
streampark-console/streampark-console-service/src/main/resources/mapper/core/DistributionTaskMapper.xml:
##########
@@ -0,0 +1,28 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<!DOCTYPE mapper
+ PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
+ "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
+<mapper namespace="org.apache.streampark.console.core.mapper.DistributionTaskMapper">
+ <resultMap id="BaseResultMap" type="org.apache.streampark.console.core.entity.DistributionTask">
Review Comment:
We don't need to define this BaseResultMap because it has never been used.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/DistributionTaskServiceImpl.java:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.console.core.service.impl;
+
+import org.apache.streampark.console.base.util.ConsistentHash;
+import org.apache.streampark.console.base.util.JacksonUtils;
+import org.apache.streampark.console.core.bean.FlinkTaskItem;
+import org.apache.streampark.console.core.entity.Application;
+import org.apache.streampark.console.core.entity.DistributionTask;
+import org.apache.streampark.console.core.enums.DistributionTaskEnum;
+import org.apache.streampark.console.core.enums.EngineTypeEnum;
+import org.apache.streampark.console.core.mapper.DistributionTaskMapper;
+import org.apache.streampark.console.core.service.DistributionTaskService;
+import org.apache.streampark.console.core.service.application.ApplicationActionService;
+
+import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.beans.factory.annotation.Qualifier;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Service;
+import org.springframework.transaction.annotation.Propagation;
+import org.springframework.transaction.annotation.Transactional;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
+import java.util.stream.Collectors;
+
+@Slf4j
+@Service
+@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
+public class DistributionTaskServiceImpl extends ServiceImpl<DistributionTaskMapper, DistributionTask>
+ implements
+ DistributionTaskService {
+
+ /**
+ * Server name
+ */
+ private String serverName;
Review Comment:
It is recommended to rename to "serverId"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2337027227
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [38 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "Mrart (via GitHub)" <gi...@apache.org>.
Mrart commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2336273439
Thanks for contributing, I'm leaving a few comments,
1) We support yarn jobs?
2) We can have more test cases to protect the stability of the program. So far we have only tested consistent hashes,.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2342546468
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [44 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "HxpSerein (via GitHub)" <gi...@apache.org>.
HxpSerein commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2336558073
Thank you, @Mrart, for your review.
I fully agree with your suggestion to add more test cases to enhance the program's stability. So far, we've only tested consistent hashing, and additional tests for other scenarios will be helpful moving forward.
> We can have more test cases to protect the stability of the program. So far we have only tested consistent hashes.
This PR modifies the task distribution logic in ApplicationActionServiceImpl. For single-node scenarios, the current logic will still be executed locally. In multi-node HA scenarios, since Streampark currently doesn’t support multi-node integration testing (IT), and adding IT tests in the short term is challenging, I plan to add unit tests (UT) for logic correctness. Once the resource center is integrated, we can proceed with multi-node cluster deployment testing.
> We only support flink yarn jobs? we need supported k8s and spark jobs.
Since the resource center is not yet developed and HA is incomplete, the current testing support is somewhat limited. To ensure code stability, I propose to temporarily support only Flink tasks, with support for Kubernetes and Spark jobs added once the resource center is completed. Do you have any suggestions on this approach? @Mrart @wolfboys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "HxpSerein (via GitHub)" <gi...@apache.org>.
HxpSerein commented on code in PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#discussion_r1753082332
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -240,6 +240,19 @@ create table `t_flink_sql` (
) engine=innodb auto_increment=100000 default charset=utf8mb4 collate=utf8mb4_general_ci;
+-- ----------------------------
+-- Table structure for t_distribution_task
+-- ----------------------------
+drop table if exists `t_distribution_task`;
Review Comment:
Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature-3960] Add task distribution for streampark [incubator-streampark]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4040:
URL: https://github.com/apache/incubator-streampark/pull/4040#issuecomment-2342659076
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040) **Quality Gate passed**
Issues
 [43 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4040&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4040&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4040)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org