You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/10/21 03:23:17 UTC
[GitHub] [inlong] woofyzhao opened a new pull request, #6249: [INLONG-6248][Manager] Add sort status query plugin and api
woofyzhao opened a new pull request, #6249:
URL: https://github.com/apache/inlong/pull/6249
- Fixes #6248
### Motivation
There are some pieces missing with regard to the overall inlong task management, for example
- Sort job status is not known once after the sort job starts.
- Intermediate status such as group CONFIGURING may stuck forever if manager is killed or exit abnormally.
- The status machine is complex and may not integrate well with both standard mode or lightweight mode.
### Modification
- Add sort job status query interface.
- Add intermediate status cleanup task on manager recovery startup.
- optimize the status machine to better integrate with both standard and lightweight mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] woofyzhao commented on a diff in pull request #6249: [INLONG-6248][Manager] Add sort status query plugin interface and group status cleanup task
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #6249:
URL: https://github.com/apache/inlong/pull/6249#discussion_r1004174485
##########
inlong-manager/manager-web/src/main/resources/application-dev.properties:
##########
@@ -82,3 +82,7 @@ data.cleansing.interval.seconds=1800
data.cleansing.before.days=10
# The maximum size of data to be deleted in batch, default is 100
data.cleansing.batchSize=100
+
+# If turned on, in configuring group status will be set to failed after restart
+# Only set this to true when a single manager instance is deployed
+group.status.cleanup.enabled = false
Review Comment:
OK. Turns out the reset interface already exists. Then this task is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] woofyzhao commented on a diff in pull request #6249: [INLONG-6248][Manager] Add sort status query plugin interface and group status cleanup task
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #6249:
URL: https://github.com/apache/inlong/pull/6249#discussion_r1002859918
##########
inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/InlongClient.java:
##########
@@ -99,10 +99,11 @@ static InlongClient create(String serviceUrl, ClientConfiguration configuration)
* List group status
*
* @param groupIds inlong group id list
+ * @param credentials auth info to query sort task such as sort cluster token
* @return map of inlong group status list
* @throws Exception the exception
*/
- Map<String, InlongGroupStatusInfo> listGroupStatus(List<String> groupIds) throws Exception;
+ Map<String, InlongGroupStatusInfo> listGroupStatus(List<String> groupIds, String credentials) throws Exception;
Review Comment:
Agreed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #6249: [INLONG-6248][Manager] Add sort status query plugin interface and group status cleanup task
Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #6249:
URL: https://github.com/apache/inlong/pull/6249#discussion_r1002857824
##########
inlong-manager/manager-client/src/main/java/org/apache/inlong/manager/client/api/InlongClient.java:
##########
@@ -99,10 +99,11 @@ static InlongClient create(String serviceUrl, ClientConfiguration configuration)
* List group status
*
* @param groupIds inlong group id list
+ * @param credentials auth info to query sort task such as sort cluster token
* @return map of inlong group status list
* @throws Exception the exception
*/
- Map<String, InlongGroupStatusInfo> listGroupStatus(List<String> groupIds) throws Exception;
+ Map<String, InlongGroupStatusInfo> listGroupStatus(List<String> groupIds, String credentials) throws Exception;
Review Comment:
I suggest reserve `listGroupStatus(List<String> groupIds)` and new add `listGroupStatus(List<String> groupIds, String credentials)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow merged pull request #6249: [INLONG-6248][Manager] Add sort status query plugin interface and optimize status logic
Posted by GitBox <gi...@apache.org>.
healchow merged PR #6249:
URL: https://github.com/apache/inlong/pull/6249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6249: [INLONG-6248][Manager] Add sort status query plugin interface and group status cleanup task
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6249:
URL: https://github.com/apache/inlong/pull/6249#discussion_r1002861416
##########
inlong-dashboard/package-lock.json:
##########
@@ -4238,19 +4238,19 @@
"arrify": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/arrify/-/arrify-1.0.1.tgz",
- "integrity": "sha512-3CYzex9M9FGQjCGMGyi6/31c8GJbgb0qGyrx5HWxPd0aCwh4cB2YjMb2Xf9UuoogrMrlO9cTqnB5rI5GHZTcUA==",
+ "integrity": "sha1-iYUI2iIm84DfkEcoRWhJwVAaSw0=",
Review Comment:
Please do not commit this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #6249: [INLONG-6248][Manager] Add sort status query plugin interface and group status cleanup task
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #6249:
URL: https://github.com/apache/inlong/pull/6249#discussion_r1004169825
##########
inlong-manager/manager-web/src/main/resources/application-dev.properties:
##########
@@ -82,3 +82,7 @@ data.cleansing.interval.seconds=1800
data.cleansing.before.days=10
# The maximum size of data to be deleted in batch, default is 100
data.cleansing.batchSize=100
+
+# If turned on, in configuring group status will be set to failed after restart
+# Only set this to true when a single manager instance is deployed
+group.status.cleanup.enabled = false
Review Comment:
It is recommended to add an interface to change the group status from "in configuration" to "configuration failure" (for modify_time is x hours ago),
so as to avoid the unavailability of this configuration when multiple inlong manager applications are used,
and also more flexibly support users to call and process on demand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] woofyzhao commented on a diff in pull request #6249: [INLONG-6248][Manager] Add sort status query plugin interface and group status cleanup task
Posted by GitBox <gi...@apache.org>.
woofyzhao commented on code in PR #6249:
URL: https://github.com/apache/inlong/pull/6249#discussion_r1002862908
##########
inlong-dashboard/package-lock.json:
##########
@@ -4238,19 +4238,19 @@
"arrify": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/arrify/-/arrify-1.0.1.tgz",
- "integrity": "sha512-3CYzex9M9FGQjCGMGyi6/31c8GJbgb0qGyrx5HWxPd0aCwh4cB2YjMb2Xf9UuoogrMrlO9cTqnB5rI5GHZTcUA==",
+ "integrity": "sha1-iYUI2iIm84DfkEcoRWhJwVAaSw0=",
Review Comment:
My bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org