You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@inlong.apache.org by GitBox <gi...@apache.org> on 2022/11/02 02:32:19 UTC
[GitHub] [inlong] gosonzhang opened a new issue, #6364: [Improve][DataProxy] Add DataProxy node load information
gosonzhang opened a new issue, #6364:
URL: https://github.com/apache/inlong/issues/6364
### Description
When the system is running, the SDK will pull the list of DataProxy node addresses of the cluster from the Manager, and select several DataProxy from the list for data reporting processing. With the operation of the system, DataProxy's load will be changed due to changes in the number of connected SDK connections and the amount of messages sent by the business. It is possible that the load of DataProxy will become worse and cannot provide external services, or the load becomes lower which no SDK connecting, and it's service capability cannot be effectively exerted.
In order to solve this problem, I want to add the load information to the DataProxy node, and expose this load information to the Manager and SDK:
1. DataProxy regularly counts its own load, and reports the load information to the Manager through heartbeat;
2. The Manager updates stores the load information reported by each DataProxy, and provides it to the SDK when the SDK requests the DataProxy information;
3. Based on the DataProxy IP list and load information provided by the Manager, the SDK randomly selects the remaining DataProxy nodes after excluding the Top N DataProxy nodes with excessive load, and performs data transmission services; the SDK periodically obtains the DataProxy ip and load information from the Manager;
4. After the SDK establishes a connection with DataProxy, it decides whether to switch DataProxy for data reporting through the load information returned by DataProxy in the heartbeat response;
5. The SDK operates according to 3, when selecting a new DataProxy to establish a connection to report data, avoid the DataProxy node of Top N with excessive load.
The whole idea is similar to the following picture:
Let's see if it is OK, if there is no problem, I will try to complete the processing of this part.
### InLong Component
InLong Manager, InLong DataProxy, InLong SDK
### Are you willing to submit PR?
- [X] Yes, I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] github-actions[bot] commented on issue #6364: [Improve][DataProxy] Add DataProxy node load information
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6364:
URL: https://github.com/apache/inlong/issues/6364#issuecomment-1368603999
This issue is stale because it has been open for 60 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gosonzhang commented on issue #6364: [Improve][DataProxy] Add DataProxy node load information
Posted by "gosonzhang (via GitHub)" <gi...@apache.org>.
gosonzhang commented on issue #6364:
URL: https://github.com/apache/inlong/issues/6364#issuecomment-1713483008
The issue is finished and closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gosonzhang closed issue #6364: [Improve][DataProxy] Add DataProxy node load information
Posted by "gosonzhang (via GitHub)" <gi...@apache.org>.
gosonzhang closed issue #6364: [Improve][DataProxy] Add DataProxy node load information
URL: https://github.com/apache/inlong/issues/6364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org