You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/04/06 12:55:17 UTC
[GitHub] [skywalking] chenhaipeng opened a new pull request, #8828: ADD kafka consumers number default = 1,enhance consumption capacity
chenhaipeng opened a new pull request, #8828:
URL: https://github.com/apache/skywalking/pull/8828
### Add kafka consumers number
- [x] Update the documentation to include this new feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1095950155
> I am not familiar with today's Kafka now, please correct me if I am wrong.
>
> When you use multiple OAP servers, and every server has multiple consumers for one topic, do you need any extra configurations?
>
> And how the consumers(if multiple) are assigned to different topics?
The configuration in kafka-fetcher works on multiple topics. I don't have any special configuration for individual topics online, but there is a configuration enableTopics (not in this PR) to control whether a topic is enabled, and the other is my online The characteristics of the modified kafka-fetcher, the link becomes OAP->MQ->OAP, the purpose is to reduce the configuration of kafka on the agent side, and put the control of kafka on the platform side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng merged PR #8828:
URL: https://github.com/apache/skywalking/pull/8828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1092742527
I am not talking about this PR, I want to make this to make Kafka fetchers work better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1098683177
> @chenhaipeng I merged that one, if your proposal still makes sense, please update.
I have updated to support multiple instances of consumers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1090273636
`application.yml`, `configuration-vocabulary.md`, `changes.md` should be updated as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1095951318
> I am not talking about this PR, I want to make this to make Kafka fetchers work better.
The configuration in kafka-fetcher works on multiple topics. I don't have any special configuration for individual topics online, but there is a configuration enableTopics (not in this PR) to control whether a topic is enabled, and the other is my online The characteristics of the modified kafka-fetcher, the link becomes OAP->MQ->OAP, the purpose is to reduce the configuration of kafka on the agent side, and put the control of kafka on the platform side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1092699899
I am not familiar with today's Kafka now, please correct me if I am wrong.
When you use multiple OAP servers, and every server has multiple consumers for one topic, do you need any extra configurations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1092567105
> > @chenhaipeng Hi, I notice we need to configure the `consumePartitions` on cluster mode of OAP. That is very unfriendly. Would you like to improve the feature like this PR?
>
> consumePartitions is original feature,this feature only add consumer instances,there is no change in any of the original features
The point here I think is, this may be not needed anymore due to multiple consumers are available. That thing was added to manually set consumer to avoid duplicated consuming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1100701538
> @chenhaipeng e2e tests for Kafka receivers fail, please check the codes locally, and test.
I've fixed the bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1092693896
> > > @chenhaipeng Hi, I notice we need to configure the `consumePartitions` on cluster mode of OAP. That is very unfriendly. Would you like to improve the feature like this PR?
> >
> >
> > consumePartitions is original feature,this feature only add consumer instances,there is no change in any of the original features
>
> The point here I think is, this may be not needed anymore due to multiple consumers are available. That thing was added to manually set consumer to avoid duplicated consuming.
This is indeed a problem. The two configuration parameters, consumePartitions and consumers, are a bit conflicting. Maybe I didn't enable isSharding and consumePartitions in the production environment, so I didn't find this problem. So how to solve this problem is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1098922138
@chenhaipeng e2e tests for Kafka receivers fail, please check the codes locally, and test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1098683412
@dmsolr PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1090255480
> So, generally, you want an OAP server having multiple consumers?
It depends on the speed of producers and consumers. If MQ accumulates, increase the number of consumers and reduce delay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1092711558
>
> I am not familiar with today's Kafka now, please correct me if I am wrong.
>
> When you use multiple OAP servers, and every server has multiple consumers for one topic, do you need any extra configurations?
>
> And how the consumers(if multiple) are assigned to different topics?
> I am not familiar with today's Kafka now, please correct me if I am wrong.
>
> When you use multiple OAP servers, and every server has multiple consumers for one topic, do you need any extra configurations?
>
> And how the consumers(if multiple) are assigned to different topics?
no need,this feature is added just to enhance the consumption power of a single OAP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1098126644
@chenhaipeng I merged that one, if your proposal still makes sense, please update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] dmsolr commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
dmsolr commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1096451566
I have submitted a PR to simplify kafka configuration in cluster mode.
For this PR, could you share more details of your case? Or had you met any problems in the production environment? @chenhaipeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1096460145
Please follow https://github.com/apache/skywalking/pull/8857, I hope that could be landed first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1095957528
@dmsolr Could you take a step in involved in this? I feel like @chenhaipeng seems to misunderstand my point.
To @dmsolr @chenhaipeng, I don't want simply change this fetcher in this way, I want the consumer could be better organized. Could we have a discussion here about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] dmsolr commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
dmsolr commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1091447394
@chenhaipeng Hi, I notice we need to configure the `consumePartitions` on cluster mode of OAP. That is very unfriendly. Would you like to improve the feature like this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Superskyyy commented on a diff in pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
Superskyyy commented on code in PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#discussion_r853434124
##########
docs/en/setup/backend/configuration-vocabulary.md:
##########
@@ -215,6 +215,7 @@ core|default|role|Option values: `Mixed/Receiver/Aggregator`. **Receiver** mode
| - | - | groupId | A unique string that identifies the consumer group to which this consumer belongs.| - | skywalking-consumer |
| - | - | createTopicIfNotExist | If true, this creates Kafka topic (if it does not already exist). | - | true |
| - | - | partitions | The number of partitions for the topic being created. | SW_KAFKA_FETCHER_PARTITIONS | 3 |
+| - | - | consumers | The number of consumers create.| SW_KAFKA_FETCHER_CONSUMERS | 1 |
Review Comment:
```suggestion
| - | - | consumers | The number of consumers to create. | SW_KAFKA_FETCHER_CONSUMERS | 1 |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1098683357
> @chenhaipeng I merged that one, if your proposal still makes sense, please update.
I have updated to support multiple instances of consumers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1091326865
@chenhaipeng A question, how do these consumers avoid consuming duplicated messages? I am away from using Kafka for a very long time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1090239688
So, generally, you want an OAP server having multiple consumers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1092565414
> @chenhaipeng Hi, I notice we need to configure the `consumePartitions` on cluster mode of OAP. That is very unfriendly. Would you like to improve the feature like this PR?
consumePartitions is original feature,this feature only add consumer instances,there is no change in any of the original features
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] chenhaipeng commented on pull request #8828: ADD kafka consumers number default = 1,enhance consumption capacity
Posted by GitBox <gi...@apache.org>.
chenhaipeng commented on PR #8828:
URL: https://github.com/apache/skywalking/pull/8828#issuecomment-1092563441
> @chenhaipeng A question, how do these consumers avoid consuming duplicated messages? I am away from using Kafka for a very long time.
consuming duplicated messages，Depends on different groupId，this feature only add consumer instances,will not cause duplicate messages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org