You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/06/20 02:28:32 UTC
[GitHub] [rocketmq] cserwen opened a new issue, #4483: [POP] Make lite pull consumer support POP mode to consume
cserwen opened a new issue, #4483:
URL: https://github.com/apache/rocketmq/issues/4483
**FEATURE REQUEST**
1. Please describe the feature you are requesting.
- Support pop mode for `DefaultLitePullConsumer`.
2. Provide any additional detail on your proposed use case for this feature.
- Currently, POP mode is only supported in Push consumer. However, in the push consumption mode, users can only passively accept messages to consume, and cannot actively control the speed of consumption (the function of limited flow is not very flexible).
- As we thought, pull consumer can achieve this purpose very well. However, in some business scenarios, there may be thousands of instances in the consumer group (which will automatically expand and shrink with traffic). If the client-side rebalance mode is still used, it will lead to unbalanced consumption traffic and frequent rebalance. So it is necessary to make pull consumption to support POP mode.
- Possible code looks like this:
```java
public static void main(String[] args) throws Exception {
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer("lite_pull_consumer_test");
litePullConsumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
litePullConsumer.subscribe("pop-topic", "*");
litePullConsumer.setNamesrvAddr("localhost:9876");
litePullConsumer.setClientRebalance(false);
litePullConsumer.start();
try {
while (running) {
List<MessageExt> messageExts = litePullConsumer.poll();
for (MessageExt messageExt : messageExts) {
System.out.printf("%s=%s\n", messageExts.size(), messageExts);
litePullConsumer.ack(messageExt);
// litePullConsumer.changePopInvisibleTime(messageExt); //retry
}
}
} finally {
litePullConsumer.shutdown();
}
}
```
3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
- should-have
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhouxinyu commented on issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
zhouxinyu commented on issue #4483:
URL: https://github.com/apache/rocketmq/issues/4483#issuecomment-1166965871
BTW, do you use the POP technology in your production environment? If yes, any feedback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhouxinyu commented on issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
zhouxinyu commented on issue #4483:
URL: https://github.com/apache/rocketmq/issues/4483#issuecomment-1166963439
Partly `Yes`, SimpleConsumer will receive messages from a proxy, usually, it's an LB, then the proxy will pop messages from the 500 machines in a round-robin manner. It's obvious that the clients don't need to send requests to all backend servers but should improve concurrency to reduce `false empty response`. In the future, the proxy could long-polling from all the backend machines to reduce client pressure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen commented on issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
cserwen commented on issue #4483:
URL: https://github.com/apache/rocketmq/issues/4483#issuecomment-1159970569
> Hi, There is no need to do that, lite pull is widely used in the streaming scenario while POP is unsuitable.
>
> For your needs, I think the new SimpleConsumer API is more suitable and could better leverage the POP ability. Please refer to [here](https://github.com/apache/rocketmq-clients/blob/master/java/client-apis/src/main/java/org/apache/rocketmq/client/apis/consumer/SimpleConsumer.java) for more information.
@zhouxinyu Got it. But it seems that it can only be used after 5.0. Looking forward to the 5.0 version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhouxinyu commented on issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
zhouxinyu commented on issue #4483:
URL: https://github.com/apache/rocketmq/issues/4483#issuecomment-1159939691
Hi, There is no need to do that, lite pull is widely used in the streaming scenario while POP is unsuitable.
For your needs, I think the new SimpleConsumer API is more suitable and could better leverage the POP ability. Please refer to [here](https://github.com/apache/rocketmq-clients/blob/master/java/client-apis/src/main/java/org/apache/rocketmq/client/apis/consumer/SimpleConsumer.java) for more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen commented on issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
cserwen commented on issue #4483:
URL: https://github.com/apache/rocketmq/issues/4483#issuecomment-1167013251
> BTW, do you use the POP technology in your production environment? If yes, any feedback?
@zhouxinyu Yes, we are already using the POP consumption function, and through some optimizations (flexible delay messages instead of level delay), it has shown a good use effect. But there is currently such a scenario that the pull consumer needs to support the POP mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen closed issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
cserwen closed issue #4483: [POP] Make lite pull consumer support POP mode to consume
URL: https://github.com/apache/rocketmq/issues/4483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] zhouxinyu commented on issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
zhouxinyu commented on issue #4483:
URL: https://github.com/apache/rocketmq/issues/4483#issuecomment-1167062257
>
Got it, please pay some attention to [SimpleConsumer](https://github.com/apache/rocketmq-clients/tree/master/java) and the implementation of server-side: https://github.com/apache/rocketmq/pull/4225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] ferrirW commented on issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by GitBox <gi...@apache.org>.
ferrirW commented on issue #4483:
URL: https://github.com/apache/rocketmq/issues/4483#issuecomment-1165579850
> Hi, There is no need to do that, lite pull is widely used in the streaming scenario while POP is unsuitable.
>
> For your needs, I think the new SimpleConsumer API is more suitable and could better leverage the POP ability. Please refer to [here](https://github.com/apache/rocketmq-clients/blob/master/java/client-apis/src/main/java/org/apache/rocketmq/client/apis/consumer/SimpleConsumer.java) for more information.
In our case, the user has 500 machines and needs to explicitly call the receive interface for logic processing.
Can SimpleConsumer achieve this effect, similar to pop ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen closed issue #4483: [POP] Make lite pull consumer support POP mode to consume
Posted by "cserwen (via GitHub)" <gi...@apache.org>.
cserwen closed issue #4483: [POP] Make lite pull consumer support POP mode to consume
URL: https://github.com/apache/rocketmq/issues/4483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org