You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/06/19 10:58:49 UTC
[GitHub] [pulsar] newur opened a new pull request #10982: Replace endless while loops with MessageListner
newur opened a new pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982
### Motivation
Using a MessageListener is simpler than using a endless while loop. The loop might work for super basic hello-world examples, but would trap any other app with its endless nature. Also the loop might motivate users to move the while loop into an own, custom build thread, which is also unnecessary complicated.
### Modifications
Replace the `while` loop with a `MessageListener`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-868123320
@BewareMyPower @MarvinCai could you please take a look at the new changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Add MessageListener example in addition to endless while loop
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-870460479
This branch is behind master, so it does not know 2.8.1. I can do a rebase or create a new PR, whatever you would prefer @Anonymitaet
/update, okay I just did the rebase to keep the change in a single PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865903329
@BewareMyPower in my opinion this is a very different and more advanced use case that supports your claim. I read the current doc in the spirit of 'Look, here is a basic example how you can fetch messages one by one'.
But the example has the major drawback that it traps the app (main thread) so it can never do anything else than receiving messages. This is a limitation that goes far beyond what I would call reasonable. The user will most likely want its app to be not trapped (like I would like my SpringBoot/Quarkus/whatever app to finish the normal startup).
So the listener approach is similar complex to read/write compared to the current docu, without the trap thingy.
```
final List<String> messages = new ArrayList<>();
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionName("my-sub")
.topic("my-topic")
.subscribe();
for (int i = 0; i < 10; i++) {
// NOTE: If you're using receive() with timeout, you must check null for timeout
messages.add(consumer.receive().getValue());
}
messages.forEach(System.out::println);
consumer.close();
```
If I read that right this `receive` call is blocking, so either I get a timeout or the app is stuck till the message arrives. Again I would never want/expect that behavior in my app. Plus, this use case is on a totally different level of complexity, compared to what is currently done.
Maybe you can outline a bit more when the current `while` loop would be desirable. I totally fail to see any use case where I would prefer to have my main thread blocked while consuming messages one by one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-870424895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet merged pull request #10982: [docs] Add MessageListener example in addition to endless while loop
Posted by GitBox <gi...@apache.org>.
Anonymitaet merged pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865903329
@BewareMyPower in my opinion this is a very different and more advanced use case that supports your claim. I read the current doc in the spirit of 'Look, here is a basic example how you can fetch messages one by one'.
But the example has the major drawback that it traps the app (main thread) so it can never do anything else than receiving messages. This is a limitation that goes far beyond what I would call reasonable. The user will most likely want its app to be not trapped (like I would like my SpringBoot/Quarkus/whatever app to finish the normal startup).
So the listener approach is similar complex to read/write to current one, without the trap thingy.
```
final List<String> messages = new ArrayList<>();
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionName("my-sub")
.topic("my-topic")
.subscribe();
for (int i = 0; i < 10; i++) {
// NOTE: If you're using receive() with timeout, you must check null for timeout
messages.add(consumer.receive().getValue());
}
messages.forEach(System.out::println);
consumer.close();
```
If I read that right this `receive` call is blocking, so either I get a timeout or the app is stuck till the message arrives. Again I would never want/expect that behavior in my app. Plus, this use case is on a totally different level of complexity, compared to what is currently done.
Maybe you can outline a bit more when the current `while` loop would be desirable. I totally fail to see any use case where I would prefer to have my main thread blocked while consuming messages one by one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865464250
Yes, all versioned docs are stored in [pulsar/site2/website/versioned_docs/](https://github.com/apache/pulsar/tree/master/site2/website/versioned_docs). You can "batch and replace" all occurrences with just a few clicks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865973755
I think `receive` and `messageListener` are still different pattern, with former one you decide when to proactively pull and act on the message, the later one is like Lambda just got triggered on every message dispatched to consumer. They are both useful in different scenario.
Even if we replace `receive` example with a `messageListener` one, someone might still try to copy & paste that code snippet without knowing how it works under the hood and might not suit their use case. Which IMO is no better than a receive blocking their main thread.
Just my two cents, I think we can provide basic example (simple main example with receive) as well as advanced example (messageListener) to just showcase user what they can do with Pulsar API(not how you can create enterprise-grade production-ready application with Pulsar). And if we really want to prevent user from using example as _real_ production code we should addd explicit warning.
If we really want to show some best practices the community is using when building application with Pulsar we can add a new section with real world code snippet of how Pulsar API is used in production to produce/consume message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865903329
@BewareMyPower in my opinion this is a very different and more advanced use case that supports your claim. I read the current doc in the spirit of 'Look, here is a basic example how you can fetch messages one by one'.
But the example has the major drawback that it traps the app (main thread) so it can never do anything else than receiving messages. This is a limitation that goes far beyond what I would call reasonable. The user will most likely want its app to be not trapped (like I would like my SpringBoot/Quarkus/whatever app to finish the normal startup).
So the listener approach is similar complex to read/write to current one, without the trap thingy.
```
final List<String> messages = new ArrayList<>();
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionName("my-sub")
.topic("my-topic")
.subscribe();
for (int i = 0; i < 10; i++) {
// NOTE: If you're using receive() with timeout, you must check null for timeout
messages.add(consumer.receive().getValue());
}
messages.forEach(System.out::println);
consumer.close();
```
If I read that right this `receive` call is blocking, so either I get a timeout or the app is stuck till the message arrives. Again I would never want/expect that behavior my app. Plus, this use case is on a totally different level of complexity, compared to what is currently done.
Maybe you can outline a bit more when the current `while` loop would be desirable. I totally fail to see any use case where I would prefer to have my main thread blocked while consuming messages one by one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865246513
@Anonymitaet the versioned docs are the content in `site2/website/versioned_docs`?
Sure I would update them when the change gets approval. How far back do you normally port the docu back? There are quite some :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-869893130
I updated the versioned docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-869893130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865973755
I think `receive` and `messageListener` are still different pattern, with former one you decide when to proactively pull and act on the message, the later one is like Lambda just got triggered on every message dispatched to consumer. They are both useful in different scenario.
Even if we replace `receive` example with a `messageListener` one, someone might still try to copy & paste that code snippet without knowing how it works under the hood and might not suit their use case. Which IMO is no better than a receive blocking their main thread.
Just my two cents, I think we can provide basic example (simple main example with receive) as well as advanced example (messageListener) to just showcase user what they can do with Pulsar API(not how you create enterprise-grade production-ready application with Pulsar). And if we really want to prevent user from using example as _real_ production code we should addd explicit warning.
If we want to show some best practices the community is using when building application with Pulsar we can add a new section with real world code snippet of how Pulsar API is used in production to produce/consume message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
BewareMyPower edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865685162
@newur It's a synchronous vs. asynchronous compare. The synchronous way that uses `receive` method is easier to use for newbies. I've seen many newbies that tried to use the asynchronous way that sets the listener while they didn't realize that they should do something in the main thread.
```java
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.subscriptionName("my-sub")
.topic("my-topic")
.messageListener(new MessageListener<String>() {
@Override
public void received(Consumer<String> consumer, Message<String> msg) {
System.out.println("Receive " + msg.getValue() + " from " + msg.getMessageId());
}
})
.subscribe();
// TODO: do something else
// If you don't do anything to block here, the consumer will close immediately.
consumer.close();
```
Some newbies might think when they set the listener, the consumer will block until some state. You may think they're stupid, but the "stupid" guys exist. (Just want to say not everyone is good at the asynchronous programming)
Asynchronous programming is more complicated than synchronous programming. Given a case that you need to collect 10 messages to a list and then close the consumer. Using `receive` is easy.
Using listener is somehow complicated:
```java
final CountDownLatch latch = new CountDownLatch(10);
final List<String> messages = Collections.synchronizedList(new ArrayList<>()); // [1]
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.subscriptionName("my-sub")
.topic("my-topic")
.messageListener(new MessageListener<String>() {
@Override
public void received(Consumer<String> consumer, Message<String> msg) {
if (messages.size() < 10) { // [2]
messages.add(msg.getValue());
consumer.acknowledgeAsync(msg); // just to avoid catching the exception
}
latch.countDown();
}
})
.subscribe();
latch.await(); // [3]
messages.forEach(System.out::println); // Users may do something more complicated here
consumer.close();
```
1. You should use a thread-safe container because you'll access these messages in both main thread and listener thread.
2. Since you cannot control whether to stop calling the listener, you need to check the count of received messages in case the consumer received more than 10 messages internally.
3. You need an extra component like `CountDownLatch` for the synchronization between the main thread and the listener thread.
While using `receive` method is much easier:
```java
final List<String> messages = new ArrayList<>();
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
.subscriptionName("my-sub")
.topic("my-topic")
.subscribe();
for (int i = 0; i < 10; i++) {
// NOTE: If you're using receive() with timeout, you must check null for timeout
messages.add(consumer.receive().getValue());
}
messages.forEach(System.out::println);
consumer.close();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
BewareMyPower edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865920717
IMO, in short, the `MessageListener` should be the advanced API that's used for efficiency if users prefer asynchronous API. But the `receive` method should be the easiest API that's suitable for basic example.
> If I read that right this receive call is blocking, so either I get a timeout or the app is stuck till the message arrives.
Not completely right. If there're already messages that are cached in the consumer, the `receive` method will return immediately. It looks like you strongly dislike the synchronous `receive` API that may block the current thread. However, let's look at Kafka's API design: https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/consumer/Consumer.html#poll(java.time.Duration)
It provides the synchronous API (`poll()` method) directly. What's different from Pulsar's `receive()` API is that Kafka's `poll()` returns a container of messages while Pulsar's `receive()` returns a single message. However, Pulsar's API design just avoids collecting the messages into a container. If there're some messages that are cached, the client will only fetch a single message from the cache and pass it to user side. Therefore, in essential they're similar.
- Kafka: fetch N messages (block main thread for some time), collect to a list and return the list to the user.
- Pulsar: fetch N messages (block main thread for some time), users peek messages one by one.
> I totally fail to see any use case where I would prefer to have my main thread blocked while consuming messages one by one.
Maybe it's right to you. But lots of users uses Kafka's synchronous `poll()` API at the same time. You can also see the example in Kafka's official website: https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
```java
while (true) { // infinite loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); // block main thread for some time
for (ConsumerRecord<String, String> record : records) // peek messages one by one
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
```
Maybe some Kafka client's wrapper like SpringBoot will provide a listener like API that has some overhead. In this case, Pulsar provides the `MessageListener` that can avoid users from wrapping a listener in another thread again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Add MessageListener example in addition to endless while loop
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-870486266
@newur thanks for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865942441
Looks like we agree to disagree on this topic.
For me 'Kafka is doing the same' is not a use case and I hope Pulsar strive to provide a better user experience than Kafka. I would even assume (of course I can not prove that point) that Kafka is in kind of an advantage here, because all major frameworks provide an integration that abstracts this problem fully away (with annotations and the like). As long as pulsar does not have that, we need to develop other strategies, like better docu. :)
Anyway, I would like to follow your suggestion and just have both cases in the docu. Not sure if I have time today, so probably later this week I can provide another proposal.
> It looks like you strongly dislike the synchronous receive API that may block the current thread.
I have yet to meet a developer that likes his/her main thread to be block. :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865634749
@BewareMyPower what characterizes the difference? What value does it add to keep the `while` example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865246513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865973755
I think `receive` and `messageListener` are still different pattern, with former one you decide when to proactively pull and process the message, the later one is like Lambda just got triggered on every message dispatched to consumer. They are both useful in different scenario.
Even if we replace `receive` example with a `messageListener` one, someone might still try to copy & paste that code snippet without knowing how it works under the hood and might not suit their use case. Which IMO is no better than a receive blocking their main thread.
Just my two cents, I think we can provide basic example (simple main example with receive) as well as advanced example (messageListener) to just showcase user what they can do with Pulsar API(not how you can create enterprise-grade production-ready application with Pulsar). And if we really want to prevent user from using example as _real_ production code we should addd explicit warning.
If we really want to show some best practices the community is using when building application with Pulsar we can add a new section with real world code snippet of how Pulsar API is used in production to produce/consume message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865973755
I think `receive` and `messageListener` are still different pattern, with former one you decide when to proactively pull and act on the message, the later one is like Lambda just got triggered on every message dispatched to consumer. They are both useful in different scenario.
Even if we replace `receive` example with a `messageListener` one, someone might still try to copy & paste that code snippet without knowing how it works under the hood and might not suit their use case. Which IMO is no better than a receive blocking their main thread.
Just my two cents, I think we can provide basic example (simple main example with receive) as well as advanced example (messageListener) to just showcase user what they can do with Pulsar API(not how you can create enterprise-grade production-ready application with Pulsar). And if we really want to prevent user from using example as _real_ production code we should addd explicit warning.
If we want to show some best practices the community is using when building application with Pulsar we can add a new section with real world code snippet of how Pulsar API is used in production to produce/consume message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-866143222
> If we really want to show some best practices the community is using when building application with Pulsar we can add a new section with real world code snippet of how Pulsar API is used in production to produce/consume message.
That sounds awesome. I would assume their is a high demand for such a section. What would you consider to be useful measures to create and populate such a section? @MarvinCai
(I cannot really imagine the kind of circumstances where a non-enterprise person/group even dares to deal with pulsar/kafka and such stuff. Not saying that this does not happen. :D)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet merged pull request #10982: [docs] Add MessageListener example in addition to endless while loop
Posted by GitBox <gi...@apache.org>.
Anonymitaet merged pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865246513
@Anonymitaet the versioned docs are the content in `site2/website/versioned_docs`?
Sure I would update them when the change gets approval. How far back do you normally port the docu? There are quite some :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-869256461
@newur thanks for your contribution. Does this affect only master or other versioned docs?
If latter, could you please help update all affected versions? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-864704279
@newur thanks for your contribution. Does this affect only master or other versioned docs?
If latter, could you please help update all affected versions? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865472623
Using `receive`/`receiveAsync` and using `MessageListener` are two different ways for consumer, I think it's better to add a section for the example code that uses `MessageListener`, not just replace the `receive` way with `MessageListener`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865472623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865464250
Yes, all versioned docs are stored in [pulsar/site2/website/versioned_docs/](https://github.com/apache/pulsar/tree/master/site2/website/versioned_docs). You can "batch and replace" all occurrences with just a few clicks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-866143222
> If we really want to show some best practices the community is using when building application with Pulsar we can add a new section with real world code snippet of how Pulsar API is used in production to produce/consume message. @MarvinCai
That sounds awesome. I would assume their is a high demand for such a section. What would you consider to be useful measures to create and populate such a section?
(I cannot really imagine the kind of circumstances where a non-enterprise person/group even dares to deal with pulsar/kafka and such stuff. Not saying that this does not happen. :D)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-870424895
> I updated the versioned docs. @BewareMyPower @Anonymitaet
@newur thanks, 2.8.1 doc does not need to be updated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-869600794
@newur FYI, https://github.com/apache/pulsar/pull/10988 is an example. The documents of older pulsar versions are under `site2/website/versioned_docs` directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865920717
IMO, in short, the `MessageListener` should be the advanced API that's used for efficiency if users prefer asynchronous API. But the `receive` method should be the easiest API that's suitable for basic example.
> If I read that right this receive call is blocking, so either I get a timeout or the app is stuck till the message arrives.
Not completely right. If there're already messages that are cached in the consumer, the `receive` method will return immediately. It looks like you strongly dislike the synchronous `receive` API that may block the current thread. However, let's look at Kafka's API design: https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/consumer/Consumer.html#poll(java.time.Duration)
It provides the synchronous API (`poll()` method) directly. What's different from Pulsar's `receive()` API is that Kafka's `poll()` returns a container of messages while Pulsar's `receive()` returns a single message. However, Pulsar's API design just avoids collecting the messages into a container. If there're some messages that are cached, the client will only fetch a single message from the cache and pass it to user side. Therefore, in essential they're similar.
- Kafka: fetch N messages (block main thread for some time), collect to a list and return the list to the user.
- Pulsar: fetch N messages (block main thread for some time), users peek messages one by one.
> I totally fail to see any use case where I would prefer to have my main thread blocked while consuming messages one by one.
Maybe it's right to you. But lots of users uses Kafka's synchronous `poll()` API at the same time. You can also see the example in Kafka's official website: https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
```java
while (true) { // infinite loop
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); // block main thread for some time
for (ConsumerRecord<String, String> record : records) // peek messages one by one
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
```
Maybe some Kafka client's wrapper like SprintBoot will provide a listener like API that has some overhead. In this case, Pulsar provides the `MessageListener` that can avoid users from wrapping a listener in another thread again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865942441
Looks like we agree to disagree on this topic.
For me 'Kafka is doing the same' is not a use case and I hope Pulsar strive to provide a better user experience than Kafka. I would even assume (of course I can not prove that point) that Kafka is in kind of an advantage here, because all major frameworks provide an integration that abstracts this problem fully away (with annotations and the like). As long as pulsar does not have that, we need to develop other strategies, like better docu. :)
Anyway, I would like to follow the suggestion you made and just have both cases in the docu. Not sure if I have time today, so probably later this week I can provide another proposal.
> It looks like you strongly dislike the synchronous receive API that may block the current thread.
I have yet to meet a developer that likes his main thread to be block. :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#discussion_r655055855
##########
File path: site2/docs/client-libraries-java.md
##########
@@ -243,16 +243,14 @@ Once you've instantiated a {@inject: javadoc:PulsarClient:/client/org/apache/pul
Consumer consumer = client.newConsumer()
.topic("my-topic")
.subscriptionName("my-subscription")
+ .messageListener(myMessageListener)
.subscribe();
```
-The `subscribe` method will auto subscribe the consumer to the specified topic and subscription. One way to make the consumer listen on the topic is to set up a `while` loop. In this example loop, the consumer listens for messages, prints the contents of any received message, and then [acknowledges](reference-terminology.md#acknowledgment-ack) that the message has been processed. If the processing logic fails, you can use [negative acknowledgement](reference-terminology.md#acknowledgment-ack) to redeliver the message later.
+The `subscribe` method will auto subscribe the consumer to the specified topic and subscription. One way to make the consumer listen on the topic is to add a `MessageListener`. In this example, the consumer listens for messages, prints the contents of any received message, and then [acknowledges](reference-terminology.md#acknowledgment-ack) that the message has been processed. If the processing logic fails, you can use [negative acknowledgement](reference-terminology.md#acknowledgment-ack) to redeliver the message later.
Review comment:
@BewareMyPower could you please help review? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-869893130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListner
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865246513
@Anonymitaet the versioned docs are the content in `site2/website/versioned_docs`?
Sure I would update them when the change gets approval. How far back do you normally port the docu? There are quite some :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865942441
Looks like we agree to disagree on this topic.
For me 'Kafka is doing the same' is not a use case and I hope Pulsar strive to provide a better user experience than Kafka. I would even assume (of course I can not prove that point) that Kafka is in kind of an advantage here, because all major frameworks provide an integration that abstracts this problem fully away (with annotations and the like). As long as pulsar does not have that, we need to develop other strategies, like better docu. :)
Anyway, I would like to follow your suggestion and just have both cases in the docu. Not sure if I have time today, so probably later this week I can provide another proposal.
> It looks like you strongly dislike the synchronous receive API that may block the current thread.
I have yet to meet a developer that likes his main thread to be block. :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-870460479
This branch is behind master, so it does not know 2.8.1. I can do a rebase or create a new PR, whatever you would prefer @Anonymitaet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur edited a comment on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur edited a comment on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-869893130
I updated the versioned docs. @BewareMyPower @Anonymitaet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] newur commented on pull request #10982: [docs] Replace endless while loops with MessageListener
Posted by GitBox <gi...@apache.org>.
newur commented on pull request #10982:
URL: https://github.com/apache/pulsar/pull/10982#issuecomment-865635305
> You can "batch and replace" all occurrences with just a few clicks.
Can you please tell me more about this? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org