You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/04/19 09:11:02 UTC
Slack digest for #general - 2018-04-19
2018-04-18 10:24:21 UTC - Ben Stopford: @Ben Stopford has joined the channel
----
2018-04-18 17:37:56 UTC - Igor Zubchenok: Hi @Sijie Guo
here is the broker side log:
----
2018-04-18 17:54:16 UTC - Igor Zubchenok: @Igor Zubchenok uploaded a file: <https://apache-pulsar.slack.com/files/U9T9CC2P2/FA98X3S13/-.txt|Untitled>
----
2018-04-18 19:24:39 UTC - Sijie Guo: @Igor Zubchenok checking now
+1 : Igor Zubchenok
----
2018-04-18 19:38:55 UTC - Sijie Guo: @Igor Zubchenok @Vasily Yanov I think the problem is on the pulsar admin web service. it supposes to give 307 back to client so the client can relocate the right topic owner. However I think when it tries to give 307, the pulsar web service catches the 307 and convert it to 500 :disappointed:. should be a straightforward fix and will see if we can patch 1.22.0 as well.
----
2018-04-18 19:43:17 UTC - Igor Zubchenok: So how do your pulsar instances work?
----
2018-04-18 19:44:22 UTC - Sijie Guo: good question. I am still looking through the git commits to see if it is a regression in 1.22.0
heavy_check_mark : Igor Zubchenok
----
2018-04-18 19:52:17 UTC - Sijie Guo: @Igor Zubchenok - so createSubscription is a new rest endpoint added in 1.22.0
----
2018-04-18 19:53:14 UTC - Sijie Guo: so I guess this feature was added but not used in multiple-nodes deployment.
----
2018-04-18 19:53:27 UTC - Sijie Guo: typically you don’t need explicitly create subscription
----
2018-04-18 19:53:45 UTC - Sijie Guo: when you create consumers, it implicitly creates subscriptions.
----
2018-04-18 19:54:09 UTC - Sijie Guo: so that’s probably why it wasn’t noticed when it is added to 1.22.0
----
2018-04-18 19:54:26 UTC - Sijie Guo: because most of the users prior to 1.22.0 don’t explicitly create subscription
----
2018-04-18 20:00:28 UTC - Igor Zubchenok: I see, the case is
1. I need to create a subscription without any consumer, cause I'm not ready to consume yet
2. I have multiple publishers to topic afterwards
3. I need to start consuming messages without any loss
If I skip point 1, I miss messages until a consumer (with subscription) is created, even ConsumerConfiguration has nothing about start message id, like createSubscription method has.
----
2018-04-18 20:01:35 UTC - Sijie Guo: @Igor Zubchenok let me see if there is any alternatives to help with your case without a fix
smiley : Igor Zubchenok
----
2018-04-18 20:13:10 UTC - Sijie Guo: @Igor Zubchenok :
> I need to create a subscription without any consumer, cause I’m not ready to consume yet
are you using cli tool or curl the http service directly or use the PulsarAdmin directly?
----
2018-04-18 20:14:57 UTC - Igor Zubchenok: PulsarAdmin original java client (`org.apache.pulsar:pulsar-client-admin-original:1.22.0-incubating`)
----
2018-04-18 20:15:29 UTC - Igor Zubchenok: using `org.apache.pulsar:pulsar-client-admin:1.22.0-incubating` is not possible due to #1409 issue
----
2018-04-18 20:16:52 UTC - Sijie Guo: @Igor Zubchenok :
if you are programming creating subscription using PulsarAdmin, you probably can just use PulsarClient:
PulsarClient client = …;
Consumer consumer = client.subscribe(topic, subscription);
consumer.close();
this will trigger broker create the subscription.
does that work for you?
----
2018-04-18 20:21:00 UTC - Igor Zubchenok: If I create a consumer without any message listener it will not trigger any message receiving?
----
2018-04-18 20:29:57 UTC - Sijie Guo: the broker might deliver messages if there are messages (because it is a push model). but hose messages will not be consumed by the consumer because we close it after subscribe. so effectively “subscribe” here is to exercise the creating subscription logic.
----
2018-04-18 20:31:04 UTC - Igor Zubchenok: Ok, I guess I can live with this for a while)
----
2018-04-18 20:31:13 UTC - Igor Zubchenok: Testing...
----
2018-04-18 20:34:36 UTC - Sijie Guo: yeah you can get around if you are creating subscription in a programming way. that says there is no workaround for @Vasily Yanov since he is using “curl -x PUT”. I have to fix that issue.
----
2018-04-18 20:38:50 UTC - Igor Zubchenok: I suspiciously noted this workaround is much faster on a single node cluster
so there are my tests for 1000 subscriptions (my ping to node is 50ms):
14/12/12 seconds with consumer create and close
21/22/22 seconds with admin.create subscription
----
2018-04-18 20:42:41 UTC - Sijie Guo: are you creating subscriptions on same topic? or different topics?
----
2018-04-18 20:44:02 UTC - Sijie Guo: can you show me your code if possible?
----
2018-04-18 20:45:14 UTC - Igor Zubchenok: different topics
----
2018-04-18 20:46:41 UTC - Igor Zubchenok: I have implemented RxJava wrappers around pulsar client and admin, but if you need I can prepare for you equivalent unit test.
----
2018-04-18 20:47:59 UTC - Sijie Guo: oh no worries then.
----
2018-04-18 20:49:21 UTC - Igor Zubchenok: So I make subscriptions to different topics.
It will be easy for me and I'll be happy to create a unit test if it can help you with making Pulsar better!
----
2018-04-18 20:51:36 UTC - Sijie Guo: I think the different is admin.createSubscription is using http protocol, which I think each createSubscription is creating a new connection and close it. (If I remember correctly, http/1.0 doesn’t support multiplexing). while create consumer/close is binary protocol over tcp connection, so connection is multiplexed for different commands.
----
2018-04-18 20:54:30 UTC - Igor Zubchenok: This make sense. I would think PulsarAdmin should be better or some features should be moved to client.
----
2018-04-18 20:54:50 UTC - Sijie Guo: yeah
----
2018-04-18 20:55:05 UTC - Sijie Guo: agreed we will look more into how to improve pulsar admin
----
2018-04-18 20:55:48 UTC - Igor Zubchenok: :slightly_smiling_face: So... Should I file an issue(s) to github about something?
----
2018-04-18 20:57:47 UTC - Igor Zubchenok: Suggestions:
1. Bug with redirect to proper master
2. Moving pulsar admin to protobuf protocol
3. Adding start with earliest message id for a subscription when a consumer is created for a first time?
----
2018-04-18 20:58:58 UTC - Sijie Guo: @Igor Zubchenok yes please.
for 3), I think it is already implemented in master and will be available in upcoming 2.0
----
2018-04-18 20:59:11 UTC - Sijie Guo: I am actually working on 1.
----
2018-04-18 20:59:23 UTC - Sijie Guo: but if you are filing an issue, that’s great :slightly_smiling_face:
----
2018-04-18 20:59:26 UTC - Igor Zubchenok: Will do for 1 and 2. Will drop links here.
----
2018-04-18 21:01:43 UTC - Igor Zubchenok: One more question, how do you think, will I have a performance improvement (more like than 1000/12sec with ping 50ms) on creating topics with subscriptions when I have a cluster with more nodes (3,10,100)? Same when I have more client instances?
Is it scaleable?
----
2018-04-18 21:03:20 UTC - Sijie Guo: @Igor Zubchenok:
you probably can use subscribeAsyc to parallel your creation. so that you don’t need to do one by one.
e.g.
```
----
2018-04-18 21:03:31 UTC - Sijie Guo: for (int i = 0; i < 100; i++) {
}
----
2018-04-18 21:03:34 UTC - Igor Zubchenok: I do use async everywhere. Already.
----
2018-04-18 21:03:48 UTC - Sijie Guo: typed too fast …
see_no_evil : Igor Zubchenok
----
2018-04-18 21:05:39 UTC - Sijie Guo: CompletableFuture[] futures = new CompletableFuture[100];
For (int i = 0; I < 100; I++) {
Futures = client.subscribeAsyc(“topic-” + I, “sub-” + i);
}
CompletableFuture.allOf(futures).join();
----
2018-04-18 21:05:42 UTC - Sijie Guo: something like this
----
2018-04-18 21:06:37 UTC - Igor Zubchenok: I totally understand, this is 'basics' actually. I already use async methods everywhere.
----
2018-04-18 21:06:55 UTC - Igor Zubchenok: 1000/12sec is in async mode, not one-by-one.
----
2018-04-18 21:07:03 UTC - Sijie Guo: oh
----
2018-04-18 21:07:20 UTC - Sijie Guo: sorry I misunderstood
----
2018-04-18 21:07:32 UTC - Igor Zubchenok: No problem.
----
2018-04-18 21:08:18 UTC - Sijie Guo: it is interesting, it is probably bound on metadata updates. will have to look into more. do you mind filing an issue for the performance question? so I can remember to look into it later.
----
2018-04-18 21:08:53 UTC - Igor Zubchenok: My guess that ZooKeeper could be a bottle neck in such case.
----
2018-04-18 21:09:34 UTC - Sijie Guo: yes. but 1000 request - 12 sec. that is too suspicious
----
2018-04-18 21:10:32 UTC - Igor Zubchenok: Ok! I'll file an issue for that too with a unit test for reproducing.
----
2018-04-18 21:11:55 UTC - Igor Zubchenok: And I'll run my performance unit test on a server in the same datacenter where singlenode and multinode Pulsars are located. (currently I run tests remotely from my pc)
----
2018-04-18 21:12:34 UTC - Matteo Merli: > I think the different is admin.createSubscription is using http protocol, which I think each createSubscription is creating a new connection and close it. (If I remember correctly, http/1.0 doesn’t support multiplexing). while create consumer/close is binary protocol over tcp connection, so connection is multiplexed for different commands.
The multiple requests should be using HTTP/1.1 with Keep-Alive
----
2018-04-18 21:24:22 UTC - Igor Zubchenok: I'll prepare a unit test for you guys and fill an issue.
----
2018-04-18 21:35:41 UTC - Sijie Guo: @Igor Zubchenok fyi, I have sent a fix out for the redirect issue and added an integration test for create subscription as well to ensure we don’t have any regression in future.
+1 : Igor Zubchenok
----
2018-04-18 21:37:43 UTC - Igor Zubchenok: Do you plan to have a 1.22.1+ release?
----
2018-04-18 21:50:35 UTC - Sijie Guo: will raise a discussion to apache mail list.
----
2018-04-18 22:44:50 UTC - Igor Zubchenok: The issue with PulsarAdmin.createSubscription:
<https://github.com/apache/incubator-pulsar/issues/1615>
----
2018-04-18 22:59:12 UTC - Igor Zubchenok: CreateSubscription performance issue:
<https://github.com/apache/incubator-pulsar/issues/1616>
----
2018-04-18 22:59:31 UTC - Igor Zubchenok: good job!
----
2018-04-19 02:40:49 UTC - Chris Tava: @Chris Tava has joined the channel
----
2018-04-19 07:12:58 UTC - Sijie Guo: @Igor Zubchenok thank you for creating the issues
----