You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/08/29 09:11:02 UTC
Slack digest for #general - 2018-08-29
2018-08-28 13:35:53 UTC - Daniel Root: Thanks! I'll give it a try.
----
2018-08-28 16:21:17 UTC - Jon Bock: @William: It’s certainly the most common question that comes up with any new technology--older technologies almost always have a bigger ecosystem simply because they’ve been in the market longer and as a result are used by more people. But there is certainly a long list of examples of when new technology was more compelling than the established technologies and as a result ultimately gained adoption and momentum even with a smaller ecosystem.
The real question is not what technology has the biggest ecosystem (in which case everyone should have just continued to use Oracle and DB2 databases for all data storage and querying, for example), but which is the best solution for what you’re trying to do. If the scenario for which any technology was originally architected and designed isn’t like your scenario, the size of the ecosystem might not be relevant.
bananadance : Yuvaraj Loganathan, Karthik Ramasamy
----
2018-08-28 16:21:24 UTC - Jon Bock: Examples of scenarios where Pulsar shines include when your use case needs high throughput and consistent low latency, when topic scalability is important, where you need both pub-sub and message queue support, where you need to deploy in a small footprint (e.g. edge), where data durability is important, where unbounded stream storage (at low cost) is useful, where lightweight processing of data flowing through is useful, and where operational burdens need to be low. Other solutions like Kafka and PubNub were designed for different scenarios, you’ll be best at assessing what your own requirements are.
----
2018-08-28 16:23:28 UTC - Jon Bock: Pulsar and its ecosystem continue to grow rapdily, and because it has less technical debt than technologies that have been around for a number of years it can still evolve rapidly--there have been a lot of great features added in just the last 4 months, for example, including a number of connectors.
----
2018-08-28 16:52:51 UTC - William: @Jon Bock that is FANTASTIC thanks. My understanding of Pulsar was that it’s like Kafka, but with namespacing and georeplication as more primary players.
----
2018-08-28 16:53:15 UTC - William: Can you explain what I’m missing?
----
2018-08-28 17:02:43 UTC - Matteo Merli: @William there are deep architectural differences between Pulsar and Kafka. These differences affect the feature set, performance and operability.
One such example is the separation of service and storage layers (for this last one we’re using Apache BookKeeper). There’s a quick slide deck here on the subject: <https://www.slideshare.net/merlimat/apache-pulsar-storage-architecture> and blog post <https://streaml.io/blog/pulsar-segment-based-architecture>
The separation of serving and storage is also what enables us to provide support tiered storage.
Another quick example is the support for queues (eg: multiple consumers on a single partition within a single subscription/consumer-group). More on this blog post: <https://streaml.io/blog/pulsar-streaming-queuing>
----
2018-08-28 17:05:40 UTC - William: Thanks, I’ll dive into those links now. @Matteo Merli I just put together that you’re a Streamlio founder!
----
2018-08-28 19:56:07 UTC - Shalin: I am using ansible to set up my pulsar cluster, is there any way I could check if zookeeper cluster meta data has been initialized? , because on running my ansible playbook after the first time would call the `initialize_cluster_metadata` , which throws the `KeeperException$NodeExistsException: KeeperErrorCode = NodeExists`
----
2018-08-28 19:57:13 UTC - Shalin: So I could use that check to ensure on running the playbook, it does not run the command again.
----
2018-08-28 19:57:27 UTC - Sijie Guo: @Shalin: were you starting brokers before initialize_cluster_metadata?
----
2018-08-28 19:57:51 UTC - Sijie Guo: there is a race condition if you start brokers before initialze_cluster_metadata
----
2018-08-28 19:58:11 UTC - Shalin: No. But the second time the command executes, the brokers would be up and running.
----
2018-08-28 20:00:22 UTC - Sijie Guo: <https://github.com/apache/incubator-pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/broker-deployment.yaml#L85>
----
2018-08-28 20:00:59 UTC - Sijie Guo: @Shalin you can use this command to check if zookeeper cluster metadata is initialized or not. we are using it for the helm deployment.
----
2018-08-28 20:01:41 UTC - Sijie Guo: basically `bin/pulsar zookeeper-shell -server <zookeeper> ls /admin/clusters/<cluster-name>`
----
2018-08-28 20:06:33 UTC - Shalin: Awesome, I'll try it out. Thanks .
----
2018-08-28 21:46:15 UTC - Pierre-Antoine Mills: @Pierre-Antoine Mills has joined the channel
----
2018-08-28 21:53:18 UTC - Pierre-Antoine Mills: Hi everyone, does the Context.publish method publish across all the Clusters or only on the one holding the Function ?
----
2018-08-28 21:55:02 UTC - Sijie Guo: Across the cluster
----
2018-08-28 21:55:25 UTC - Matteo Merli: (if replication is setup)
----
2018-08-28 21:56:24 UTC - Matteo Merli: it publishes on a topic in local cluster. When replication is set, data can be asynchronously replicated to all other clusters
+1 : Pierre-Antoine Mills, Sijie Guo
----
2018-08-29 00:11:11 UTC - Sijie Guo: thanks @Matteo Merli for clarification
----
2018-08-29 03:39:48 UTC - penghui li: How can i get admin-rest-api-swagger.json
----
2018-08-29 03:45:09 UTC - penghui li: Sorry i found it.
----
2018-08-29 03:46:01 UTC - Matteo Merli: Yes, it gets created when you do `mvn package -Pswagger`
----
2018-08-29 06:36:19 UTC - timo: @timo has joined the channel
----
2018-08-29 06:44:12 UTC - Poule: @William I think with Kafka just can't just add machines, you have to rebalance manually your data by telling Kafka what to copy on what machine.
----
2018-08-29 06:51:09 UTC - Poule: I also suggest you to read extensively the Kafka mailing list to get a good grasp of the problems you can run into
----
2018-08-29 09:03:02 UTC - Steven King: @Steven King has joined the channel
----