You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/05/09 09:11:02 UTC
Slack digest for #general - 2018-05-09
2018-05-08 17:08:18 UTC - Daniel Ferreira Jorge: Hi, are there plans to support Google Cloud Storage or just S3 on the tiered storage? Also, do we have an ETA on version 2? Just an estimate... a week? a month? 6 months?
----
2018-05-08 17:20:38 UTC - Matteo Merli: Hi @Daniel Ferreira Jorge, the tiered storage will have a pluggable backend implementation. The target for 2.1 release is to have S3 backend. Following releases might included HDFS and GCS, based on feedback.
----
2018-05-08 17:21:56 UTC - Matteo Merli: > Also, do we have an ETA on version 2? Just an estimate... a week? a month? 6 months?
The 2.0 release is “complete” at this point. The release process has started. We’re fixing few packaging/license headers stuff. ETA is ~1week to be announced.
+1 : Byron
----
2018-05-08 17:24:26 UTC - Daniel Ferreira Jorge: @Matteo Merli Regarding the 2.0, it is GREAT news! Also, if there is a pluggable backend implementation, maybe I will be able to contribute with a GCS backend...
+1 : jia zhai
----
2018-05-08 17:25:36 UTC - Daniel Ferreira Jorge: Is the S3 backend ready? So I can copy/paste some stuff? :slightly_smiling_face:
----
2018-05-08 17:29:51 UTC - Matteo Merli: @Ivan Kelly and @jia zhai are hard at work on that :slightly_smiling_face:
----
2018-05-08 17:33:11 UTC - Daniel Ferreira Jorge: Great!
----
2018-05-08 17:45:29 UTC - Josh West: <https://pulsar.incubator.apache.org/docs/latest/getting-started/pulsar-2.0> :disappointed:
----
2018-05-08 17:45:35 UTC - Josh West: first link from the main docs
----
2018-05-08 17:49:00 UTC - Daniel Ferreira Jorge: @Josh West Try with "Pulsar-2.0" (<https://pulsar.incubator.apache.org/docs/latest/getting-started/Pulsar-2.0/>)
----
2018-05-08 17:49:14 UTC - Josh West: :slightly_smiling_face:
----
2018-05-08 17:49:15 UTC - Josh West: yay
----
2018-05-08 17:49:18 UTC - Matteo Merli: yes, there’s a typo
----
2018-05-08 17:49:24 UTC - Matteo Merli: in the link
----
2018-05-08 17:49:41 UTC - Matteo Merli: also, few of the docs are not fully updated yet
----
2018-05-08 17:50:14 UTC - Josh West: this is as 2.0 get's rolled out yeah?
----
2018-05-08 17:50:17 UTC - Josh West: will be patient
----
2018-05-08 17:51:49 UTC - Matteo Merli: :slightly_smiling_face:
----
2018-05-08 20:34:31 UTC - Idan: @Idan has joined the channel
----
2018-05-08 20:36:13 UTC - Idan: Hi all. we are using SQS of amazon as our messaging system. it was big mistake as it didnt fit our low latency needs. however we got some nice features which we relay on them. like auto re-try and dead letter queues automaticlly forwarding after number of tr-tries
----
2018-05-08 20:36:26 UTC - Idan: do you think we can get same features with PULSAR?
----
2018-05-08 20:36:57 UTC - Idan: i read alot about it. our first concerns would be the 'devops' around it.
----
2018-05-08 20:37:08 UTC - Piotr: I´ve actually also wondered about dead letter queues... Is there some way of achieving that with pulsar?
----
2018-05-08 20:38:19 UTC - Piotr: @Idan if with devops you mean managed hosting I´ve also been checking up on that and they haven´t gotten around to that yet. So you´ll have to host & manage your clusters yourself
----
2018-05-08 20:38:52 UTC - Idan: well.. how hard was it for you to build and maintain it's cluster?
----
2018-05-08 20:39:01 UTC - Idan: iam talking about throughput of 100 m/s
----
2018-05-08 20:39:08 UTC - Idan: latency should be 10 ms per message
----
2018-05-08 20:39:15 UTC - Idan: and must have persistency
----
2018-05-08 20:40:53 UTC - Piotr: haven´t gotten there yet myself. But I´m sure somebody else can answer that. So far the help I´ve seen everyone been getting by the pulsar guys has been outstanding
----
2018-05-08 20:42:32 UTC - Piotr: You´ve got plenty in the docs if you haven´t read through it already <https://pulsar.incubator.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/>
----
2018-05-08 20:43:13 UTC - Idan: we got like 9 microservices in our real-time flow. we have specific sequence where each microservices is Vert.X based
----
2018-05-08 20:43:32 UTC - Idan: we need low latency message queue to stitched between them
----
2018-05-08 20:43:50 UTC - Idan: got like 2 weeks for rollout;P
----
2018-05-08 20:44:04 UTC - Idan: so I need to understand how difficult would it to get this on air
----
2018-05-08 20:45:09 UTC - Piotr: I guess you should read up on the pros & cons of pulsar vs other systems. For me pulsar is a much better option than rabbitmq and kafka because it is designed to support a huge number topics, and rabbit & kafka are not
----
2018-05-08 20:45:32 UTC - Piotr: your needs may differ
----
2018-05-08 20:45:42 UTC - Idan: do you have any idea about dashboard
----
2018-05-08 20:45:45 UTC - Idan: of pulsar
----
2018-05-08 20:45:50 UTC - Idan: to see queue's status, etc..
----
2018-05-08 20:45:55 UTC - Idan: monitoring
----
2018-05-08 20:47:04 UTC - Idan: ok i see they got one
----
2018-05-08 20:47:29 UTC - Idan: perhaps guys here can give some idea how to handle DLQ's after X retries using PULSAR
+1 : Piotr
----
2018-05-08 20:47:39 UTC - Idan: @Piotr thank you
----
2018-05-08 20:56:03 UTC - Ali Ahmed: @Piotr @Idan dead letter queues are prevalent in things like rabbit mq , it should be simple to add in pulsar
----
2018-05-08 20:57:28 UTC - Ali Ahmed: one possibility is small function in pulsar computer layer that’s coming in 2.0 , where retry logic and routing can be handled with high throughput and low latency
----
2018-05-08 21:00:20 UTC - Ali Ahmed: @Idan it will very simple to go to production with a small pulsar cluster for your micro-services , and you can scale up without rebalancing or downtime
----
2018-05-08 21:09:41 UTC - Matteo Merli: @Idan It’s definitely possible to have similar functionality as SQS with Pulsar. Some companies have built SQS compatible gateways and migrated to Pulsar.
* Auto-retry: This is done by setting an “acknowledge-timeout” on the consumer. See <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#ackTimeout-long-java.util.concurrent.TimeUnit->
* Dead-letter-queue: there is no currently support for. It’ not difficult to add, but it would need some support on the broker side to deal with the counter. At this point, an application can build its own logic by keeping a cache of the message ids received in the last X time and publish on the DLQ topic (and acknowledge on consumer) when the counter reaches N
Regarding dashboard and monitoring. There is a basic dashboard that shows the current status: <http://pulsar.apache.org/docs/latest/admin/Dashboard/>
Metrics are exposed in Prometheus format. There is already a Grafana set of dashboards for that: <http://pulsar.apache.org/docs/latest/deployment/Monitoring/>
----
2018-05-08 23:45:19 UTC - jia zhai: Hi @Daniel Ferreira Jorge FYI. This is the issue tracking s3 backend work:
<https://github.com/apache/incubator-pulsar/issues/1511>
----
2018-05-09 05:17:19 UTC - Idan: Merlimat i have like 4 days put this on production. Do you think its doneable?
----
2018-05-09 05:18:53 UTC - Idan: Also iam looking for some feedback from people who got this in production...can you reffer to me any pleasr? We have doubts going this way or kafka
----
2018-05-09 05:19:04 UTC - Ali Ahmed: it should be, do you zookeeper cluster available ?
----
2018-05-09 05:19:19 UTC - Idan: Yes we havr zookeeper for our vertx instances
----
2018-05-09 05:20:06 UTC - Matteo Merli: @Idan in Kafka you cannot replicate a queue like SQS as it is
----
2018-05-09 05:20:20 UTC - Idan: Iam worried about the tiny things to tune over the cluster.. thats not just for small poc'ing we need to take here rigy decisions as we have 2 weeks for rollout and SQS fu*** us
----
2018-05-09 05:20:25 UTC - Matteo Merli: the model is that you can only have 1 active consumer per partition
----
2018-05-09 05:20:44 UTC - Idan: Also worried about community as PULSAR pretty new
----
2018-05-09 05:20:50 UTC - Idan: If we stuck or anything
----
2018-05-09 05:21:37 UTC - Matteo Merli: since you mentioned 100 msg/s. that’s seems very easy to manage
----
2018-05-09 05:27:38 UTC - Idan: True, kafka isnt sqs aswell but its a known 'beast' also has docs forums community so i can focus on code adaption. Big doubt as if I had more time i would try PULSAR for sure. It looks great.
----
2018-05-09 05:28:49 UTC - Idan: 100 nsgs for a strat ofcourse we will expect 10k in the neqr future
----
2018-05-09 05:31:44 UTC - Matteo Merli: sure, my point was that at that scale you shouldn’t expect to face particular problems, or require any particular tuning
----