You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/08/17 09:11:03 UTC
Slack digest for #general - 2019-08-17
2019-08-16 10:12:35 UTC - Sijie Guo: HI
----
2019-08-16 10:12:42 UTC - Sijie Guo: sorry I missed the notification
----
2019-08-16 10:13:04 UTC - Sijie Guo: the producer defines the schema because it produces messages (modify the state of the topic).
----
2019-08-16 10:15:40 UTC - Alexandre DUVAL: Sure but why a subscription doesn't do the same thing when a topic does not exist?
----
2019-08-16 10:16:16 UTC - Alexandre DUVAL: "why" is a large question, but you get my point :stuck_out_tongue:.
----
2019-08-16 10:24:37 UTC - Sijie Guo: subscription also trigger creation of a topic and define the schema for the topic if it doesn’t exist, no?
----
2019-08-16 10:25:01 UTC - Maxim Shatunov: @Maxim Shatunov has joined the channel
----
2019-08-16 10:27:40 UTC - Alexandre DUVAL: In my case it's triggering the creation of the topic but let the default schema even if custom one is defined. So it is possibly an issue on the pulsar4s client now that you told me the normal behavior. Thx.
----
2019-08-16 10:28:24 UTC - Sijie Guo: you might check the behavior of java client
----
2019-08-16 12:37:08 UTC - Alexandre DUVAL: ok, fixed, the implicit in pulsar4s was just conflict with another, mb. sorry.
----
2019-08-16 13:24:15 UTC - Aaron: Has anyone encountered the exception `java.lang.UnsatisfiedLinkError: org.apache.pulsar.shade.org.xerial.snappy.SnappyNative.maxCompressedLength(I)I` when trying to use SNAPPY compression?
----
2019-08-16 13:37:34 UTC - Alexandre DUVAL: When I delete a topic there is cache about its schema?
----
2019-08-16 14:08:08 UTC - Richard Sherman: Does pulsar handle zookeeper session timeouts gracefully?
----
2019-08-16 14:22:12 UTC - Chris Bartholomew: Yes, Pulsar will automatically redeliver the message. This is a way to manually force redelivery. There is some debate about whether it should be public here: <https://github.com/apache/pulsar/pull/51#discussion_r82476470>. It is used a lot in the tests to, I assume, speed up test execution (no need to wait for timer to fire).
+1 : Raman Gupta
----
2019-08-16 14:42:32 UTC - Tarek Shaar: Will a subscriber ever miss a message after establishing a subscription? In other words, if a subscriber comes online then disconnects for a long time then come back on line in the future, is there a time out on the subscription (assuming we are still within the message retention policies).
----
2019-08-16 15:06:37 UTC - balaji: @balaji has joined the channel
----
2019-08-16 15:26:43 UTC - balaji: can someone please give me some example on exactly once delivery with pulsar and flink combination. I have also posted the same question in stack overflow. <https://stackoverflow.com/questions/57523574/exactly-once-delivery-end-to-end-with-pulsar-and-flink>
----
2019-08-16 15:35:37 UTC - Addison Higham: there are few gaps in pulsar to make that possible yet, first:
- the flink pulsar consumer isn't exactly once yet. For that to happen, ideally, you need flink to track the position into the subscription so it can accurately rewind. As it stands, the existing consumer just uses a pretty normal subscription. If you look at <#CA678TN9G|flink-connector> there are some more technical details. That said... if you are willing to implement a de-dupe operator, that just keeps N number of minutes of message ids, it is fairly easy to de-dupe right after the consumer
- pulsar doesn't yet have transactions like kafka does, but it is actively being developed. When pulsar has right now is the producer dedupe, which can get you pretty far, basically, if you can control the sequence number you send along with your messages into pulsar, then your flink job fails and it sends those same sequence numbers again, it won't persist the duplicates, see this blog post for more detail: <https://jack-vanlightly.com/blog/2018/10/25/testing-producer-deduplication-in-apache-kafka-and-apache-pulsar>
+1 : balaji
----
2019-08-16 15:39:00 UTC - Addison Higham: so you can get effectively once with flink right now... it just takes more state tracking in your app by de-duping on input and using more state to build stable sequence numbers, you can get that wrong, so it isn't bullet proof. I am sure as transactions and flink integration improves there will be announcements around it
----
2019-08-16 16:53:57 UTC - Ali Ahmed: @balaji @Addison Higham exactly once is partially a misnomer here are some docs to read through to better understand things and what is possible in modern distributed systems.
<https://www.slideshare.net/BigDataSpain/endtoend-exactly-once-with-heron-pulsar-by-ivan-kelly-at-big-data-spain-2017>
<https://streaml.io/blog/exactly-once>
----
2019-08-16 16:54:56 UTC - Addison Higham: I am trying to get in the habit of saying "effectively once" but yes, "exactly once" in the context of what flink/kafka would use the term
----
2019-08-16 16:59:01 UTC - Alberto Antenangeli: @Alberto Antenangeli has joined the channel
----
2019-08-16 17:48:25 UTC - balaji: @Ali Ahmed In the slides you posted there is no concrete code example on how end to end effectively once delivery was achieved with heron and pulsar.
----
2019-08-16 18:01:41 UTC - Jerry Peng: @balaji you can also achieve exactly/effectively once using pulsar functions if only have output to a topi
----
2019-08-16 18:01:44 UTC - Jerry Peng: Topic
----
2019-08-16 18:56:47 UTC - Tarek Shaar: Am using Apache Pulsar 2.3.2. I have enabled message de-duplication in broker.conf and I am setting the producer name and seq number on each message sent. When I ask for the last seq number I am getting -1 even though I request it from the same producer name. Are there any other settings I need to take care of?
----
2019-08-16 20:22:22 UTC - Martin Svensson: Can I create a subscription that allows me to consume from the start of an existing topic?
----
2019-08-16 20:30:29 UTC - Ali Ahmed: @Martin Svensson You can use the position arg in consumer or use the reader class
```
Consumer<UserMessagePojo> consumer = client.newConsumer(JSONSchema.of(UserMessagePojo.class))
.topic(topic)
.subscriptionInitialPosition(position)
.subscriptionName("schema-tester")
.subscribe()) {
```
----
2019-08-16 20:30:55 UTC - Martin Svensson: thanks!
----
2019-08-16 21:16:56 UTC - Ming Fang: are there any plans/talks of supporting topic/queue aliasing? similar to this <https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.adm.doc/q020820_.htm>
----
2019-08-16 21:19:56 UTC - Tarek Shaar: If anyone encounters the issue I mentioned above where the seq # returned is -1, you need to set the initial seq id when constructing the producer using the producer builder method, initialSequenceId()
+1 : Chris Bartholomew
----
2019-08-16 23:11:23 UTC - Ali Ahmed: @Ming Fang there are no current plans for something like this
----
2019-08-16 23:19:12 UTC - Alexandre DUVAL: @Matteo Merli
----
2019-08-16 23:38:41 UTC - Ming Fang: ok thanks
----
2019-08-17 01:52:15 UTC - marcelomfranca: @marcelomfranca has joined the channel
----
2019-08-17 08:04:09 UTC - Kendall Magesh-Davis: Good morning. I’ve deployed pulsar 2.4 on kubernetes using the public helm chart. I had to cordon/drain a node which had a bookkeeper on it, this required deleting the local storage on that node.
Not surprisingly, that bookie is not able to successfully startup again after being deployed to a different node. It’s complaining of cookie issues.
```07:53:28.259 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [data/bookkeeper/journal/current, data/bookkeeper/ledgers/current]
07:53:28.259 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
```
```07:53:28.267 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:470) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:691) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:106) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.doMain(Main.java:221) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.server.Main.main(Main.java:203) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:314) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]```
What is the best practice for resolving this issue?
----