You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/10/02 09:11:03 UTC
Slack digest for #general - 2020-10-02
2020-10-01 10:46:30 UTC - Linton: Looks like there’s also one a Flink Forward <https://www.flink-forward.org/global-2020/conference-program#flink---pulsar--the-path-to-unified-batch-and-streaming->
----
2020-10-01 10:57:12 UTC - Ward Loos: @Ward Loos has joined the channel
----
2020-10-01 13:45:25 UTC - dipali bhat: pulsar using persistent volume claim
----
2020-10-01 13:45:34 UTC - dipali bhat: any body using this
----
2020-10-01 13:46:27 UTC - dipali bhat: the values,yaml file does not mention about pv amd pvc . please advise
----
2020-10-01 14:29:50 UTC - Evan Furman: Yep, that is the one `apachepulsar/pulsar-manager:v0.2.0`
----
2020-10-01 14:37:41 UTC - Sijie Guo: Yes @Addison Higham will given a representation in Flink Forward Global.
----
2020-10-01 14:38:47 UTC - Evan Furman: What exactly does the bookkeeper visual manager do and how is it different than this part of the `application.properties` config?
```# bookie
bookie.host=<http://bookie-1>-{{ cluster_name }}.{{ route53_zone }}:8000
bookie.enable=false```
----
2020-10-01 14:45:07 UTC - Christopher Faini: Thanks for getting back to me! In my case, I'm looking at converting an existing C++ message processing project to use Apache Pulsar. While it is true that the main reason I asked this was because there is existing code I'd like to put into the pulsar functions, this is also a very large and complex project where we need the most optimal per message performance possible. I'm not sure that any of the current language options for pulsar functions would work for us.
----
2020-10-01 15:40:36 UTC - Christopher Faini: Just an example of how important low latency is for this project: We started looking at Pulsar because there are concerns that Kafka isn't fast enough for our requirements.
----
2020-10-01 17:18:46 UTC - Alan Hoffmeister: has pulsar dropped support for protobuf schemas? I couldn't find anything related to it in the latest docs
----
2020-10-01 17:59:57 UTC - Addison Higham: No, still supported: <http://pulsar.apache.org/docs/en/concepts-schema-registry/#supported-schema-formats>
----
2020-10-01 18:01:00 UTC - Pushkar Sawant: We are running Pulsar 2.6.0 deployed using helm charts. For couple of topics i get following exception. Has anyone seen this? i get a 504 timeouts when i try to delete these two topics. Rest of the topics seem to be working fine
----
2020-10-01 18:01:00 UTC - Pushkar Sawant: ```java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException: Failed to load topic within timeout
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_252]
at org.apache.pulsar.broker.service.BrokerService.lambda$futureWithDeadline$18(BrokerService.java:836) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.util.concurrent.TimeoutException: Failed to load topic within timeout
at org.apache.pulsar.broker.service.BrokerService.loadOrCreatePersistentTopic(BrokerService.java:955) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$13(BrokerService.java:731) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:277) ~[org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0]
at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:130) ~[org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0]
at org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:730) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at org.apache.pulsar.broker.service.ServerCnx.lambda$null$14(ServerCnx.java:841) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996) ~[?:1.8.0_252]
at org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$16(ServerCnx.java:792) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628) ~[?:1.8.0_252]
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996) ~[?:1.8.0_252]
at org.apache.pulsar.broker.service.ServerCnx.handleSubscribe(ServerCnx.java:777) ~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:215) ~[org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:191) ~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:153) ~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) ~[io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) ~[io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) ~[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) ~[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
... 4 more```
----
2020-10-01 18:02:51 UTC - Addison Higham: There is support, see this example: <https://github.com/apache/pulsar-helm-chart/blob/master/examples/values-bookkeeper-aws.yaml#L73>
It allows you to configure the storageclasses and then pulsar uses statefulsets with volumeclaimtemplates
----
2020-10-01 19:45:30 UTC - Jim M.: I keep getting ErrorNativeIoException failed:connection rest by peer
----
2020-10-02 00:44:52 UTC - Addison Higham: Hi @Pushkar Sawant there are some known issues that can happen sometimes if something goes wrong with zookeeper and bookkeeper and the topic fails to load once it can sometimes get stuck in a bad state. There are fixes for this I believe in 2.6.1 (or will be released in 2.6.2). In the interim, you can unload the topic and that ~may~ should fix it
----
2020-10-02 05:59:02 UTC - Priyath Gregory: @Priyath Gregory has joined the channel
----
2020-10-02 06:07:22 UTC - Priyath Gregory: Hey guys, apologies in advance if this isn't the right place for this.
Was going through the pulsar-io kinesis consumer source code and noticed that checkpointing is periodically done in an *asynchronous* manner after records are pushed to an in-memory blocking queue. How is the possibility of record loss handled with this approach? (eg: checkpoint completed but instance failure while records are waiting in blocking queue)
<https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisRecordProcessor.java>
----
2020-10-02 08:44:33 UTC - Shivji Kumar Jha: Hi, I made some CPP / Python client changes in <https://github.com/apache/pulsar/pull/8173|my PR> and I see the same suite of <https://pipelines.actions.githubusercontent.com/EXIZfX8ixSbQEfqDA00dMJrEtloTMSwjlUs9NqJqrTcJfd7NaR/_apis/pipelines/1/runs/109338/signedlogcontent/3?urlExpires=2020-10-02T08%3A29%3A52.0688216Z&urlSigningMethod=HMACV1&urlSignature=EMEmcnANRlznJ0g5jEuQv6FU1m7tdr3tTH14AqOJPX0%3D|tests failing>. Don't quite see why though, the errors seem completely unrelated. I see the tests run passing in a different PR though. Need help understanding this.
----
2020-10-02 08:46:44 UTC - Shivji Kumar Jha: Hi @Penghui Li Is this a known issue?
----
2020-10-02 08:49:04 UTC - Shivji Kumar Jha: <https://pipelines.actions.githubusercontent.com/EXIZfX8ixSbQEfqDA00dMJrEtloTMSwjlUs9NqJqrTcJfd7NaR/_apis/pipelines/1/runs/109354/signedlogcontent/3?urlExpires=2020-10-02T08%3A33%3A36.9834223Z&urlSigningMethod=HMACV1&urlSignature=TQvSxixR9wgHyeRxjSE09yHpckzki2ztZP8oI1SBZgw%3D|Another failing run>, again apparently unrelated. But these tests have failed in multiple runs so I wanted to check them.
----
2020-10-02 08:49:07 UTC - Penghui Li: Looks you can try to rerun the test
----
2020-10-02 08:50:13 UTC - Penghui Li: This one looks related to the github action environment
----
2020-10-02 08:51:16 UTC - Shivji Kumar Jha: ```/pulsarbot run-failure-checks```
?
----
2020-10-02 08:51:25 UTC - Penghui Li: yes
+1 : Shivji Kumar Jha
----
2020-10-02 08:52:45 UTC - Shivji Kumar Jha: @Penghui Li Can you assign me the <https://github.com/apache/pulsar/issues/8176|issue> / <https://github.com/apache/pulsar/pull/8173|PR> and Sijie, @Yunze Xu as reviewers please? Happy to have other reviewers if required.
----
2020-10-02 08:54:58 UTC - Shivji Kumar Jha: @Penghui Li didnt trigger test runs with that
----
2020-10-02 08:56:30 UTC - Penghui Li: done
----
2020-10-02 08:57:26 UTC - Penghui Li: Since yunze is no a committer yet, you can ping him if need a review.
----
2020-10-02 08:57:57 UTC - Shivji Kumar Jha: sure, I will do that, Because he has been involved :-)
----
2020-10-02 08:58:20 UTC - Shivji Kumar Jha: started now, may be it was queued.
----