You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/08/09 09:11:07 UTC
Slack digest for #general - 2019-08-09
2019-08-08 09:16:15 UTC - divyasree: ok i wont use hereafter..
----
2019-08-08 09:43:05 UTC - Alexandre DUVAL: @Jerry Peng
----
2019-08-08 09:43:21 UTC - Alexandre DUVAL: im interested in the answer
----
2019-08-08 09:48:53 UTC - Alexandre DUVAL: ```PulsarProducer.send() failed: org.apache.pulsar.client.api.PulsarClientException: java.lang.IllegalStateException: recycled already {:exception=>java.util.concurrent.ExecutionException: org.apache.pulsar.client.
api.PulsarClientException: java.lang.IllegalStateException: recycled already} ``` what means this?
----
2019-08-08 10:07:29 UTC - Alexandre DUVAL: on exception catch my code retried the send and it did it 180 000times, then my consumer received the same message 180 000 times. I'm currently in ProducerImpl.java to check how it works, but if you have infos.. :slightly_smiling_face:
----
2019-08-08 13:29:11 UTC - Alexandre DUVAL: other point, is it possible to unsubscribe automatically when no consumer are active on consumerConfig?
----
2019-08-08 15:58:18 UTC - Grant Wu: Anyone have any idea what to do with
```
503 Failed to create producer: Cannot create producer on topic with backlog quota exceeded: org.apache.nifi.processor.exception.ProcessException: Failed to register processor to WebSocket service due to: java.io.IOException: Failed to connect <censored> due to: java.util.concurrent.ExecutionException: org.eclipse.jetty.websocket.api.UpgradeException: 503 Failed to create producer: Cannot create producer on topic with backlog quota exceeded
```
----
2019-08-08 15:59:22 UTC - Yuvaraj Loganathan: <https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#backlog-quotas>
----
2019-08-08 15:59:35 UTC - Grant Wu: sorry
----
2019-08-08 15:59:38 UTC - Grant Wu: Of note is:
----
2019-08-08 15:59:40 UTC - Grant Wu: ```
root@pulsar-broker-5d47994776-fxs2h:/pulsar# bin/pulsar-admin namespaces get-backlog-quotas public/solution
{ }
```
----
2019-08-08 15:59:44 UTC - Grant Wu: So there doesn’t appear to be a backlog set
----
2019-08-08 16:00:37 UTC - Yuvaraj Loganathan: Unless you are using pulsar with this pull request <https://github.com/apache/pulsar/pull/4320> You need to set a very large backlog
----
2019-08-08 16:00:47 UTC - Yuvaraj Loganathan: by default it is 10gb i think
----
2019-08-08 16:03:04 UTC - Grant Wu: Okay thanks!!!
----
2019-08-08 17:15:25 UTC - Sam Leung: A couple of admin actions were added in 2.4.0 which include deleting subscription, clearing backlog, and peeking at messages in a subscription.
----
2019-08-08 17:17:23 UTC - Sam Leung: There’s a broker setting ```brokerServicePurgeInactiveFrequencyInSeconds
How often broker checks for inactive topics to be deleted (topics with no subscriptions and no one connected)```
----
2019-08-08 17:18:22 UTC - Sam Leung: I wonder if there’s something similar for deleting subscriptions.
----
2019-08-08 17:23:11 UTC - Addison Higham: I created this: <https://github.com/apache/pulsar/issues/4920>
----
2019-08-08 17:28:10 UTC - Jerry Peng: @Alexandre DUVAL you can not pass in option as a CLI flag but you can just set it in your function config yaml file that you submit for your function. Does that make sense?
----
2019-08-08 17:29:54 UTC - Jerry Peng: > isn’t the write quorum across multiple nodes?
Yes but alternatively instead of increasing the write quorum (i.e. copies of your data) you can also choose to strip your data across more nodes to increase read throughput.
----
2019-08-08 17:54:11 UTC - Alexandre DUVAL: it does :slightly_smiling_face:. thanks.
----
2019-08-08 17:54:36 UTC - Alexandre DUVAL: that's the point :slightly_smiling_face:
----
2019-08-08 17:54:49 UTC - Addison Higham: and a proposed fix: <https://github.com/apache/pulsar/pull/4921>
----
2019-08-08 18:00:02 UTC - Aaron: Can anyone help with this?
----
2019-08-08 18:15:27 UTC - Sam Leung: I have a question about `Message has been dropped for non-persistent topic producer-id ...`. I am producing a message to a persistent topic that has an undersized bookie cluster (e.g. ensemble/ack quorum is supposed to be 3, I only have 2 instances). When I see that message, is there any chance it’s referring to my supposed persistent topic’s message being dropped, or is there some metadata exchange that’s getting dropped?
Note in the example logs below, I have 3 producers on one topic, and one consumer on a different topic.
----
2019-08-08 18:19:51 UTC - Sam Leung: Yeah sorry, misread the question at first :stuck_out_tongue:
----
2019-08-08 20:31:17 UTC - Federico Ponzi: @Federico Ponzi has joined the channel
----
2019-08-08 20:41:42 UTC - Tarek Shaar: We are trying to move to Apache Pulsar but I am getting a lot of resistance from management due to lack of GUI based admin and monitoring tooling. Does Pulsar only support Java, REST and CLI tools for monitoring and administration?
----
2019-08-08 20:43:45 UTC - Jerry Peng: @Tarek Shaar pulsar does have an open source UI in which you can perform certain actions. For metrics, people usually use Prometheus and Graphana to monitor the cluster
----
2019-08-08 20:45:43 UTC - Tarek Shaar: What's the open source UI that is used? What kinds of actions does it support?
----
2019-08-08 20:54:03 UTC - Sam Leung: We have deduplication on, and it seems like that could generate that message.
----
2019-08-08 21:35:13 UTC - Ali Ahmed: @Tarek Shaar it mostly basic read only operations , there is another ui <https://github.com/bbonnin/pulsar-express> which has more features
----
2019-08-08 21:41:12 UTC - Tarek Shaar: Ok I will take a look. We have been using Sonic MQ here and its admin GUI is very strong (from protocol set up to cluster administration, topic and queue monitoring and much more) so when I present my case to the management about Pulsar the question about tooling is always coming up. Let me check this one out
----
2019-08-08 21:54:08 UTC - Sam Leung: I see the broker in `MessageDeduplication.java` updates `highestSequencedPushed.put(producerName, sequenceId);` before persist happens.
If an error in persistence happens, my producer client re-sends the same message without changing the sequenceId right? And that message would be dropped at the broker because it thinks it’s a duplicate.
The only time `highestSequencedPushed` is cleared is when dedup is disabled or after an inactivity timer, not when an error in persistence happens.
----
2019-08-08 22:29:55 UTC - Sam Leung: Looks like it’s a bug in dedupe. I turned dedupe off and I’m seeing the `Re-Sending 1 messages to server` retry until timeout, and throwing me a `TimeoutException` from Producer.send() as expected.
Will file a bug tomorrow.
----
2019-08-08 22:46:55 UTC - Victor Li: Thank you! @Sijie Guo
----
2019-08-09 01:33:46 UTC - Yi Tang: @Sam Leung yeah, checked these actions, thanks
----
2019-08-09 02:34:02 UTC - Luo Tian: Help: I started a function in my pulsar instance, shows:
```json
{
"numInstances" : 1,
"numRunning" : 0,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : false,
"error" : "",
"numRestarts" : 1,
"numReceived" : 0,
"numSuccessfullyProcessed" : 0,
"numUserExceptions" : 0,
"latestUserExceptions" : [ ],
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"averageLatency" : 0.0,
"lastInvocationTime" : 0,
"workerId" : "c-local-fw-10.0.2.131-8080"
}
} ]
}
```
----
2019-08-09 02:34:48 UTC - Luo Tian: why is `numRunning` 0?
----
2019-08-09 02:35:28 UTC - Luo Tian: BTW, it does not produce any output, and log, and dead letter (bother log and dead letter topics specified when creating)
----
2019-08-09 02:47:00 UTC - Sijie Guo: FYI. beside pulsar-express, I have also known:
Yahoo! Japan has developed an internal UI tool. @Nozomi Kurihara will present it in the pulsar meetup (beijing) next week. He might also be thinking of open sourcing the tool as well.
@tuteng is also developing a pulsar-manager to manage multiple pulsar clusters. that one is also to be open sourced soon.
----
2019-08-09 02:47:28 UTC - Sijie Guo: numRestarts: 1
----
2019-08-09 02:47:36 UTC - Sijie Guo: it seems that there are exceptions thrown
----
2019-08-09 02:47:43 UTC - Sijie Guo: might be worth checking the function log
----
2019-08-09 02:48:10 UTC - Luo Tian: thanks
----
2019-08-09 02:56:37 UTC - Luo Tian: It shows a schema incompatible error,
----
2019-08-09 02:57:00 UTC - Luo Tian: The source topic is of json messages, but does not set any schema.
----
2019-08-09 02:57:21 UTC - Luo Tian: while the function using java native interface as `Function<String, String>`
----
2019-08-09 02:59:18 UTC - Nozomi Kurihara: Yahoo! JAPAN is now developing an UI tool for admin operations. We're going to open it early September.
----
2019-08-09 05:08:40 UTC - Sijie Guo: oh I see. I think Function attempts to use a JSON / String schema to consume a topic without schema. Currently it prevents a consumer with schema to consume data from a topic without a schema.
----
2019-08-09 05:09:20 UTC - Sijie Guo: We can consider introducing a flag to allow function user bypass the schema validation.
----
2019-08-09 05:19:01 UTC - Luo Tian: <https://github.com/apache/pulsar/issues/4857>
----
2019-08-09 05:19:10 UTC - Luo Tian: I think it is the same as this issue
----
2019-08-09 05:41:05 UTC - Luo Tian: can I use `pulsar-admin schemas upload` to apply a schema to the topic?
----
2019-08-09 05:52:44 UTC - Luo Tian: I figured it out.
----
2019-08-09 05:53:11 UTC - Luo Tian: uploaded the schema you showed in the comment.
----
2019-08-09 06:10:48 UTC - Sijie Guo: cool
----
2019-08-09 06:14:16 UTC - jia zhai: @Aaron This looks like a bug, Would you please help open an issue in github?
<https://github.com/apache/pulsar/issues/new?assignees=&labels=type%2Fbug&template=bug_report.md&title=>
----
2019-08-09 06:20:19 UTC - Luo Tian: There seems no documentation for exact content of a schema.
----
2019-08-09 06:34:26 UTC - Kim Christian Gaarder: I reported a bug <https://github.com/apache/pulsar/issues/4912>
This is basic functionality in the java client Reader that is not working as expected.
Do I need to do more in order for the bug to be considered for fixing?
----
2019-08-09 06:35:57 UTC - Sijie Guo: will triage and pick it up
----
2019-08-09 06:38:53 UTC - Kim Christian Gaarder: @Sijie Guo awesome, It should be pretty easy to create some unit tests in Pulsar directly to reproduce this and get some regression tests, I have never written any unit-tests in Pulsar yet, so I opted to use a separate project for reproduction.
+1 : Sijie Guo
----
2019-08-09 06:39:30 UTC - Sijie Guo: yeah.
----
2019-08-09 06:39:32 UTC - Sijie Guo: will check
----
2019-08-09 07:20:58 UTC - ppgoml: @ppgoml has joined the channel
----
2019-08-09 07:29:15 UTC - jinfeng105: @jinfeng105 has joined the channel
----
2019-08-09 07:33:34 UTC - ppgoml: Hello, could pulsar share zookeeper with other applications?
----
2019-08-09 07:34:37 UTC - Ali Ahmed: @ppgoml yes
----
2019-08-09 07:37:34 UTC - Diego Salvi: @Diego Salvi has joined the channel
----
2019-08-09 07:41:39 UTC - Diego Salvi: Hi! I have a problem with a newly created topic (non partitioned created from admin api). When I attempt to open a reader on it i get a long wait and then a failure with "6 lookup request timedout after ms 30000" and not much more infos
----
2019-08-09 07:41:48 UTC - ppgoml: @Ali Ahmed thanks, where can I set the root znode dedicated for pulsar?
----
2019-08-09 07:43:39 UTC - Diego Salvi: org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 6 lookup request timedout after ms 30000
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:247)
at org.apache.pulsar.client.impl.ReaderBuilderImpl.create(ReaderBuilderImpl.java:69)
Unfortunately a I cannot find any more logs explaining why it happened
----
2019-08-09 07:47:30 UTC - Sijie Guo: is your client able to connect to your brokers?
or can you explain a bit how did you set up the cluster?
----
2019-08-09 07:54:26 UTC - Diego Salvi: acually I have 3 bookie and one pulsar broker
----
2019-08-09 07:55:48 UTC - Diego Salvi: I created 3 topics with PulsarAdmin
----
2019-08-09 07:56:52 UTC - Sijie Guo: bin/pulsar-admin brokers list <pulsar-cluster>
----
2019-08-09 07:57:16 UTC - Sijie Guo: can you show me the result of running this command?
----
2019-08-09 07:57:45 UTC - Diego Salvi: can be run directly from api too?
----
2019-08-09 07:57:57 UTC - Diego Salvi: like /admin/v2/broker-stats/topics?
----
2019-08-09 07:59:50 UTC - Sijie Guo: /admin/v2/<cluster-name>
----
2019-08-09 08:01:57 UTC - Diego Salvi: <http://localhost:9372/admin/v2/clusters/mycluster>
----
2019-08-09 08:02:01 UTC - Diego Salvi: gave me {}
----
2019-08-09 08:02:28 UTC - Diego Salvi: a... I forgot to add... for one topic I can create thereader... for another no
----
2019-08-09 08:03:55 UTC - Diego Salvi: <http://localhost:9372/admin/v2/brokers/mycluster>
----
2019-08-09 08:05:33 UTC - Diego Salvi: ["dev1:9372"]
----
2019-08-09 08:25:26 UTC - Sijie Guo: ```
bin/pulsar initialize-cluster-metadata \
--cluster pulsar-cluster-test\
--zookeeper hdp1:2181/kobe \
--configuration-store hdp1:2181/kobe \
--web-service-url <http://hdp1:8080> \
--web-service-url-tls <https://hdp1:8443> \
--broker-service-url <pulsar://hdp1:6650> \
--broker-service-url-tls <pulsar+ssl://hdp1:6651>
```
in conf/bookkeeper.conf:
```
zkServers=hdp1:2181
zkLedgersRootPath=/kobe/ledgers
```
in conf/broker.conf
```
zookeeperServers=zk:2181/kobe
configurationStoreServers=zk:2181/kobe
```
----
2019-08-09 08:25:53 UTC - Sijie Guo: are you able to connect to dev1:9372 ?
----
2019-08-09 08:35:14 UTC - Diego Salvi: directly? checking
----
2019-08-09 08:36:51 UTC - Diego Salvi: Yes I can
----
2019-08-09 08:36:57 UTC - Diego Salvi: checked with telnet
----
2019-08-09 08:39:28 UTC - Sijie Guo: Can you show the example code how do you configure the reader?
----
2019-08-09 08:45:34 UTC - Diego Salvi: sure... going to get te code
----
2019-08-09 08:48:00 UTC - Diego Salvi: try (Reader<byte[]> pulsarReader = pulsarClient.newReader()
.startMessageId(lastConsumedMessageId)
.topic(topicName)
.create()) {
/* reading logic */
}
----
2019-08-09 08:48:06 UTC - Diego Salvi: it fails on the create
----
2019-08-09 08:48:36 UTC - Diego Salvi: topicName is the same used for creation, lastConsumedMessageId is actually == MessageId.earliest
----
2019-08-09 08:48:47 UTC - Diego Salvi: the client itself is created with
----
2019-08-09 08:49:18 UTC - Diego Salvi: PulsarClient pulsarClient = PulsarClient.builder().serviceUrlProvider(new ServiceUrlProvider() {
@Override
public void initialize(PulsarClient pc) {
}
@Override
public String getServiceUrl() {
return pulsarBrokerURI.get();
}
}).build();
----
2019-08-09 08:49:33 UTC - Diego Salvi: pulsarBrokerURI is just a supplier to provide the right url
----
2019-08-09 08:50:34 UTC - Sijie Guo: http:// or pulsar:// ?
----
2019-08-09 09:05:38 UTC - Diego Salvi: hum actually is <pulsar://dev1:6650>
----
2019-08-09 09:05:48 UTC - Diego Salvi: which is listening and i can connect to
----
2019-08-09 09:07:16 UTC - Sijie Guo: No broker log?
----
2019-08-09 09:09:55 UTC - Diego Salvi: this is what I found
----
2019-08-09 09:09:57 UTC - Diego Salvi: 19-08-08-14-10-19 [idc7/ns/events] Failed to get create topic reader
19-08-08-14-10-19 Client closing. URL: <pulsar://dev1:6650>
19-08-08-14-09-19 [idc7/ns/events][reader-69e6697932] Subscribing to topic on cnx [id: 0x2be67bab, L:/10.168.10.79:52044 - R:dev1/10.168.10.81:6650]
19-08-08-14-10-19 [idc7/ns/events][reader-69e6697932] Failed to subscribe to topic on dev1/10.168.10.81:6650
19-08-08-14-10-19 [id: 0x2be67bab, L:/10.168.10.79:52044 - R:dev1/10.168.10.81:6650] request 2 timed out after 30000 ms
19-08-08-14-10-19 [id: 0x28390d63, L:/10.168.10.79:48170 ! R:dev1/10.168.10.79:6650] Disconnected
19-08-08-14-10-19 [id: 0x2be67bab, L:/10.168.10.79:52044 ! R:dev1/10.168.10.81:6650] Disconnected
----