You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/11/10 09:11:04 UTC
Slack digest for #general - 2018-11-10
2018-11-09 10:55:04 UTC - tuteng: ./pulsar/bin/pulsar-admin source localrun --archive /data/pulsar/pulsar/pulsar-io/canal/target/pulsar-io-canal.jar --classname org.apache.pulsar.canal.CanalSource --tenant public --namespace default --name canal --source-config-file /data/pulsar/canal-source-config-file.yml Exception in thread "main" com.beust.jcommander.ParameterException: Unknown option: --this$0
at com.beust.jcommander.JCommander.parseValues(JCommander.java:742)
at com.beust.jcommander.JCommander.parse(JCommander.java:282)
at com.beust.jcommander.JCommander.parse(JCommander.java:265)
at org.apache.pulsar.functions.runtime.LocalRunner.main(LocalRunner.java:87)
----
2018-11-09 10:55:42 UTC - tuteng: What is the problem?
----
2018-11-09 10:56:28 UTC - tuteng: standalone mode
----
2018-11-09 10:58:01 UTC - tuteng: myself source: <https://github.com/AmateurEvents/pulsar/tree/feature/pulsar-io-support-canal/pulsar-io/canal/src/main/java/org/apache/pulsar/canal>
----
2018-11-09 10:58:32 UTC - Ivan Kelly: where is canal-source-config-file.yml
----
2018-11-09 10:59:53 UTC - tuteng: local file: configs:
zkServers: "10.4.239.22:2181"
batchSize: 5120
destination: "example"
username: ""
password: ""
----
2018-11-09 11:15:08 UTC - Ivan Kelly: anything in ./logs?
----
2018-11-09 11:16:57 UTC - Ivan Kelly: can you try with PULSAR_LOG_LEVEL=DEBUG in the environment
----
2018-11-09 12:40:25 UTC - tuteng: logs are normal. No information received for pulsar broker
----
2018-11-09 12:50:21 UTC - Ivan Kelly: what version is this?
----
2018-11-09 13:07:20 UTC - tuteng: I'm developing on the master branch, and I don't know which version. Sorry, this is history commit <https://github.com/AmateurEvents/pulsar/commits/feature/pulsar-io-support-canal>
----
2018-11-09 13:08:41 UTC - Ivan Kelly: Ok, are you building the whole pulsar repo each time then?
----
2018-11-09 13:11:08 UTC - tuteng: yes, use the command mvn clean install -DskipTests
----
2018-11-09 13:12:01 UTC - tuteng: Today is the first compilation and installation.
----
2018-11-09 13:12:31 UTC - Ivan Kelly: Ok, can you put a log line in LocalRunner.main to print out the full commandline?
----
2018-11-09 13:18:47 UTC - tuteng: OK, I'll try. Thank you very much.
----
2018-11-09 13:19:49 UTC - Ivan Kelly: it's possible a change to functions may have broken localrun for I/O so let me know if you find anything
----
2018-11-09 13:57:53 UTC - Grégory Guichard: @Grégory Guichard has joined the channel
----
2018-11-09 14:35:40 UTC - jia zhai: Oh, IC @Ivan Kelly @tuteng <https://github.com/apache/pulsar/pull/2892>
There was a change to these command in master
----
2018-11-09 16:42:41 UTC - Matteo Merli: @tuan nguyen anh it’s strange. On my laptop with the default config the throughput is ~300K msg/s (300 MBytes/s) :
```
08:40:36.080 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 297539.6 msg/s --- 2324.5 Mbit/s --- Latency: mean: 2.723 ms - med: 2.100 - 95pct: 4.679 - 99pct: 7.302 - 99.9pct: 17.603 - 99.99pct: 719.831 - Max: 721.191
08:40:46.168 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 313669.3 msg/s --- 2450.5 Mbit/s --- Latency: mean: 2.969 ms - med: 1.684 - 95pct: 4.267 - 99pct: 8.664 - 99.9pct: 43.156 - 99.99pct: 1462.783 - Max: 1463.527
08:40:56.020 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - ------------------- DONE -----------------------
08:40:56.051 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 3.010 ms - med: 1.808 - 95pct: 4.784 - 99pct: 9.524 - 99.9pct: 106.352 - 99.99pct: 1461.311 - 99.999pct: 1463.439 - Max: 1463.527
```
----
2018-11-09 17:02:39 UTC - tuan nguyen anh: amazing, i don't understand why this so my solution isiun it macbook
----
2018-11-09 17:34:12 UTC - Matteo Merli: @tuan nguyen anh Can you open an issue on github and recap all your steps?
----
2018-11-09 19:13:36 UTC - Emma Pollum: What is the best way to give the JVM more more memory for the broker?
----
2018-11-09 19:15:08 UTC - Matteo Merli: edit `conf/pulsar_env.sh` :
`PULSAR_MEM=${PULSAR_MEM:-"-Xms8g -Xmx8g -XX:MaxDirectMemorySize=16g"}`
----
2018-11-09 19:15:41 UTC - Emma Pollum: Thank you!
----
2018-11-09 21:44:05 UTC - Beast in Black: @Matteo Merli as a followup to all those annoying questions I asked yesterday, I have a couple more questions which popped up when I was making the change to have my consumer type be shared. These questions are due to what I understand is the round-robin nature of the `shared` consumer type and the documentation for the same available at `<https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/#Exclusive-u84f1>`
*To recap:* In my use case, each app instance (C++ app using the pulsar CPP client) will subscribe to a global topic (persistent as well as non-persistent) using a unique consumer subscription name i.e. each app instance will use a unique subscription name which is different from all the rest of the app instances. This unique name is persisted on each app instance so when the app comes back after a crash it will continue to use the same subscription name as before.
Thus, on each app instance there will be exactly one consumer and one subscriber for the topic, and this consumer's type will be set to `shared` to allow the app to restart after a crash and to re-subscribe to the topic using the same consumer name it used as before without experiencing pulsar client errors related to trying to resubscribe to an existing (i.e unreaped) broker consumer connection when using the same subscription name.
*Here come the questions:*
1. If new messages were published to that topic while the app instance was offline due to the crash/restart, when it connects back using the same subscription name to the same shared consumer (to which no other subscriptions will ever connect), will it receive/consume the messages which it missed while it was down?
2. After coming back up from the crash, will it receive/consume all new messages published after it came back up? Essentially, the question asks whether the round-robin nature of shared consumers will cause message receipt issues when using the same subscription name for the same shared consumer after an app crash/restart.
3. On each app instance, since there is only a single consumer/subscriber for the given topic, can message ordering still be guaranteed despite the round-robin nature of a shared consumer?
----
2018-11-09 22:47:50 UTC - Sanjeev Kulkarni: @tuteng The fix to your problem is at <https://github.com/apache/pulsar/pull/2970>
----
2018-11-10 00:34:31 UTC - Libin: @Libin has joined the channel
----