You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/04/07 09:11:01 UTC
Slack digest for #general - 2019-04-07
2019-04-06 13:34:08 UTC - Ryan Samo: Hey guys,
I’m checking out Sinks in Pulsar IO. It looks like a Sink it just a wrapped Consumer, is that a correct assessment? Just asking so I can look into adjusting performance a bit with params such as receiverQueueSize, etc. This is in regards to building custom sinks or maybe even adjusting sinks that are currently available.
----
2019-04-06 13:40:54 UTC - Samuel Sun: Hi , one question for buffer of MESSAGE(aaaa, 61 61 61 61):
```
00 00 00 48 00 00 00 17 08 09 4A 13 08 00 12 0F
08 41 10 19 18 FF FF FF FF FF FF FF FF FF 01 0E
01 D5 CA 13 EE 00 00 00 1F 0A 14 70 72 6F 64 75
63 65 72 5F 6E 61 6D 65 5F 73 69 6E 67 6C 65 10
00 18 9B CF BD 96 9F 2D 61 61 61 61 00 00 00 48
00 00 00 17 08 09 4A 13 08 00 12 0F 08 41 10 1A
18 FF FF FF FF FF FF FF FF FF 01 0E 01 0A 1B 10
DB 00 00 00 1F 0A 14 70 72 6F 64 75 63 65 72 5F
6E 61 6D 65 5F 73 69 6E 67 6C 65 10 00 18 8B E7
BD 96 9F 2D 61 61 61 61
```
can I know which 4 bytes are checksum in this buffer ? according to function here <https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/api/Commands.java#L339>
----
2019-04-06 16:47:09 UTC - Jannis: @Jannis has joined the channel
----
2019-04-06 18:46:23 UTC - Ali Ahmed: @Ryan Samo sinks are wrapper over pulsar functions
----
2019-04-06 19:36:15 UTC - Ryan Samo: @Ali Ahmed yup I see that now, just wondering if functions are wrapping consumers and producers so that I can potentially downcast to get additional functionality that is not immediately exposed to the IO or Function interface?
----
2019-04-06 19:37:21 UTC - Ali Ahmed: No I don’t think you can downcast to get access to the producer consumer instance directly
----
2019-04-06 19:38:34 UTC - Ryan Samo: Ok thanks, my problem was with having a sink that couldn’t keep up with a message volume but my consumers can. Just looking at options to address the sink backlog
----
2019-04-06 19:39:04 UTC - Ryan Samo: Using partitions and parallelism does help, just not enough
----
2019-04-06 19:39:49 UTC - Ali Ahmed: I don’t there is bottleneck in sink framework the implementation might have something to do with it
----
2019-04-06 19:43:19 UTC - Ryan Samo: Yeah I think is on the receiving end, I’ll keep tweaking, thanks!
----
2019-04-06 21:51:51 UTC - Emma Pollum: I'm running into an issue with 2.3.0 proxy when tokens are enabled. I'm getting the following stack trace
----
2019-04-06 21:52:30 UTC - Emma Pollum:
----
2019-04-06 21:53:02 UTC - Emma Pollum: Some research showed me you are aware of similiar problem with the broker and that I should manually specify the numHttpServerThreads, however that configuration does not seem to exist in the proxy.conf
----
2019-04-06 21:54:04 UTC - Matteo Merli: It should be `httpNumThreads=8`
----
2019-04-06 21:56:09 UTC - Emma Pollum: Thanks!
----
2019-04-07 00:06:08 UTC - Emma Pollum: I'm trying to use tokens to authenticate through the proxy, but now when I issue commands like bin/pulsar-client produce
I recieve this error
org.apache.pulsar.proxy.server.ProxyConnection - [/127.0.0.1:47434] Unable to authenticate: Unsupported authentication mode: none
----
2019-04-07 00:08:22 UTC - Emma Pollum: I have tried submitting the command with --auth-params and --auth-plugin
----
2019-04-07 00:52:47 UTC - Anqing Xu: Pulsar seems to have a superior architecture compared to either Kafka or RabbitMQ. How come not many companies are using it? I know Yahoo has been using it internally. A Chinese company named Zhaopin is also using it
----
2019-04-07 01:54:14 UTC - Sanjeev Kulkarni: @Ryan Samo you can set the recieveQueueSize as part of the input spec in Sinkconfig. Look inside the ConsumerConfig class of SinkConfig
----
2019-04-07 01:55:46 UTC - Sanjeev Kulkarni: @Anqing Xu Pulsar came out of Yahoo into open source just in 2016.
----
2019-04-07 01:56:38 UTC - Ryan Samo: Awesome @Sanjeev Kulkarni, that’s exactly what I was looking for. I appreciate it!
----
2019-04-07 08:31:38 UTC - young: @young has joined the channel
----