You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/06/26 09:11:02 UTC

Slack digest for #general - 2018-06-26

2018-06-25 11:01:53 UTC - Sagar Gaur: hi i m trying to use Pulsar for steaming data and i m confused b/w Apache Kylin and Apache Druid can any body suggest which i should use along with Apache Pulsar...?
----
2018-06-25 11:56:02 UTC - Sagar Gaur: streaming*
----
2018-06-25 18:23:52 UTC - Jon Bock: @Sagar Gaur, can you describe more about what you’re specifically aiming to do?
----
2018-06-25 18:30:56 UTC - Karthik Palanivelu: Hello there, Can you please point me to any documentation that helps me configure two clusters in kubernetes for Geo-Replication? I found documentaion for Bare Metal Config.
----
2018-06-25 18:40:48 UTC - Ivan Kelly: @Karthikeyan Palanivelu it shouldn't be any different really, though the configuration servers need to be accessible externally
----
2018-06-25 18:44:10 UTC - Karthik Palanivelu: Are you referring to Zookeeper, If so how would I access them from One k8s cluster to other?
----
2018-06-25 18:45:28 UTC - dba: Hey guys. I am trying to connect from C# using the binary protocol. I am using the "Google.Protobuf" package from NuGet which is now version 3.6.0 and only goes back to 3.0.0. So I upgraded PulsarApi.proto to "proto3" (maybe that is why I am having some problems). Any plans on upgrading to proto3? I am trying to send a CommandConnect, but failing so far. Could someone show me a packet of a CommandConnect? Meaning every byte of the first frame you send in order to get a CommandConnected response? That would help a lot to see if I am getting the sizes (frame size and command size as big endian) and command right.
----
2018-06-25 18:48:00 UTC - Ivan Kelly: @dba there's no plans to go to proto3 for the pulsar interface. proto3 and proto2 are incompatible afaik. pulsar uses a modified version of proto2, so upgrading is more than just changing a dependency
----
2018-06-25 18:48:29 UTC - Ivan Kelly: @Karthikeyan Palanivelu yes, the global zookeeper cluster (renamed configuration cluster in 2.0)
----
2018-06-25 18:51:45 UTC - Sijie Guo: &gt; proto3 and proto2 are incompatible afaik

the binary formats are compatible. only the library is incompatible.
----
2018-06-25 18:53:04 UTC - Sijie Guo: however there is probably no plans to upgrade to proto3 on the broker side, since we did a lot of optimization on reducing the object allocation for java proto2.
----
2018-06-25 18:53:56 UTC - Sijie Guo: but if you write a C# client, you can use the PulsarApi.proto and still using proto3 to generate proto2 code, it will work fine.
----
2018-06-25 18:56:06 UTC - Ivan Kelly: @Sijie Guo really? proto3 has an entirely different concept of whether fields are optional
----
2018-06-25 18:57:45 UTC - dba: @Sijie Guo That sounds great, could you help me out with the bytes for the first frame? I think protobuf is creating the wrong byte array for the frame/command or something like that.....
----
2018-06-25 19:00:02 UTC - Karthik Palanivelu: @Ivan Kelly Can you please help me how I can configure the Global ZK with the zookeepers from other cluster to access in K8s?
----
2018-06-25 19:01:16 UTC - Sijie Guo: @Ivan Kelly proto3 can generate proto2 code.
----
2018-06-25 19:02:10 UTC - Ivan Kelly: @Karthikeyan Palanivelu I've not done it, so I can't really help
----
2018-06-25 19:02:25 UTC - Ivan Kelly: I've taken a note to look into it
----
2018-06-25 19:07:28 UTC - Karthik Palanivelu: Thanks @Ivan Kelly
----
2018-06-25 19:32:21 UTC - Karthik Palanivelu: @Matteo Merli Any suggestion or help configuring the zk across k8s clusters to test geo-replication?
----
2018-06-25 19:59:55 UTC - Sijie Guo: @dba

&gt; That sounds great, could you help me out with the bytes for the first frame? I think protobuf is creating the wrong byte array for the frame/command or something like that.....

did you follow the framing here? <https://pulsar.incubator.apache.org/docs/latest/project/BinaryProtocol/#Simplecommands-extspb> I mean did you prefixed with sizes?
----
2018-06-25 20:08:20 UTC - Sijie Guo: @Karthik Palanivelu: I think it is a non straightforward task to configure a global zookeeper that spans over multiple k8s clusters.

one way I can think of based on you current testing environment (k8s on ec2)

- for each datacenter, you can add labels for the machines used for running zookeeper, and use selector in k8s to select those machines for running zookeeper and use nodePort to expose the port at node level.
- update the zookeeper conf to include those machines and node ports.

However I haven’t tested this approach before.
----
2018-06-25 20:28:08 UTC - Matteo Merli: @Karthikeyan Palanivelu An additional option is to deploy the ZK cluster for global config store outside of K8S.

And finally, just remainder that the global config store is not “strictly” required: you can also manually set the configuration on the namespaces against each cluster. That might be a reasonable option if you only have few namespaces and the configuration does not change frequently.
----
2018-06-25 20:30:00 UTC - Karthik Palanivelu: That’s interesting @Matteo Merli like how we did for standalone to test geo-replication ?
----
2018-06-25 20:31:21 UTC - Karthik Palanivelu: Thanks @Sijie Guo let me try yours and keep you posted on complexity
----
2018-06-25 20:32:31 UTC - Matteo Merli: Yes, it’s very similar, you just need to make sure (“manually”) that the configs are the same for every cluster
----
2018-06-25 20:33:37 UTC - Matteo Merli: Obviously that’s not very convenient if you have multiple tenants or if the config keeps changing, but otherwise is really fine.
----
2018-06-25 20:41:40 UTC - Karthik Palanivelu: Definitely I will be having many tenants. I need to find a easier way than. Based on global ZK, it helps on finding the cluster alone correct? Does it help in any other configuration like namespace creation across clusters? Another thought I have is with this given complexity on ZK and limitation of bookie/node; I feel like going with bare metal implementations. Correct me if I am wrong.
----
2018-06-25 20:49:56 UTC - Sijie Guo: @Karthik Palanivelu:

&gt; limitation of bookie/node

the limitation of bookie per node only exists when using daemon set. you can use statefulset to run bookies, however it requires you setup additional storage for provisioning k8s persistent volumes. stateful is convient when you are using cloud providers k8s services, where those persistent volumes are easier to provision. there is a working effort in k8s to turn on local persistent volume, which I believe it is going beta in k8s 1.10 in April. (I haven’t followed up with the recent 1.11, but I guess it is getting production ready soon).

&gt; given complexity on ZK

yes. zookeeper is not very friendly to k8s environment. there is some setup required for cross k8s clusters (in different data centers). it does like a bare metal implementation for a global zookeeper setup spanning multiple k8s clusters.
----
2018-06-25 20:57:26 UTC - Carlton Hanna: @Carlton Hanna has joined the channel
----
2018-06-25 21:00:18 UTC - Yuwei Jiang: Hi guys, a general question, does Pulsar have MQTT connectors? meaning if I have sensor data, what’s the best way to publish them to Pulsar?
----
2018-06-25 21:00:38 UTC - Karthik Palanivelu: Thanks @Sijie Guo
----
2018-06-25 21:01:15 UTC - Sijie Guo: @Yuwei Jiang we are adding MQTT protocol into pulsar proxy. it is planned for 2.2 release, which will be coming by end of July.
----
2018-06-25 21:02:02 UTC - Sijie Guo: we will have a PIP sent out this week or next week soon. I will point it to you once it is sent out, you can watch the progress then
----
2018-06-25 21:02:59 UTC - Yuwei Jiang: Hi @Sijie Guo much appreciated!
----
2018-06-25 22:02:50 UTC - Martin Kunev: Hi, I am trying to integrate pulsar 2 into an existing application and I'm having some trouble with the REST API. There seem to be contradictions in the documentation. As far as I understand clusters are no longer part of namespace names, but in the documentation (here: <https://pulsar.incubator.apache.org/docs/v2.0.1-incubating/admin-api/namespaces/>) the REST API includes a cluster component. When I try to create a namespace without cluster component I get 405 (which is not documented here: <https://pulsar.incubator.apache.org/docs/v2.0.1-incubating/reference/RestApi/#/admin/namespaces/:tenant/:namespace>). What is the right way to create a namespace?
----
2018-06-25 22:41:18 UTC - chris: @Martin Kunev the docs need to be updated for v2. The rest apis for v2 are available under admin/v2 so to create a namespace you can use admin/v2/namespaces/my-tenant/my-namespace
----
2018-06-26 03:09:49 UTC - Sagar Gaur: @Jon Bock I m trying to build a Real time data Analytics Project for which i want to use Apache pulsar for streaming data &amp; i m confused how i can stream the data with pulsar into Apache Druid...(PS: i doing this for a Online Shopping Site and i m using Scraped data )
----
2018-06-26 03:36:02 UTC - Ali Ahmed: @Sagar Gaur the simplest way to do this would be write a pulsar connector
----
2018-06-26 03:36:50 UTC - Ali Ahmed: you would essentially wrap around tranquility
----
2018-06-26 03:39:54 UTC - Sagar Gaur: @Ali Ahmed can u send me any link or steps on how to create a pulsar connector ?
----
2018-06-26 03:40:50 UTC - Ali Ahmed: here is an example
<https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java>
----
2018-06-26 06:06:31 UTC - dba: @Sijie Guo Yes I did follow the documentation. I must be doing something wrong. Will find a test client and fire up wireshark :slightly_smiling_face:
----
2018-06-26 06:28:21 UTC - Sagar Gaur: @Ali Ahmed it seems like the link u send is using "AWS" is there another way to create a Pulsar connector on the system (laptop) itself...
----
2018-06-26 06:46:33 UTC - Sanjeev Kulkarni: @Sagar Gaur Pulsar IO is intended to connect pulsar to external systems(either pulling data from external sources or getting data out of pulsar to external systems).
----
2018-06-26 06:46:56 UTC - Sanjeev Kulkarni: You can run Pulsar IO either on your laptop or inside a pulsar cluster itself
----
2018-06-26 06:47:18 UTC - Sanjeev Kulkarni: <https://github.com/apache/incubator-pulsar/tree/master/pulsar-io> contains a list of connectors that are available out of box from within github
----
2018-06-26 07:29:23 UTC - Sagar Gaur: @Sanjeev Kulkarni i know that and i have already downloaded all the folders and files from github .What i m asking is how to deploy it..How can i get the scrapped data of online shopping store and stream it to the Apache Druid...
----
2018-06-26 07:29:54 UTC - Sagar Gaur: i have already downloaded the scrap data..
----
2018-06-26 08:19:58 UTC - Bogdan BUNECI: Hi ! I’m trying to consume a topic via websocket consumer API but i receive invalid topic name format.
----
2018-06-26 08:20:52 UTC - Bogdan BUNECI: ```11:13:35.549 [pulsar-web-40-20] WARN org.apache.pulsar.websocket.ConsumerHandler - [127.0.0.1:45974] Failed in creating subscription null on topic <persistent://public/default/my-topic/ws-sub>
java.lang.IllegalArgumentException: Invalid topic name format```
----
2018-06-26 08:22:05 UTC - Sijie Guo: topic seems to be ” <persistent://public/default/my-topic>” and “subscription” seems to be “ws-sub”.

there seems you are using the wrong topic name?
----
2018-06-26 08:22:33 UTC - Bogdan BUNECI: my producer is
----
2018-06-26 08:23:10 UTC - Bogdan BUNECI: `#!/usr/bin/env python3

import pulsar

client = pulsar.Client(‘<pulsar://localhost:6650’>)

producer = client.create_producer(‘my-topic’)

for i in range(10):
producer.send((‘Hello-%d’ % i).encode(‘utf-8’))

client.close()`
----
2018-06-26 08:24:02 UTC - Bogdan BUNECI: consumer
----
2018-06-26 08:24:11 UTC - Bogdan BUNECI: ```#!/usr/bin/env python3

import websocket, base64, json

TOPIC = '<ws://localhost:8080/ws/consumer/persistent/public/default/my-topic/my-sub>'

ws = websocket.create_connection(TOPIC)

print (1)

while True:
msg = json.loads(ws.recv())
if not msg: break

print ("Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))

# Acknowledge successful processing
ws.send(json.dumps({'messageId' : msg['messageId']}))

ws.close()```
----
2018-06-26 08:26:20 UTC - Sijie Guo: @Bogdan BUNECI are you using pulsar 2.0? or 1.22?
----
2018-06-26 08:26:30 UTC - Bogdan BUNECI: 2.0
----
2018-06-26 08:26:52 UTC - Sijie Guo: if you are using pulsar 2.0, try “<ws://localhost:8080/ws/v2/consumer/persistent/public/default/my-topic/my-sub>”
----
2018-06-26 08:26:54 UTC - Bogdan BUNECI: compiled from apache-pulsar-2.0.1-incubating
----
2018-06-26 08:27:12 UTC - Sijie Guo: public/default/my-topic/my-sub is v2 format
----
2018-06-26 08:27:25 UTC - Sijie Guo: &lt;tenant&gt;/&lt;namespace&gt;/&lt;topic&gt;/&lt;subscription&gt;
----
2018-06-26 08:28:02 UTC - Bogdan BUNECI: thanks a lot for your quick and helpful reply !
+1 : Sijie Guo
----
2018-06-26 08:57:15 UTC - Sagar Gaur: @Sijie Guo Sir do u think u can tell me the right way to deploy this withPulsar
----