You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/08/23 09:11:02 UTC
Slack digest for #general - 2018-08-23
2018-08-22 15:07:53 UTC - Ravi: already did this on command. same with streamlio
----
2018-08-22 16:10:36 UTC - Ravi: did u check
----
2018-08-22 16:11:31 UTC - Ravi: @Matteo Merli please let me know
----
2018-08-22 17:33:16 UTC - Ali Ahmed: I think it should be the same I wll try to find out
----
2018-08-22 18:58:56 UTC - chris: @Ravi
Which streamlio image are you referring to? the streamlio/sandbox:latest?
In that image the pulsar data dir is /pulsar/data
you should be able to bind a mount with this command
docker run -d \
--name streamlio-sandbox \
-p 6650:6650 \
-p 8080:8080 \
-p 8000:8000 \
-v <full-path-to-local-dir>:/pulsar/data \
streamlio/sandbox
another was is have docker create a volume for you
<https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume>
docker run -d \
--name streamlio-sandbox \
-p 6650:6650 \
-p 8080:8080 \
-p 8000:8000 \
-v myvol:/pulsar/data \
streamlio/sandbox
----
2018-08-22 18:59:25 UTC - Ravi: did not work, i tried both
----
2018-08-22 18:59:32 UTC - Ravi: yes latest
----
2018-08-22 18:59:42 UTC - Ravi: i tried via compose and a straight run
----
2018-08-22 19:00:25 UTC - chris: what is the error you are getting?
----
2018-08-22 19:00:40 UTC - Ravi: pulsar bind keeps quitting after 1 sec
----
2018-08-22 19:00:50 UTC - Ravi: i posted that already
----
2018-08-22 19:06:20 UTC - Ravi: docker run -d --name streamlio-sandbox1 -p 6650:6650 -p 192.168.1.33:8081:8080 -p 192.168.1.33:8001:8000 -v /appvol/streamlio/data:/pulsar/data streamlio/sandbox
----
2018-08-22 19:09:29 UTC - Ravi: 2018-08-22 19:08:38.380 INFO ConnectionPool:63 | Created connection for <pulsar://localhost:6650>
2018-08-22 19:08:38.381 INFO ClientConnection:285 | [127.0.0.1:39650 -> 127.0.0.1:6650] Connected to broker
2018-08-22 19:08:38.381 INFO ClientConnection:1182 | [127.0.0.1:39650 -> 127.0.0.1:6650] Connection closed
2018-08-22 19:08:38.381 ERROR ClientImpl:267 | Error Checking/Getting Partition Metadata while Subscribing- 5
2018-08-22 19:08:38.381 INFO ClientConnection:195 | [127.0.0.1:39650 -> 127.0.0.1:6650] Destroyed connection
Traceback (most recent call last):
File "consumer.py", line 26, in <module>
main(sys.argv)
File "consumer.py", line 13, in main
consumer = client.subscribe(WORD_COUNT_TOPIC, SUBSCRIPTION)
File "/usr/local/lib/python2.7/site-packages/pulsar/__init__.py", line 470, in subscribe
c._consumer = self._client.subscribe(topic, subscription_name, conf)
Exception: Pulsar error: ConnectError
----
2018-08-22 19:17:24 UTC - Karthik Ramasamy: @Ravi - we are trying to see if we can reproduce it ?
----
2018-08-22 19:17:29 UTC - chris: are you running this locally or in aws?
----
2018-08-22 19:17:51 UTC - Ravi: now m getting it both on local and aws
----
2018-08-22 19:18:11 UTC - Ravi: i tried creating it using a docker compose with own network, that failed
----
2018-08-22 19:18:26 UTC - Ravi: then tried run from command that hit it as well
----
2018-08-22 19:18:54 UTC - Karthik Ramasamy: essentially you are starting the sandbox
----
2018-08-22 19:19:04 UTC - Ravi: another i noticed it when i map volumes it was creating a root directory than a user permissioned dir
----
2018-08-22 19:19:06 UTC - Karthik Ramasamy: and trying to connect a producer and consumer from the outside
----
2018-08-22 19:19:18 UTC - Ravi: yes @Karthik Ramasamy
----
2018-08-22 19:19:51 UTC - Ravi: all of these are running on my neo4j vm, i needed the sandbox there for my email ingestion poc
----
2018-08-22 19:20:25 UTC - Ravi: i cannot bind 8080 or 8000 as they are already used, so i bound another port for local to container 8080 and 8000
----
2018-08-22 19:20:40 UTC - Ravi: gave the security inbound open in aws
----
2018-08-22 19:20:51 UTC - Karthik Ramasamy: ok
----
2018-08-22 19:22:53 UTC - Karthik Ramasamy: do you have the sample program?
----
2018-08-22 19:23:56 UTC - Karthik Ramasamy: i got the docker up
----
2018-08-22 19:23:59 UTC - Ravi: i m using the same consumer.py to test
----
2018-08-22 19:24:01 UTC - Karthik Ramasamy: with the following command
----
2018-08-22 19:24:03 UTC - Karthik Ramasamy: docker run -d --name streamlio-sandbox1 -p 6650:6650 -p 8081:8080 -p 8001:8000 -v /tmp/streamlio/data:/pulsar/data streamlio/sandbox
----
2018-08-22 19:24:11 UTC - Ravi: docker run -d --name streamlio-sandbox -p 6650:6650 -p 8081:8081 -p 8000:8000 streamlio/sandbox
----
2018-08-22 19:24:16 UTC - Karthik Ramasamy: in the tutorial?
----
2018-08-22 19:24:28 UTC - Ravi: yep, thats what i used to test
----
2018-08-22 19:24:34 UTC - Ravi: docker is up
----
2018-08-22 19:24:42 UTC - Ravi: but when i run the consumer.py
----
2018-08-22 19:24:46 UTC - Ravi: it crashes
----
2018-08-22 19:25:04 UTC - Ravi: with error clientimpl:267
----
2018-08-22 19:25:13 UTC - Karthik Ramasamy: which consumer.py?
----
2018-08-22 19:25:19 UTC - Karthik Ramasamy: the one in the tutorial?
----
2018-08-22 19:25:30 UTC - Ravi: yep
----
2018-08-22 19:25:56 UTC - Ravi: it worked on my mac, now its crashing there as well
----
2018-08-22 19:26:58 UTC - Karthik Ramasamy: just checking the version of the image vs pip install
----
2018-08-22 19:27:18 UTC - Ravi: it was py 2.7
----
2018-08-22 19:27:34 UTC - Karthik Ramasamy: i mean pulsar version
----
2018-08-22 19:27:36 UTC - Ravi: we had an issue with my mac being on 3.x py
----
2018-08-22 19:27:37 UTC - Ravi: oh ok
----
2018-08-22 19:28:07 UTC - Ali Ahmed: what’s the pulsar client version installed ?
----
2018-08-22 19:28:28 UTC - Ravi: i believe it was 2.01
----
2018-08-22 19:28:39 UTC - Ali Ahmed: can you double check
----
2018-08-22 19:28:39 UTC - Karthik Ramasamy: @Ravi - do you want to try the following instructions?
----
2018-08-22 19:28:40 UTC - Karthik Ramasamy: <https://pulsar.incubator.apache.org/docs/en/standalone-docker/>
----
2018-08-22 19:29:20 UTC - Ravi: i tried that already
----
2018-08-22 19:29:46 UTC - Karthik Ramasamy: @Ali Ahmed - is taking a look
----
2018-08-22 19:30:01 UTC - Ravi: docker run -it \
-p 192.168.1.33:6650:6650 \
-p 192.168.1.33:8010:8010 \
-v /appvol/streamlio/data:/pulsar/data \
apachepulsar/pulsar:2.0.1-incubating \
bin/pulsar standalone
----
2018-08-22 19:30:05 UTC - Ravi: thats what i used
----
2018-08-22 19:30:27 UTC - Ravi: because this is on a private subnet i have to bind to specific ip
----
2018-08-22 19:31:10 UTC - Ravi: @Karthik Ramasamy i need to show this working to customer via email ingestion next week :disappointed:
----
2018-08-22 19:31:18 UTC - Ravi: do you hv a aws ami ?
----
2018-08-22 19:31:26 UTC - Ali Ahmed: ```
pip list | grep pulsar
pulsar-client 2.1.0
```
----
2018-08-22 19:31:57 UTC - Ali Ahmed: for me running the container locally with producer consumer , everything works fine
----
2018-08-22 19:33:25 UTC - Ali Ahmed: here is the container images I am using
```
streamlio/sandbox latest dca06028a146
```
----
2018-08-22 19:33:25 UTC - Karthik Ramasamy: @Ravi - let us take one step at a time
----
2018-08-22 19:34:03 UTC - Karthik Ramasamy: can you run in a local machine?
----
2018-08-22 19:34:12 UTC - Ravi: hmm yea need to get this working.
----
2018-08-22 19:34:18 UTC - Ravi: local is also crashing
----
2018-08-22 19:34:24 UTC - Karthik Ramasamy: @Ali Ahmed - is checking on the pulsar docker now
----
2018-08-22 19:34:45 UTC - Karthik Ramasamy: we will get it working - it should be more an env issue
----
2018-08-22 19:34:49 UTC - Karthik Ramasamy: or some version issue
----
2018-08-22 19:34:58 UTC - Ravi: docker run -itd \
-p 192.168.1.33:6650:6650 \
-p 192.168.1.33:8010:8010 \
-v $PWD/data:/pulsar/data \
apachepulsar/pulsar:2.0.1-incubating \
bin/pulsar standalone
----
2018-08-22 19:35:08 UTC - Ravi: that is what i m going to run for now
----
2018-08-22 19:36:27 UTC - Ravi: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3722add7837e apachepulsar/pulsar:2.0.1-incubating "bin/pulsar standalo…" 3 seconds ago Up 2 seconds 192.168.1.33:6650->6650/tcp, 192.168.1.33:8010->8010/tcp reverent_engelbart
----
2018-08-22 19:36:39 UTC - Ravi: now i need to test the pulsar
----
2018-08-22 19:36:58 UTC - Karthik Ramasamy: we are just following the instructions in pulsar docker to get it up
----
2018-08-22 19:38:06 UTC - Ravi: i m as well
----
2018-08-22 19:39:30 UTC - Ali Ahmed: can you check do you have a zombie python process running ?
----
2018-08-22 19:41:13 UTC - Ravi: no
----
2018-08-22 19:45:05 UTC - Karthik Ramasamy: ok we followed the pulsar OSS docker instructions and it works fine
----
2018-08-22 19:45:32 UTC - Karthik Ramasamy: can you check the ```pip list | grep pulsar``` ?
----
2018-08-22 19:45:43 UTC - Karthik Ramasamy: and see the version
----
2018-08-22 19:46:08 UTC - Ravi: $ netstat -tupln|grep 8010
(No info could be read for "-p": geteuid()=1007 but you should be root.)
tcp 0 0 192.168.1.33:8010 0.0.0.0:* LISTEN -
$ pip list | grep pulsar
pulsar-client 2.1.0
$
----
2018-08-22 19:46:15 UTC - Ali Ahmed: I can get everything running fine don’t see any issues
----
2018-08-22 19:46:31 UTC - Ravi: :slightly_smiling_face: which makes this super weird
----
2018-08-22 19:46:42 UTC - Ravi: initially i thought it was my compose file
----
2018-08-22 19:46:54 UTC - Ravi: can u try my compose file and see if that works ?
----
2018-08-22 19:47:01 UTC - Karthik Ramasamy: the only difference is the subnet
----
2018-08-22 19:47:05 UTC - Karthik Ramasamy: please send it
----
2018-08-22 19:47:19 UTC - Ali Ahmed: the docker image you are using is 2.0.1
```
Ravi.giggso [11 minutes ago]
docker run -itd \
-p 192.168.1.33:6650:6650 \
-p 192.168.1.33:8010:8010 \
-v $PWD/data:/pulsar/data \
apachepulsar/pulsar:2.0.1-incubating \
bin/pulsar standalone
```
----
2018-08-22 19:47:24 UTC - Ravi: version: '2'
services:
pulsar:
image: "apachepulsar/pulsar:2.0.1-incubating"
ports:
- "6650:6650"
- "8010:8010"
- "8001:8001"
networks:
- streamsnet
tty: true
networks:
streamsnet:
driver: bridge
----
2018-08-22 19:47:40 UTC - Ali Ahmed: that’s mis matched with the client
----
2018-08-22 19:48:00 UTC - Ali Ahmed: you need to upgrade the docker image to 2.1.0
----
2018-08-22 19:48:01 UTC - Ravi: dont think thats the issue, but we can change it to latest
----
2018-08-22 19:48:09 UTC - Ali Ahmed: or downgrade the client
----
2018-08-22 19:48:46 UTC - Ravi: doing it now
----
2018-08-22 19:49:24 UTC - Ravi: 5d14822b5101 apachepulsar/pulsar:latest "bin/pulsar standalo…" 6 seconds ago Up 6 seconds 192.168.1.33:6650->6650/tcp, 192.168.1.33:8010->8010/tcp amazing_beaver
----
2018-08-22 19:49:59 UTC - Ravi: running this
----
2018-08-22 19:50:00 UTC - Ravi: $ cat pulsarconsumer.py
import pulsar
client = pulsar.Client('<pulsar://192.168.1.33:6650>')
consumer = client.subscribe('my-topic',
subscription_name='my-sub')
while True:
msg = consumer.receive()
print("Received message: '%s'" % msg.data())
consumer.acknowledge(msg)
client.close()
----
2018-08-22 19:50:29 UTC - Ravi: 2018-08-22 19:50:17.669 INFO ConnectionPool:63 | Created connection for <pulsar://192.168.1.33:6650>
2018-08-22 19:50:17.670 ERROR ClientConnection:329 | [<none> -> <pulsar://192.168.1.33:6650>] Failed to establish connection: Connection refused
2018-08-22 19:50:17.670 INFO ClientConnection:1182 | [<none> -> <pulsar://192.168.1.33:6650>] Connection closed
2018-08-22 19:50:17.670 ERROR ClientImpl:267 | Error Checking/Getting Partition Metadata while Subscribing- 5
2018-08-22 19:50:17.670 INFO ClientConnection:195 | [<none> -> <pulsar://192.168.1.33:6650>] Destroyed connection
Traceback (most recent call last):
File "pulsarconsumer.py", line 5, in <module>
subscription_name='my-sub')
----
2018-08-22 19:50:55 UTC - Ali Ahmed: can you check
```
ps aux | grep consumer
```
----
2018-08-22 19:51:25 UTC - Ali Ahmed: I suspect some prior consumer process is still running
----
2018-08-22 19:51:26 UTC - Ravi: $ sudo netstat -tupln |grep 6650
tcp 0 0 192.168.1.33:6650 0.0.0.0:* LISTEN 28931/docker-proxy
----
2018-08-22 19:51:43 UTC - Ravi: i did docker system prune before starting
----
2018-08-22 19:52:20 UTC - Ali Ahmed: but consumer process is in your local machine ?
----
2018-08-22 19:53:00 UTC - Ravi: all are in local for testing
----
2018-08-22 19:54:56 UTC - Ravi: @Ali Ahmed can u test my compose file
----
2018-08-22 19:56:04 UTC - Ravi: also can you check if /pulsar/data is owned by root or local user?
----
2018-08-22 19:57:14 UTC - Ali Ahmed: ```
version: '2'
services:
pulsar:
image: "apachepulsar/pulsar:2.1.0-incubating"
ports:
- "6650:6650"
- "8010:8010"
- "8001:8001"
networks:
- streamsnet
tty: true
networks:
streamsnet:
driver: bridge
```
----
2018-08-22 19:57:18 UTC - Ali Ahmed: you mean this ?
----
2018-08-22 19:57:20 UTC - Ravi: yes
----
2018-08-22 20:01:58 UTC - Ravi: it is a tcp port right?
----
2018-08-22 20:02:37 UTC - Ali Ahmed: yes but the compose isn’t correct
----
2018-08-22 20:02:54 UTC - Ali Ahmed: it doesn’t do anything
----
2018-08-22 20:02:56 UTC - Ravi: why what is the issue
----
2018-08-22 20:03:01 UTC - Ravi: it will roll in
----
2018-08-22 20:03:05 UTC - Ali Ahmed: you need this
```
version: '2'
services:
pulsar:
image: "apachepulsar/pulsar:2.1.0-incubating"
ports:
- "6650:6650"
- "8010:8010"
- "8001:8001"
command: >
/bin/bash -c
"bin/apply-config-from-env.py conf/standalone.conf
&& bin/pulsar standalone"
networks:
- streamsnet
networks:
streamsnet:
driver: bridge
```
----
2018-08-22 20:03:15 UTC - Ravi: change 2.1 to latest
----
2018-08-22 20:03:26 UTC - Ravi: you can remove the command:
----
2018-08-22 20:03:28 UTC - Ali Ahmed: 2.1 is latest you are missing the command
----
2018-08-22 20:03:41 UTC - Ravi: thats ok, we can make 2.1 as latest
----
2018-08-22 20:03:49 UTC - Ali Ahmed: the above compose is working for me
----
2018-08-22 20:03:52 UTC - Ravi: i m trying to make it exactly same as mine
----
2018-08-22 20:04:07 UTC - Ravi: ok let me run this
----
2018-08-22 20:04:24 UTC - Ravi: do you have a consumer / producer py sample
----
2018-08-22 20:04:26 UTC - Ali Ahmed: just copy what I have pasted above
----
2018-08-22 20:04:50 UTC - Ali Ahmed: ```
import pulsar
import random
from random import randint
from time import sleep
import sys
SENTENCES_TOPIC = 'sentences'
# This iterates continuously through a list sequence in random order
def random_cycle(ls):
local_ls = ls[:] # create defensive copy
while True:
random.shuffle(local_ls)
for e in local_ls:
yield e
def main(args):
# Create a pulsar client instance with reference to the broker
client = pulsar.Client('<pulsar://localhost:6650>')
# Build a producer instance on a specific topic
producer = client.create_producer(SENTENCES_TOPIC)
# Collection of sentences to serve as random input sequence
sentences = random_cycle([
"the cow jumped over the moon",
"an apple a day keeps the doctor away",
"four score and seven years ago",
"snow white and the seven dwarfs",
"i am at two with nature"
])
for sentence in sentences:
sleep(0.05) # Throttle messages with a 50 ms delay
print('Sending sentence: %s' % sentence)
producer.send(sentence.encode('utf-8')) # Publish randomly selected sentence to Pulsar
client.close()
if __name__ == '__main__':
main(sys.argv)
```
----
2018-08-22 20:05:21 UTC - Ali Ahmed: ```
import pulsar
import sys
import json
WORD_COUNT_TOPIC = 'sentences'
SUBSCRIPTION = 'my-sub'
TIMEOUT = 10000
def main(args):
# Create a pulsar client instance with reference to the broker
client = pulsar.Client('<pulsar://localhost:6650>')
consumer = client.subscribe(WORD_COUNT_TOPIC, SUBSCRIPTION)
while True:
try:
# try and receive messages with a timeout of 10 seconds
msg = consumer.receive(timeout_millis=TIMEOUT)
print("Received message: %s" % msg.data())
consumer.acknowledge(msg) # send ack to pulsar for message consumption
except Exception:
print("No message received in the last 10 seconds")
client.close()
if __name__ == '__main__':
main(sys.argv)
```
----
2018-08-22 20:05:41 UTC - Ravi: these are same as your wget files
----
2018-08-22 20:06:32 UTC - Ali Ahmed: they are from the sandbox but consumer has been changed
----
2018-08-22 20:09:17 UTC - Ravi: Ahmed!
----
2018-08-22 20:09:22 UTC - Ravi: the compose worked!
----
2018-08-22 20:13:54 UTC - Ravi: actually it fails if i run it on background with a -d option
----
2018-08-22 20:14:03 UTC - Ravi: daemon is not working @Ali Ahmed
----
2018-08-22 21:40:51 UTC - Ali Ahmed: not sure about the daemon I will try it later but are you blocked by that ?
----
2018-08-22 23:03:02 UTC - Ravi: yes
----
2018-08-22 23:03:11 UTC - Ravi: but i will use the regular
----
2018-08-23 07:21:18 UTC - Matti-Pekka Laaksonen: I am trying to install a Pulsar cluster on "bare metal" (virtual machines on Openstack). I've been following the documentation and I've modified the Ansible playbooks to suit my use case a bit better. I can get the Zookeeper process working just fine, but the cluster metadata initialization doesn't work. I don't get any error messages, the process simply hangs
----
2018-08-23 07:22:05 UTC - Matti-Pekka Laaksonen:
----
2018-08-23 07:22:54 UTC - Matti-Pekka Laaksonen:
----
2018-08-23 07:23:11 UTC - Matti-Pekka Laaksonen: I'm using Pulsar version 2.1.0
----
2018-08-23 07:24:05 UTC - Matti-Pekka Laaksonen: Do you have any tips where I should begin looking into this? Does the broker need to be accessible when initializing the metadata? Currently I have not yet installed and started the broker/bookie services
----
2018-08-23 07:24:36 UTC - Matti-Pekka Laaksonen: I also tried this with a domain name instead of the private IP address for the service urls
----
2018-08-23 07:26:29 UTC - penghui li: Try to telnet 10.60.1.16:2181
----
2018-08-23 07:29:37 UTC - Matti-Pekka Laaksonen: Connection to 10.60.1.16 2181 port [tcp/*] succeeded!
----
2018-08-23 07:29:44 UTC - Matti-Pekka Laaksonen: With netcat, that is
----
2018-08-23 07:40:09 UTC - Matti-Pekka Laaksonen: Can I initialize the metadata if the rest of the cluster doesn't exist and the service addressess don't respond or resolve to anything?
----
2018-08-23 08:20:22 UTC - Ali Ahmed: is zookeeper healthy and ready for write operations ?
----
2018-08-23 08:40:22 UTC - Matti-Pekka Laaksonen: If I type "stats" in the telnet session, I get "This ZooKeeper instance is not currently serving requests"
----
2018-08-23 08:40:28 UTC - Matti-Pekka Laaksonen: So I guess not
----
2018-08-23 08:40:48 UTC - Matti-Pekka Laaksonen: Is there any other health check for Zookeeper, similar to the bookie sanity check?
----
2018-08-23 08:41:04 UTC - Ali Ahmed: zookeeper does not have a quorum
----
2018-08-23 08:41:35 UTC - Ali Ahmed: you should check whether zookeeper nodes can reach each other from there hosts
----
2018-08-23 08:43:13 UTC - Matti-Pekka Laaksonen: Thank you! It seems I can not telnet to the other Zookeeper nodes from another Zookeeper node. My first guess this has something to do with how I've setup the security groups (firewall rules in Openstack)
----
2018-08-23 08:44:13 UTC - Ali Ahmed: there is nothing specific about pulsar zookeeper you can cannot to any working zookeeper instance
----
2018-08-23 08:45:56 UTC - Matti-Pekka Laaksonen: Alright, this helps with Googling. This is my first time working with Zookeeper of any kind
----
2018-08-23 08:51:57 UTC - Matti-Pekka Laaksonen: Ok. I have currently only allowed the port 2888 and 3888 for intra-cluster communication for Zookeeper nodes. Do I need to allow port 2181 also?
----
2018-08-23 08:52:51 UTC - Matti-Pekka Laaksonen: I thought users from outside the cluster use port 2181 and Zookeeper nodes use 2888 and 3888 when communicating with each other
----
2018-08-23 09:08:42 UTC - Matti-Pekka Laaksonen: And on further inspection it seems that I have _not_ allowed the ports 2888 and 3888. I'll fix them, but do I still need to allow 2181 also?
----