You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/10/17 09:11:02 UTC
Slack digest for #general - 2018-10-17
2018-10-16 14:42:07 UTC - Martin Svensson: is there any configuration needed to enable reading back from tiered storage e.g. new subscription reading offloaded data?
I haven’t been successful with either the java client or the pulsar-perf consumer
----
2018-10-16 16:05:49 UTC - Rodrigo Malacarne: Hello, in the Pulsar standalone running under Docker is it somehow possible to use the command line to create tenants/namespaces/topics?
----
2018-10-16 16:07:06 UTC - Grant Wu: Is there anything wrong with using the `pulsar` and `pulsar-admin` clients?
----
2018-10-16 16:10:38 UTC - Rodrigo Malacarne: @Grant Wu, sorry for not being so clear... my question is more related to Docker, as I'm not familiarized with it ... my question is how to send commands to the docker container, not Pulsar itself ...
----
2018-10-16 16:10:52 UTC - Grant Wu: ah.
----
2018-10-16 16:11:09 UTC - Grant Wu: How are you running the Docker container? `docker run`?
----
2018-10-16 16:11:17 UTC - Rodrigo Malacarne: Yes ...
----
2018-10-16 16:12:04 UTC - Grant Wu: Okay, then you’ll probably want to expose the ports with `-p` and then you can point the clients at `localhost` (assuming the container is running locally)
----
2018-10-16 16:13:25 UTC - Grant Wu: I believe that’d be `-p 6650:6650 -p 8080:8080`
----
2018-10-16 16:16:02 UTC - Rodrigo Malacarne: @Grant Wu, thank you! I've tested "sudo docker exec 4945dca9fc00 ./bin/pulsar-admin tenants create my-tenant"
----
2018-10-16 16:16:25 UTC - Rodrigo Malacarne: Let me check if I can find the tentant now ...
----
2018-10-16 16:16:28 UTC - Grant Wu: You can also use `docker exec` to run the client inside the docker container, yeah
+1 : Rodrigo Malacarne
----
2018-10-16 16:16:56 UTC - Grant Wu: Although I don’t think you _should_ need sudo?
----
2018-10-16 16:17:20 UTC - Rodrigo Malacarne: @Grant Wu, it asks for sudo somehow ... :disappointed:
----
2018-10-16 16:18:23 UTC - Grant Wu: :confused:
----
2018-10-16 16:18:46 UTC - Rodrigo Malacarne: @Grant Wu, just checked under "<http://192.168.15.96:8080/admin/v2/tenants/my-tenant/>"
----
2018-10-16 16:18:55 UTC - Rodrigo Malacarne: It's there.... :slightly_smiling_face:
----
2018-10-16 16:18:58 UTC - Grant Wu: Note that the ports being exposed needs to happen on container creation
----
2018-10-16 16:18:59 UTC - Rodrigo Malacarne: So it works :slightly_smiling_face:
----
2018-10-16 16:19:35 UTC - Rodrigo Malacarne: Hum ...
----
2018-10-16 16:19:43 UTC - Rodrigo Malacarne: Let me check this too ... Thanks for the advise!
----
2018-10-16 16:21:31 UTC - Rodrigo Malacarne: It was created directly in the "standalone" cluster ... and the standalone cluster itself has the port 8080 and 6650 assigned to it ...
----
2018-10-16 16:22:07 UTC - Rodrigo Malacarne: I believe you have to specify these ports while creating the cluster... not the tenant/namespace/topic ...
----
2018-10-16 16:22:27 UTC - Rodrigo Malacarne: Something like ```./bin/pulsar-admin clusters create pulsar-cluster-1 --url <http://pulsar1:8080> --broker-url <pulsar://pulsar1:6650>```
----
2018-10-16 16:23:05 UTC - Grant Wu: Yes, when I said you need to expose/publish the ports I meant to `docker run`
----
2018-10-16 16:24:07 UTC - Rodrigo Malacarne: @Grant Wu, yes, absolutely !!!
----
2018-10-16 16:24:19 UTC - Rodrigo Malacarne: Thank you very much!
----
2018-10-16 16:52:46 UTC - Igor Zubchenok: Hello guys!
I have an issue with logs, all of them goes to `/var/log/syslog` instead of `logs` folder. Do you have any clue why this could happen?
I have files in `logs` folder, but all of them are empty
```/opt/pulsar/logs # ls -l
total 0
-rw-r--r-- 1 root root 0 Oct 16 00:00 bookkeeper.log
-rw-r--r-- 1 root root 0 Oct 16 00:00 pulsar-broker.log
-rw-r--r-- 1 root root 0 Oct 10 16:07 pulsar.log```
----
2018-10-16 16:55:39 UTC - Grant Wu: How did you install Pulsar?
----
2018-10-16 18:10:27 UTC - Paul Wistrand: @Paul Wistrand has joined the channel
----
2018-10-16 20:52:51 UTC - Igor Zubchenok: I just unpacked tar to `/opt/pulsar` directory
----
2018-10-16 21:02:07 UTC - Jeff Ballard: @Jeff Ballard has joined the channel
----
2018-10-16 21:02:50 UTC - Ali Ahmed: @Igor Zubchenok do you have an env variable defined like ```PULSAR_LOG_DIR=```
----
2018-10-16 21:02:59 UTC - Ali Ahmed: how do you start the services ?
----
2018-10-16 21:05:25 UTC - Igor Zubchenok: No, I don't have PULSAR_LOG_DIR.
I start them as a service `systemctl start pulsar/bookkeeper`
There is systemd service file:
```[Unit]
Description=Apache Pulsar
Documentation=<https://pulsar.incubator.apache.org/>
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/opt/pulsar/bin/pulsar broker
Restart=on-failure
SyslogIdentifier=pulsar
LimitNOFILE=64536
LimitNPROC=8192
[Install]
WantedBy=multi-user.target
```
----
2018-10-16 21:06:39 UTC - Ali Ahmed: @Igor Zubchenok you are piping things to stdout not to log file systemctl is capturing stdout and putting it in `/var/log/syslog`
----
2018-10-16 21:06:50 UTC - Ali Ahmed: it has nothing to do with pulsar
----
2018-10-16 21:08:02 UTC - Igor Zubchenok: I see, sorry, seems its our fault. Could you advice how to fix this?
----
2018-10-16 21:10:15 UTC - Ali Ahmed: you can look into documentation for systemctl to specify service log location or use ```pulsar-daemon``` command to launch a daemon process
----
2018-10-16 21:11:17 UTC - Zuyu Zhang: @Zuyu Zhang has joined the channel
----
2018-10-16 21:11:21 UTC - Ali Ahmed: for the prior option it will be something like this
```
[Service]
Type=forking
ExecStart=/usr/local/bin/binary1 agent -config-dir /etc/sample.d/server
StandardOutput=/var/log1.log
StandardError=/var/log2.log
```
----
2018-10-16 21:38:12 UTC - Igor Zubchenok: Appreciate your reply Ahmed! Thanks!
----
2018-10-16 21:46:47 UTC - Igor Zubchenok: I see broker/bookie started with `-Dlog4j.configurationFile=log4j2.yaml` parameter. So it's not clear why logs aren't stored at logs folder. Is log4j2 config ignored?
----
2018-10-16 21:49:59 UTC - Ali Ahmed: I don’t think that will work
----
2018-10-16 21:50:44 UTC - Ali Ahmed: pulsar folder should have a ./conf/log4j2.yaml you need to override the file
----
2018-10-16 22:09:29 UTC - Nathanial Murphy: I'm currently reading through <http://pulsar.apache.org/docs/latest/project/BinaryProtocol/> and I can't find any details on how Reader starts at an arbitrary position within a given topic/partition. Is this documented anywhere?
----
2018-10-16 22:12:28 UTC - Matteo Merli: @Nathanial Murphy the reader was created after that guide was put together..
In any case, the reader is is piggybacking on the same consumer commands. The only difference, from client perspective is that it specifies 2 additional args when subscribing :
<https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L221>
`durable=false` and `start_message_id=...`
----
2018-10-16 22:13:13 UTC - Nathanial Murphy: Is there any reason you wouldn't have the cursor durability set to true?
----
2018-10-16 22:14:10 UTC - Nathanial Murphy: Looking to use this as an event store, so being able to start at position X and recover my position after a crash would be amazing.
----
2018-10-16 22:15:15 UTC - Matteo Merli: Cursor durability is associated with a Pulsar subscription. Subscription is used to retain data until it gets processed and acknowledged.
----
2018-10-16 22:16:01 UTC - Matteo Merli: In that case, the subscription is managed by the system, while Reader is to allow the flexibility of managing the message id in the application domain
----
2018-10-16 22:16:11 UTC - Nathanial Murphy: Ah, I see where you're going
----
2018-10-16 22:16:54 UTC - Matteo Merli: if you want to make sure the data is retained, without a subscription, you have to set the time based retention policy (possibly with -1 time which means infinite)
----
2018-10-16 22:27:03 UTC - Nathanial Murphy: Just firming up some baseline knowledge to make sure my plans are ok. Please bare with me :sweat_smile:
For every topic, there can be N subscriptions. All of these subscriptions retain separate cursors for what they consider "unacknowledged".
With the above in mind, is there any way to start a subscription at an arbitrary position? Is it just the `start_message_id`?
----
2018-10-16 22:31:07 UTC - Grant Wu: there is an admin command called reset cursor
----
2018-10-16 22:31:18 UTC - Grant Wu: it would be nice to have a way to start a consumer/subscription at an arbitrary time/location
----
2018-10-16 22:31:35 UTC - Grant Wu: I was considering writing a PIP for this as well but work has a lot of other demands on my time
----
2018-10-16 22:32:19 UTC - Matteo Merli: there’s the option to create a new subscription either at the earliest/latest message
----
2018-10-16 22:34:45 UTC - Nathanial Murphy: Could I start the subscription at the earliest message and then just follow up with a cumulative `ack` for the "first" message of the subscription?
----
2018-10-16 22:35:20 UTC - Matteo Merli: sure
----
2018-10-16 22:35:40 UTC - Matteo Merli: I didn’t think of that, but that should work
----
2018-10-16 22:36:18 UTC - Grant Wu: how would you know how far to ack?
----
2018-10-16 22:36:27 UTC - Grant Wu: you’d have to maintain that out of band wouldn’t you
----
2018-10-16 22:37:34 UTC - Matteo Merli: yes, which goes back that maybe you don’t need a subscription, but rather just the Reader
----
2018-10-16 22:38:47 UTC - Nathanial Murphy: Reader + externally maintained cursor it sounds like
----
2018-10-17 00:17:45 UTC - Dexter Bradshaw: @Dexter Bradshaw has joined the channel
----
2018-10-17 08:47:21 UTC - Xiongfei Jin: @Xiongfei Jin has joined the channel
----