You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/04/13 09:11:02 UTC
Slack digest for #general - 2019-04-13
2019-04-12 11:43:51 UTC - Kim Christian Gaarder: @Kim Christian Gaarder has joined the channel
----
2019-04-12 12:13:33 UTC - Kim Christian Gaarder: Pulsar documentation state that ordering guarantees only apply per partition for partitioned topics. I wanted to know whether this applies only to write-ordering, and whether there is still is a read-ordering guarantee here that ensures that messages read by multiple consumers from a partitioned topic will be seen in the same order across partitions, consumers and subscriptions?
----
2019-04-12 12:13:50 UTC - Kim Christian Gaarder: I am investigating using a partitioned Pulsar topic as a transaction log in a distributed system and wanted to know whether this will ensure consistent read-ordering despite that writes across partitions may get re-ordered. i.e. Will multiple exclusive subscriptions to the partioned topic all see messages in the same order, and also if the subscribe and consume at very different times (assuming unlimted message retention)
----
2019-04-12 12:31:48 UTC - Adam F: I'm also interested in using Pulsar as a transaction log.
Looked at the docs quickly and saw this: <https://pulsar.apache.org/docs/latest/cookbooks/PartitionedTopics/#Routingmodes-6b001>
(that info appears to be identical to another page <https://pulsar.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/#Routingmodes-sq0su6>)
----
2019-04-12 12:33:32 UTC - Adam F: At the link, there is a table that lays out 3 different routing modes, each of which provides a different ordering guarantee, ranging from per-message-key bucket ordering to per-producer ordering to no guarantees
----
2019-04-12 12:35:42 UTC - Adam F: (note: I have no practical Pulsar experience, just have been reviewing the docs)
----
2019-04-12 12:39:57 UTC - Adam F: On a related note, this person's blog reports on their experience doing chaos testing with Pulsar and other similar distributed systems, and other posts talk about designing transaction logs, etc; <https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-messages-on-an-apache-pulsar-cluster>
----
2019-04-12 12:42:25 UTC - Adam F: (spoiler alert: "...regarding the claims of zero message loss and ordering guarantees, Apache Pulsar has come out flawless.")
----
2019-04-12 12:46:07 UTC - Adam F: I'm interested to hear more about your investigations @Kim Christian Gaarder -- use-case and other components you're taking a look at
----
2019-04-12 14:33:32 UTC - Roman: @Roman has joined the channel
----
2019-04-12 14:49:07 UTC - Fredrick P Eisele: I am running Pulsar-Dashboard with ...
export SERVICE_URL=<http://ledger2.scope.vu.edu:8080/>
...id does not seem to be connecting to my pulsar cluster.
I look in my broker.conf where...
webServicePort=8080
...any ideas on how to proceed in debugging the problem?
If I point my browser at SERVICE_URL ...
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /. Reason:
<pre> Not Found</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.12.v20180830</a><hr/>
</body>
</html>
----
2019-04-12 14:55:05 UTC - David Kjerrumgaard: @Fredrick P Eisele Can you verify that the service is running on the ledger2 server and that port 8080 is open?
----
2019-04-12 15:24:51 UTC - Fredrick P Eisele: @David Kjerrumgaard Here is the status of the ledger2 service
$ sudo systemctl status pulsar.service
● pulsar.service - Pulsar Broker Service
Loaded: loaded (/etc/systemd/system/pulsar.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-04-11 22:17:23 UTC; 17h ago
Process: 8820 ExecStart=/opt/services/apache-pulsar/latest/bin/pulsar-daemon start broker (code=exited, status=0/SUCCESS)
Main PID: 8836 (java)
Tasks: 229 (limit: 4915)
CGroup: /system.slice/pulsar.service
└─8836 /usr/bin/java -cp /opt/services/apache-pulsar/latest/conf:::/opt/services/apache-pulsar/latest/lib/*: -Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefPr
Apr 11 22:17:20 scope-ledger-2 systemd[1]: Starting Pulsar Broker Service...
Apr 11 22:17:20 scope-ledger-2 pulsar-daemon[8820]: doing start broker ...
Apr 11 22:17:20 scope-ledger-2 pulsar-daemon[8820]: starting broker, logging to /opt/services/apache-pulsar/latest/logs/pulsar-broker-scope-ledger-2.log
Apr 11 22:17:23 scope-ledger-2 systemd[1]: Started Pulsar Broker Service.
----
2019-04-12 15:26:26 UTC - David Kjerrumgaard: @Fredrick P Eisele Can you run a netstat on that host and confirm that process id 8820 is listening on port 8080?
----
2019-04-12 15:36:20 UTC - Fredrick P Eisele: @David Kjerrumgaard The firewall says: ALLOW IPv4 8080/tcp from 0.0.0.0/0 so that looks like it is open and the pro
$ netstat -tulpn | grep 8836
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 8836/java
tcp 0 0 0.0.0.0:6650 0.0.0.0:* LISTEN 8836/java
$ nmap -Pn ledger2
Starting Nmap 7.70 ( <https://nmap.org> ) at 2019-04-12 10:37 CDT
Nmap scan report for ledger2 (129.59.107.170)
Host is up (0.0010s latency).
Not shown: 985 filtered ports
PORT STATE SERVICE
22/tcp open ssh
1720/tcp open h323q931
6000/tcp closed X11
6001/tcp closed X11:1
6002/tcp closed X11:2
6003/tcp closed X11:3
6004/tcp closed X11:4
6005/tcp closed X11:5
6006/tcp closed X11:6
6007/tcp closed X11:7
6009/tcp closed X11:9
6025/tcp closed x11
6059/tcp closed X11:59
8080/tcp open http-proxy
8443/tcp closed https-alt
----
2019-04-12 15:40:27 UTC - Fredrick P Eisele: @David Kjerrumgaard I am running pulsar-dashboard in a docker.
$ docker run -p 80:80 -e SERVICE_URL=$SERVICE_URL apachepulsar/pulsar-dashboard
----
2019-04-12 18:51:58 UTC - Devin G. Bost: I am attempting to migrate our usage of the Admin CLI to the Java Admin API, and I am confused about how to properly setup the `Map<String, Object> configs` property of the `SourceConfig` class.
When creating a source with the Admin CLI, I have a conf file that contains these contents:
```configs:
bootstrapServers: obfuscated
consumerConfigProperties:
auto.offset.reset: latest
sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required doNotPrompt=true
useTicketCache=false serviceName="kafka" principal="pulsar_runtime@obfuscated.COM"
useKeyTab=true keyTab="/pulsar/conf/auth/pulsar_runtime_dev.keytab" client=true;
sasl.kerberos.service.name: kafka
security.protocol: SASL_PLAINTEXT
groupId: obfuscated-log-group
topic: obfuscated-log-topic```
I want to load these into the `Map<String, Object> configs` property of the `SourceConfig` class with the Jackson YAML parser. However, it's not clear how I should handle the hierarchy of the configs. What should the Map<String, Object> contain?
----
2019-04-12 18:55:26 UTC - Devin G. Bost: Perhaps, is there a unit test somewhere with an example?
----
2019-04-12 19:13:59 UTC - David Kjerrumgaard: @Devin G. Bost <https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/test/resources/kafkaSourceConfig.yaml>
----
2019-04-12 19:16:15 UTC - David Kjerrumgaard: I don't think you need the leading `configs:` unless are nesting it inside a larger config that specifies the general source configs such as topic, namespace, parallelism, etc.
----
2019-04-12 19:16:53 UTC - Devin G. Bost: Thanks for the link and guidance!
----
2019-04-12 19:17:07 UTC - Devin G. Bost: That's perfect.
----
2019-04-12 19:54:47 UTC - Sree Vaddi: Apache Heron, Fastest Stream Processing Engine : One Year Anniversary celebrations. you are all most welcome to join.
<https://www.meetup.com/Apache-Heron-Bay-Area/events/xvmlqqyzgbtb/>
----
2019-04-12 22:03:25 UTC - Fredrick P Eisele: What should connecting to the Pulsar HTTP connection look like? Mine looks like this...
<http://ledger2:8080/>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /. Reason:
<pre> Not Found</pre></p><hr><a href="<http://eclipse.org/jetty>">Powered by Jetty:// 9.4.12.v20180830</a><hr/>
</body>
</html>
That does not seem right to me.
----
2019-04-12 22:05:59 UTC - David Kjerrumgaard: @Fredrick P Eisele So are pulsar and the dashboard running in separate docker containers?
----
2019-04-12 22:06:37 UTC - David Kjerrumgaard: @Fredrick P Eisele Is the <http://ledger2.scope.vu.edu|ledger2.scope.vu.edu> server a physical machine?
----
2019-04-12 22:15:36 UTC - Fredrick P Eisele: No, it is a vm on an openstack cloud.
----
2019-04-12 22:17:18 UTC - Fredrick P Eisele: @David Kjerrumgaard They are separate; pulsar is running on an openstack vm and pulsar-dashboard is running in a docker container on my desktop machine, (not on the openstack cloud).
----
2019-04-12 22:22:25 UTC - Fredrick P Eisele: @David Kjerrumgaard What do you get when you point your browser at a Pulsar http port?
(I am trying to take pulsar-dashboard out of the equation.)
----
2019-04-12 22:26:52 UTC - Ali Ahmed: @Fredrick P Eisele it should 404 at the base level
try this “/admin/v2/tenants”
----
2019-04-12 22:39:42 UTC - Fredrick P Eisele: @Ali Ahmed excellent <http://ledger2:8080/admin/v2/tenants> works fine.
----
2019-04-12 22:40:03 UTC - Ali Ahmed: @Fredrick P Eisele are you running master or 2.3 release ?
----
2019-04-12 22:42:04 UTC - Fredrick P Eisele: pulsar is the 2.3.0 release but pulsar-dashboard is <https://github.com/apache/pulsar/tree/master> .
----
2019-04-12 22:43:10 UTC - Ali Ahmed: I would try the 2.3.0 dashboard
----
2019-04-12 22:43:53 UTC - Ali Ahmed: check the ```docker-compose/standalone-dashboard/docker-compose.yml ```
----
2019-04-12 22:44:17 UTC - Ali Ahmed: compose file to see how to use the dashboard container image
----
2019-04-12 22:53:00 UTC - Fredrick P Eisele: @Ali Ahmed thanks I will give that a try. (tomorrow)
----
2019-04-13 08:15:37 UTC - Kevin Brown: @Kevin Brown has joined the channel
----