You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/10/06 09:11:05 UTC
Slack digest for #general - 2020-10-06
2020-10-05 14:08:52 UTC - Ravi Shah: Hi Team -
I can see support for pullMode in the websocket based consumer, Do we have same pullMode based support in Node.js SDk(npm)?
----
2020-10-05 14:26:15 UTC - wim de keyzer: @wim de keyzer has joined the channel
----
2020-10-05 14:38:27 UTC - Jarrod Johnson: Does anyone have any advice on managing message schemas across an organization? I'd like to have a central place for developers to go so that they can easily view topics and their schemas to understand the data that is exposed across the organization. I don't believe the pulsar manager has the ability to view schema. Are there other options that might help with this? Has there ever been discussion about adding this ability to the management UI?
----
2020-10-05 16:14:15 UTC - Marcio Martins: Hey guys, quick question about the Java client... Let's say my client get temporarily disconnected with 1000 produced pending messages, upon re-connection I see the "Re-sending 1000 messages to server" log, but what are the ordering guarantees of these pending messages, will they still be sent in the same order as if the disconnection didn't happen?
----
2020-10-05 16:24:27 UTC - Emanuel: Hi guys anyone knows if there is a plan for functions with multiple topic input. Given we can have basic windows using the state, it could be very useful to be able to subscribe mo many topics for aggregation.
----
2020-10-05 16:40:52 UTC - Addison Higham: multiple topics can already be used, note that the argument is `--inputs` and you can define multiple topics (comma separated I believe on the CLI) or you can use a a yaml file (see the YAML example in this section where inputs is an array: <https://pulsar.apache.org/docs/en/functions-deploy/#parallelism>). Or you can use a regex with the `--topics-pattern` argument
+1 : Emanuel
----
2020-10-05 16:41:09 UTC - Addison Higham: yes, they are sent in the same order
----
2020-10-05 16:43:45 UTC - Addison Higham: pulsar-manager does have some support for schema, it can be found in the "topics" section of the UI, but it is fairly barebones support and is not necessarily great for discovery
If you have some ideas of what you are looking for, adding an issue to the pulsar-manager with some of your needs would be helpful
----
2020-10-05 17:26:15 UTC - Jarrod Johnson: Thanks @Addison Higham. I am guessing this is new in v0.2.0? I'm having some trouble getting that version of the pulsar-manager to connect to any of my environments but as soon as I can get past that I will check out what is there and if it doesn't meet our needs I will submit an issue. Appreciate the info!
----
2020-10-05 17:32:48 UTC - Addison Higham: hrm, I hope I am not leading you astray, I just realized that I was talking about StreamNative's internal fork which we use for our cloud offering which has schema support, but I am not 100% sure if it is in open source cloud-manager. @tuteng would have more context on that
----
2020-10-05 18:43:29 UTC - Joshua Decosta: Is it possible to configure the port for the metrics path?
----
2020-10-05 19:20:30 UTC - Addison Higham: for the broker? it is just served by the admin API, so it is the same port there
----
2020-10-05 19:21:56 UTC - Joshua Decosta: It seems like the metrics endpoint for zookeeper can be configured. Is it not possible for the broker as well?
Also what is the underlying server?
----
2020-10-05 19:25:28 UTC - Addison Higham: zookeeper is unique as the server it starts is *only* for Prometheus. For the broker, the metrtics servlet is embedded in the admin API jetty server
+1 : Joshua Decosta
----
2020-10-05 19:25:43 UTC - Emanuel: Thanks man
----
2020-10-05 19:37:14 UTC - Santosh Kumar: @Santosh Kumar has joined the channel
----
2020-10-05 20:02:32 UTC - Santosh Kumar: Hey guys - I am trying to connect to pulsar-proxy from a client that is not in the same network as that of pulsar. I have an aws alb in front of the pulsar-proxy and connecting to pulsar using the alb. But the connection fails with below error.
```2020-10-05 12:57:08.325 INFO [0x1187f95c0] Client:88 | Subscribing on Topic :<persistent://public/default/cluster-test>
2020-10-05 12:57:08.326 INFO [0x700004073000] HTTPLookupService:237 | Curl Lookup Request sent for <https://pulsar-xyz.com/admin/v2/persistent/public/default/cluster-test/partitions>
2020-10-05 12:57:08.896 INFO [0x700004073000] HTTPLookupService:251 | Response received for url <https://pulsar-xyz.com/admin/v2/persistent/public/default/cluster-test/partitions> code 200
2020-10-05 12:57:08.896 INFO [0x700004073000] HTTPLookupService:295 | parsePartitionData = { LookupDataResult [brokerUrl_ = ] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
2020-10-05 12:57:08.896 INFO [0x700004073000] AckGroupingTrackerEnabled:53 | ACK grouping is enabled, grouping time 100ms, grouping max size 1000
2020-10-05 12:57:08.896 INFO [0x700004073000] HandlerBase:53 | [<persistent://public/default/cluster-test>, subscription_name, 0] Getting connection from pool
2020-10-05 12:57:08.896 INFO [0x700004073000] HTTPLookupService:237 | Curl Lookup Request sent for <https://pulsar-xyz.com/lookup/v2/topic/persistent/public/default/cluster-test>
2020-10-05 12:57:08.996 WARN [0x7000040f6000] AckGroupingTrackerEnabled:99 | Connection is not ready, grouping ACK failed.
2020-10-05 12:57:09.099 WARN [0x7000040f6000] AckGroupingTrackerEnabled:99 | Connection is not ready, grouping ACK failed.
2020-10-05 12:57:09.204 WARN [0x7000040f6000] AckGroupingTrackerEnabled:99 | Connection is not ready, grouping ACK failed.
2020-10-05 12:57:09.290 INFO [0x700004073000] HTTPLookupService:251 | Response received for url <https://pulsar-xyz.com/lookup/v2/topic/persistent/public/default/cluster-test> code 200
2020-10-05 12:57:09.290 INFO [0x700004073000] HTTPLookupService:330 | parseLookupData = { LookupDataResult [brokerUrl_ = <pulsar://pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:6650>] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
2020-10-05 12:57:09.299 INFO [0x700004073000] ConnectionPool:85 | Created connection for
2020-10-05 12:57:09.299 ERROR [0x700004073000] ClientConnection:440 | [<none> -> ] Invalid Url, unable to parse: system:0 Undefined error: 0
2020-10-05 12:57:09.299 INFO [0x700004073000] ClientConnection:1372 | [<none> -> ] Connection closed
2020-10-05 12:57:09.299 INFO [0x700004073000] ClientConnection:235 | [<none> -> ] Destroyed connection
2020-10-05 12:57:09.301 WARN [0x700004073000] AckGroupingTrackerEnabled:99 | Connection is not ready, grouping ACK failed.
Traceback (most recent call last):
File "pulsar_suscriber.py", line 21, in <module>
consumer = client.subscribe(topic, 'subscription-1')
File "/Users/skumar/miniconda3/envs/py383/lib/python3.8/site-packages/pulsar/__init__.py", line 655, in subscribe
c._consumer = self._client.subscribe(topic, subscription_name, conf)
Exception: Pulsar error: ConnectError```
any help please!
----
2020-10-05 20:18:46 UTC - Addison Higham: Hi @Santosh Kumar just to give you some more context of what is happening here:
When you run through a pulsar proxy, it has two main functions:
- handling discovery (aka lookup) requests. When the client wants to use a topic, it needs to know which broker that topic is on, so it asks a lookup request to the proxy which can either process the request itself or forward it onto a random broker (depends on your proxy config)
- after you have discovered the broker that is currently serving a topic, you connect back to the proxy and tell it which broker you want to talk to. As you can see in your logs there, it is returning back the internal broker name. The client doesn't try and connect directly to that hostname, it still connects via the proxy, however, it just sends that broker hostname along and the proxy then knows how to route your connection
I add that context as hopefully it makes the logs easier to understand.
With that context, if we look at like 2-4 in your logs, it appears that we first lookup details about partitioned topic, but this topic is not partitioned, so we get nothing in the response. The second lookup gives us the details of the broker (which looks sane) and then things fail.
What I am guessing based on this is there might be some misconfiguration either in your client or the proxy. Can you share those details?
----
2020-10-05 21:06:19 UTC - Santosh Kumar: thanks @Addison Higham for the detailed reply, it’s helpful.
----
2020-10-05 21:06:26 UTC - Santosh Kumar: Here is the client code -
----
2020-10-05 21:06:31 UTC - Santosh Kumar: ```client = pulsar.Client(pulsar_url,authentication=pulsar.AuthenticationToken(token))
consumer = client.subscribe(topic, 'subscription-name')
msg = consumer.receive()```
----
2020-10-05 21:07:19 UTC - Santosh Kumar: for proxy - we are using default helm chart
----
2020-10-05 21:07:59 UTC - Santosh Kumar: is there any specific configuration, you would like to see in proxy
----
2020-10-05 21:09:33 UTC - Addison Higham: what version is your python client?
----
2020-10-05 21:09:54 UTC - Santosh Kumar: 2.6.1
----
2020-10-05 21:13:33 UTC - Addison Higham: hrm... that is somewhat surprising then. Can you share your `values.yaml` for your helm charts
----
2020-10-05 21:17:51 UTC - Santosh Kumar:
----
2020-10-05 21:18:29 UTC - Santosh Kumar: @Addison Higham ^^
----
2020-10-05 21:21:31 UTC - Addison Higham: ahhh I see, you said you have an ALB in front of pulsar-proxy? ALBs only support http, pulasr-proxy also needs to be able to speak TCP
----
2020-10-05 21:22:01 UTC - Santosh Kumar: I see
----
2020-10-05 21:22:21 UTC - Addison Higham: you can use an NLB with TLS termination
----
2020-10-05 21:22:24 UTC - Santosh Kumar: I thought HTTP port should handle it
----
2020-10-05 21:22:53 UTC - Santosh Kumar: If using NLB then I can use port 6650 which has pulsar protocol
----
2020-10-05 21:22:53 UTC - Santosh Kumar: tcp
----
2020-10-05 21:23:15 UTC - Santosh Kumar: what's the use of http then
----
2020-10-05 21:23:39 UTC - Santosh Kumar: is HTTP for specific purpose like admin jobs
----
2020-10-05 21:23:49 UTC - Addison Higham: you need both TCP 6650 and HTTP 8080. HTTP is used for all the admin calls (and can be used for discovery, hence why discovery worked). TCP is for actual messaging protocol
----
2020-10-05 21:24:02 UTC - Santosh Kumar: got it
----
2020-10-05 21:24:09 UTC - Santosh Kumar: thanks for looking into it
----
2020-10-05 21:25:12 UTC - Addison Higham: so the things I mentioned about pulsar proxy, that is all for TCP, but it *also* proxies the admin API, but that is mostly a straight forward HTTP proxy (but also needs to follow re-directs)
+1 : Santosh Kumar
----
2020-10-06 02:49:28 UTC - Alan Broddle: We are trying to follow the latest documentation for the new Pulsar Manager 0.2.0 from the github directions. <https://github.com/apache/pulsar-manager>
We started standalone Pulsar and the Pulsar Manager exactly as stated in the README.md.
The pulsar Manager login comes up at the url we expect, but when we try to log in, we get a *“The username or password is incorrect”*
```docker pull apachepulsar/pulsar:latest
docker run -d -it \
-p 6650:6650 \
-p 8080:8080 \
-v pulsardata:/pulsar/data \
-v pulsarconf:/pulsar/conf \
--name pulsar-standalone \
apachepulsar/pulsar:latest \
bin/pulsar standalone```
```docker pull apachepulsar/pulsar-manager:v0.2.0
docker run -it \
-p 9527:9527 -p 7750:7750 \
-e SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties \
--link pulsar-standalone \
apachepulsar/pulsar-manager:v0.2.0```
GUI URL:
<http://localhost:9527/#/login?redirect=%2F>
----
2020-10-06 02:51:24 UTC - tuteng: Please init username and password by the command
```CSRF_TOKEN=$(curl <http://localhost:7750/pulsar-manager/csrf-token>)
curl \
-H 'X-XSRF-TOKEN: $CSRF_TOKEN' \
-H 'Cookie: XSRF-TOKEN=$CSRF_TOKEN;' \
-H "Content-Type: application/json" \
-X PUT <http://localhost:7750/pulsar-manager/users/superuser> \
-d '{"name": "admin", "password": "apachepulsar", "description": "test", "email": "<ma...@test.org>"}'```
----
2020-10-06 02:53:19 UTC - Ali Ahmed: <https://medium.com/@gillesbarbier/building-an-event-driven-orchestration-engine-bf62d45aef5d>
----
2020-10-06 07:24:08 UTC - Enrico: How i can fix it???
----