You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/03/31 09:11:02 UTC
Slack digest for #general - 2019-03-31
2019-03-30 12:53:42 UTC - Ben S: I have an eventually strange use case for pulsar (actually I have several, but this one is strange), and I hope you can help me with the configuration. I need to deliver current safety information to possibly slow consumers (connected via wifi and possibly moving around). I thought a bit about the actual problem and I think what I want is the slow clients to drop all non-acknowledged old messages and only consume the most recent one. Thus what I want is message delivery guaranteed for the most recent message, though if there is another message inflight the client should drop all old messages. Is it possible to configure the pulsar client and server for this scenario?
----
2019-03-30 16:12:58 UTC - David Kjerrumgaard: Hi @Vikas Can you share your logs with me?
----
2019-03-30 18:33:51 UTC - Matteo Merli: Would the TTL expiration address this problem?
----
2019-03-30 18:37:29 UTC - Ben L: @Ben L has joined the channel
----
2019-03-30 19:08:31 UTC - Ben L: Hey all, I'm considering using Pulsar for a possibly abnormal use case and have a few questions. I'd like to create a stream of binary chunks of data (maximum 4MiB in size) that is geo-replicated. The chunks will be consumed into a specialized storage and indexing layer. I don't require any processing of the chunks in the streaming system as they're opaque to this part of the system, but I do need a substantial throughput of roughly 400MiB/s on average across approximately 10 million topics. The producers are occasionally very bursty and I want pulsar to act as a very large buffer for this. If this sounds absolutely insane you can stop reading here and let me know. Now onto some more concrete questions:
- How does pulsar behave with messages of this size? Are there any particular tunables I should be aware of? I know there is a 5MiB cap on message size by default.
- Is it possible to create a shared subscription where all subscribers receive every message? This is close to the shared subscription mode but not quite the same, I'm not sure how this conflicts with the exactly once guarantees etc.
- What gets slower as I add more topics and how many topics can I expect to create i.e. 1,10,100 million?
- Is it possible for the producer to made aware of how big the queue is getting and slow itself down accordingly? (I roughly control the production rate in the system)
----
2019-03-30 22:44:44 UTC - Zarrar: Hi, I am trying to run the functions using KubernetesContainerFactory, but when I enable authentication
the pods fails to start with the following error. The token passed to the pod is for a
a super user defined in the broker.
```
HTTP 401 Authentication required
Reason: HTTP 401 Authentication required
```
Pod command
```
command:
- sh
- -c
- /pulsar/bin/pulsar-admin --admin-url <http://pulsar-broker:8080> functions download
--path tenant-test/ns1/reverse2/1fdda06d-8c0b-4781-b7e8-082aeec81f4d-reverse.py
--destination-file /pulsar/reverse.py && SHARD_ID=${POD_NAME##*-} && echo shardId=${SHARD_ID}
&& PYTHONPATH=${PYTHONPATH}:/pulsar/instances/deps python /pulsar/instances/python-instance/python_instance_main.py
--py /pulsar/reverse.py --logging_directory logs/functions --logging_file reverse2
--logging_config_file /pulsar/conf/functions-logging/console_logging_config.ini
--install_usercode_dependencies True --instance_id $SHARD_ID --function_id f4cb9572-3244-4c1d-99cf-d7da18976653
--function_version 70fab446-e49b-4af6-851f-7556e3b3d1ff --function_details '{"tenant":"tenant-test","namespace":"ns1","name":"reverse2","className":"reverse","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"inputSpecs":{"<persistent://tenant-test/ns1/backwards>":{}},"cleanupSubscription":true},"sink":{"topic":"<persistent://tenant-test/ns1/forwards>"}}'
--pulsar_serviceurl <pulsar://pulsar-broker:6650> --client_auth_plugin org.apache.pulsar.client.impl.auth.AuthenticationToken
--client_auth_params token:redacted
--use_tls false --tls_allow_insecure false --hostname_verification_enabled false
--max_buffered_tuples 1024 --port 9093 --metrics_port 9094 --expected_healthcheck_interval
-1 --secrets_provider secretsprovider.ClearTextSecretsProvider --cluster_name
pulsar
```
----
2019-03-30 22:48:09 UTC - Matteo Merli: @Ben L
> - How does pulsar behave with messages of this size? Are there any particular tunables I should be aware of? I know there is a 5MiB cap on message size by default.
Having big messages leads to a higher throughput in general. There are few options that would make sense to tune and these are typically on the queue sizes (for producer, consumers and replication). These options let you control the max amount of memory in each component.
> - Is it possible to create a shared subscription where all subscribers receive every message? This is close to the shared subscription mode but not quite the same, I’m not sure how this conflicts with the exactly once guarantees etc.
You can just create multiple subscriptions (each with a different name)
> - What gets slower as I add more topics and how many topics can I expect to create i.e. 1,10,100 million?
There are Pulsar deployments with 3M topics and geo-replication. I think it would be possible to get to 10M, though with current implementation 100M is way out of reach.
The main concern comes from the amount of metadata to be stored. Also from the amount of memory involved to keep track of all the topics/consumers.
In general, most of the times it’s easier to adjust the design and deal with some degree of multiplexing, rather than tackling the harder problems.
> - Is it possible for the producer to made aware of how big the queue is getting and slow itself down accordingly? (I roughly control the production rate in the system)
Kind of, it’s a big black or white control. There is a backlog quota enforcement. Once the quota is reached on a topic, the producer will get blocked until the storage goes below quota.
----
2019-03-30 23:31:49 UTC - Vikas: hey @David Kjerrumgaard , we are able to troubleshoot the issue. Its working fine now. thank you.
----
2019-03-30 23:43:18 UTC - David Kjerrumgaard: @Vikas thats great news. Thanks for the update
----
2019-03-31 00:07:12 UTC - Jerry Peng: @Zarrar Currently function authentication is not very user friendly and is not really support in kubernetes. A more generic and user friendly implementation of authentication and authorization was just added by me i.e.
<https://github.com/apache/pulsar/pull/3735>
<https://github.com/apache/pulsar/pull/3874>
they will officially be released for 2.4 but feel free to compile master to try it out yourself. Function auth should work for process and threaded mode but for kubernetes a piece is missing in 2.3. The reason why its failing is because the initial download command i.e.
```
/pulsar/bin/pulsar-admin --admin-url <http://pulsar-broker:8080> functions download
--path tenant-zarrar/ns1/reverse2/1fdda06d-8c0b-4781-b7e8-082aeec81f4d-reverse.py
--destination-file /pulsar/reverse.py
```
doesn’t have the auth params passed. It needs to be:
```
/pulsar/bin/pulsar-admin --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params token:<token> --admin-url <http://pulsar-broker:8080> functions download
--path tenant-zarrar/ns1/reverse2/1fdda06d-8c0b-4781-b7e8-082aeec81f4d-reverse.py
--destination-file /pulsar/reverse.py
```
If you add that manually by editing the statefulset it will probably work. As I said before authentication/authorization will be more polished in 2.4
----
2019-03-31 02:11:44 UTC - xutao: @xutao has joined the channel
----
2019-03-31 04:24:05 UTC - Joe Francis: Non-persistent messages will give you any current messages (not most recent)
----