You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/08/28 09:11:03 UTC
Slack digest for #general - 2019-08-28
2019-08-27 14:38:18 UTC - Retardust: @Retardust has joined the channel
----
2019-08-27 14:55:42 UTC - Retardust: hi, assume I have chain of client -> non-persistent-topic -> function -> persistent topic. When client send message it will receive ack only after bookkeeper will persist it , am I right? it's not very clear from documentation)
----
2019-08-27 14:58:53 UTC - Retardust: Also, is there any simple way to run pulsar-functions integration test in java? I have testcontainers with pulsar, but I need to build jar file and deploy functions from test... is there any test utils maybe?
----
2019-08-27 15:57:30 UTC - Matteo Merli: @Kim Christian Gaarder Not in 2.4.0, but it will be included in 2.4.1: <https://github.com/apache/pulsar/pull/4562>
----
2019-08-27 15:58:39 UTC - Matteo Merli: For persistent topics, the ack is sent to producer on only after the message is persisted in the configure number of bookies.
For non persistent topics, the ack is sent as soon as the messages reaches the Pulsar broker
----
2019-08-27 16:44:18 UTC - Matteo Merli: <!channel> We have started to build a list of companies using or contributing to Apache Pulsar. Please DM/email if you want to get added as well! <https://pulsar.apache.org/en/powered-by/>
100 : Nicolas Ha, Grant Wu, Sijie Guo, Bruno Panuto, Devin G. Bost, Shivji Kumar Jha, Karthik Ramasamy, Vineeth Thumma, Kanaderu Fukuda, Jean-Bernard van Zuylen
muscle : Mark Addison, Grant Wu, Sijie Guo, Bruno Panuto, Jerry Peng, Karthik Ramasamy, dba, Vineeth Thumma, Federico Ponzi, Jean-Bernard van Zuylen
----
2019-08-27 17:48:02 UTC - Retardust: ok, it's not quite clear here <https://pulsar.apache.org/docs/en/concepts-messaging/>
```
Non-persistent messaging is usually faster than persistent messaging because brokers don't persist messages and immediately send acks back to the producer as soon as that message is deliver to all connected subscribers.
```
----
2019-08-27 17:49:56 UTC - Matteo Merli: Agree, that can be interpreted in multiple ways
----
2019-08-27 17:57:58 UTC - Retardust: Is there any place where I could contribute to documentation? Or will the commitment team fix that? "subscribers" -> "brokers" will be more accurate
----
2019-08-27 17:58:35 UTC - Matteo Merli: The page itself has an “Edit” button :wink:
----
2019-08-27 17:58:52 UTC - Matteo Merli:
----
2019-08-27 17:59:51 UTC - Retardust: I see, thanks)
----
2019-08-27 18:00:44 UTC - Ravi: @Matteo Merli Giggso is deep on pulsar, as the world's first message aggregator. Currently on pocs with banks. Pls do ping me for details
----
2019-08-27 18:18:42 UTC - Retardust: <https://github.com/apache/pulsar/pull/5058>
----
2019-08-27 18:21:00 UTC - Matteo Merli: :+1:
----
2019-08-27 18:36:48 UTC - Jerry Peng: <!channel> If we can also include a brief statement from each of those companies on how they are using Pulsar. That would be awesome! Similar to the powered by page for Apache Storm:
<https://storm.apache.org/Powered-By.html>
----
2019-08-27 18:38:17 UTC - Tarek Shaar: yes that would be great indeed thanks Jerry
----
2019-08-27 18:49:05 UTC - Retardust: maybe it will be useful to have a list of pulsar users communities? we have russian chat in telegram. Small, (40 members) but it's will be grown)
<https://t.me/pro_pulsar> and it would be great an interested people to join and communicate
----
2019-08-27 18:56:18 UTC - Sijie Guo: @Retardust :
Do you want to submit a pull request adding your telegram group to this page: <http://pulsar.apache.org/en/events/>
----
2019-08-27 18:56:39 UTC - Sijie Guo: This page lists some of the meetup groups and pulsar events.
----
2019-08-27 19:05:19 UTC - Retardust: Hm, it's seems not to be the right place. I'd check Community pages and not even visited the events page:) Maybe you could rename that to "Events and User Groups" or smthg?
----
2019-08-27 19:09:48 UTC - Sijie Guo: Agreed. `Events` is probably not the right name for that page. We can rename it to `Events and User Groups` or just create a new page `User Groups` for listing user groups.
----
2019-08-27 19:10:00 UTC - Sijie Guo: Are you interested in contributing a change for it?
----
2019-08-27 19:29:49 UTC - Retardust: yes, I could do that. So separate page or rename?
----
2019-08-27 19:32:01 UTC - Sijie Guo: I guess a separated page might be better :slightly_smiling_face:
----
2019-08-27 19:34:13 UTC - Retardust: ok)
ok_hand : Sijie Guo
+1 : Sijie Guo
----
2019-08-27 20:27:10 UTC - Rajiv Abraham: Hi, if I want to do CDC where the sink is a REST api, my options are Pulsar Functions and Consumers? Is there some HTTP/REST Sink Connector in Pulsar?
----
2019-08-27 20:30:22 UTC - Matteo Merli: There’s no general use HTTP sink so far
----
2019-08-27 20:31:11 UTC - Rajiv Abraham: Thanks @Matteo Merli for the quick reply
----
2019-08-27 21:04:37 UTC - Retardust: Also I wonder about netty tcp connector. There is lack of documentation. How does it works, what payload format? How to separate messages? How to receive acks?)
----
2019-08-27 21:23:40 UTC - jialin liu: Hi I followed the python example on : <https://pulsar.apache.org/docs/en/client-libraries-python/>, running Pulsar in standalone mode. But the code stuck in ‘consumer.receive()’
----
2019-08-27 21:27:59 UTC - Sijie Guo: The documentation about netty tcp connector is coming up soon. @tuteng and @Anonymitaet are working on enhancing the documents for this part. They can help you in this part.
----
2019-08-27 21:28:15 UTC - Sijie Guo: do you have any messages published?
----
2019-08-27 21:28:42 UTC - Sijie Guo: did you run producer first or consumer first?
----
2019-08-27 21:33:11 UTC - Kirill Merkushev: Hello, trying to adopt Pulsar for event sourcing for medical software in Europe - is there any way to address some GDPR requirements like compact just one key across the topic or offload after some time to tiered storage and cleanup that safely offline?
----
2019-08-27 21:37:01 UTC - Sijie Guo: @Kirill Merkushev currently compaction and offloader doesn’t support compacting a single key yet. but it should be pretty straightforward to adding some filtering logic in compaction and offloader to provide such ability. It should be a fun feature to add.
----
2019-08-27 21:37:56 UTC - Kirill Merkushev: sounds great, how can I support that? Should I fill a feature request somewhere?
----
2019-08-27 21:38:11 UTC - Kirill Merkushev: or submit a naive implementation?
----
2019-08-27 21:42:24 UTC - Sijie Guo: feel free to create a feature request in <https://github.com/apache/pulsar>
--
if you are interested in contributing, feel free to send out a pull request.
If the change is going to be a big feature, you can write a PIP for your design and share the PIP to the community in <ma...@pulsar.apache.org> mailing list. (example PIPs can be found: <https://github.com/apache/pulsar/wiki>)
----
2019-08-27 21:49:54 UTC - Alexandre DUVAL: @Sijie Guo about GDPR on tiered storage, there is currently no way to expire data in tiered storage or delete them automatically after a certain delay? (about this issue: <https://github.com/apache/pulsar/issues/4515>)
----
2019-08-27 21:50:36 UTC - Sijie Guo: @Alexandre DUVAL yes. I was aware of your issue
----
2019-08-27 21:52:32 UTC - Alexandre DUVAL: Oh ok. :slightly_smiling_face:
----
2019-08-27 21:54:15 UTC - jialin liu: yes, Hi Sijie, I think it’s related to a bug with wifi, <https://github.com/apache/pulsar/issues/4593>
----
2019-08-27 21:54:27 UTC - jialin liu: @Sijie Guo I disabled wifi, and it works
----
2019-08-27 21:54:50 UTC - Sijie Guo: Ah I see :+1:
----
2019-08-27 22:05:37 UTC - Kirill Merkushev: <https://github.com/apache/pulsar/issues/5059>
+1 : Sijie Guo, Ali Ahmed, Karthik Ramasamy
100 : Sijie Guo, Ali Ahmed
----
2019-08-28 01:01:03 UTC - Anonymitaet: @Retardust thanks for your suggestion, we will document those
----
2019-08-28 01:07:40 UTC - xue: When I was doing the pulsar SQL test, I sent 100 records to Topic, and only 99 records could be queried in Presto. The latest one is not found. What's the reason?
----
2019-08-28 01:14:46 UTC - Poule: @xue <https://github.com/apache/pulsar/issues/4976>
----
2019-08-28 01:17:46 UTC - xue: @Poule thank you!
----
2019-08-28 01:28:41 UTC - borlandor: How can I publish messages with schema on Pulsar C++ client? I have not find the usage of "producer.send" with schema.
See alse: <https://github.com/apache/pulsar/issues/5057>
----
2019-08-28 01:35:01 UTC - borlandor: How can I publish messages with schema on Pulsar C++ client? I have not find the usage of "producer.send" with schema.
See alse: <https://github.com/apache/pulsar/issues/5057>
----
2019-08-28 01:35:37 UTC - Matteo Merli: Pulsar C++ producer doesn’t provide support for serialization, though it will let you declare the schema on the producer itself
----
2019-08-28 01:36:27 UTC - Matteo Merli: The C++ client lib will not enforce that the data is actually conform with the schema. Schema support in C++ was added to allow schema support in Python and Go client wrapper
----
2019-08-28 01:41:34 UTC - xue: Is Message deduplication based on message value or key?
----
2019-08-28 01:41:49 UTC - Matteo Merli: sequence id
----
2019-08-28 01:43:36 UTC - borlandor: Thank you! @Matteo Merli
I have see some C++ schema code in <http://SchemaTest.cc|SchemaTest.cc> :
Producer producer;
ProducerConfiguration producerConf;
producerConf.setSchema(SchemaInfo(AVRO, "Avro", exampleSchema));
res = client.createProducer("topic-avro", producerConf, producer);
producer.close();
ASSERT_EQ(ResultOk, res);
But there are no examples of how to send a message with Schema.
----
2019-08-28 01:44:13 UTC - xue: @Matteo Merli thank you
----
2019-08-28 01:44:51 UTC - Matteo Merli: You would just using `producer.send()` with the bytes for the serialized avro message
----
2019-08-28 01:49:43 UTC - borlandor: That means I need to write the serialization code myself?
----
2019-08-28 03:01:13 UTC - Matteo Merli: Yes, there’s no support for serialization in the c++ lib at this point
----
2019-08-28 03:02:34 UTC - Matteo Merli: Again, the primary reason to allow schema declaration in c++ was to allow the python and go wrappers to provide schema integration, with the language specific serializations
----
2019-08-28 03:16:08 UTC - borlandor: ok, thanks!
----
2019-08-28 08:20:37 UTC - xue: There are 3 nodes in a cluster (3 brokers, 3 bookies). If I want to add a bookie to the fourth node, do I need to stop producers and consumers and re-initialize metadata by commanding bin/pulsar initialize-cluster-metadata? Do I have a more detailed description of expanding bookie?
----
2019-08-28 08:37:26 UTC - Richard Sherman: You don't need to stop producers or consumers or reinitialise the meta data.
As long as the bookie is configured the same as the others it will just talk to zookeeper and add itself to the cluster. Once added it will start to receive writes.
----
2019-08-28 08:40:25 UTC - xue: thank you, I'll try.
----