Slack digest for #general - 2019-10-19

[Apache Pulsar Slack <ap...@gmail.com>]

2019-10-18 09:43:33 UTC - Martin Kunev: Hello,
My team is having a problem with pulsar 2.4.0. We have a topic that has replication clusters properly set, but published messages only arrive to some of the clusters. The other clusters never receive the messages. With pulsar 2.3.1 there i
s no such issue. What can be the cause of this problem?
----
2019-10-18 09:57:58 UTC - AlifRuliarso: @AlifRuliarso has joined the channel
----
2019-10-18 13:24:51 UTC - Junli Antolovich: @Chris Bartholomew I followed your command and it worked for me in PowerShell!   Thanks so much for the help, appreciated.
----
2019-10-18 15:33:31 UTC - Chris Bartholomew: @Junli Antolovich Glad to hear it is working for you now. Good luck with your POC.
+1 : Junli Antolovich
----
2019-10-18 17:56:55 UTC - Retardust: does pulsar_subscription_back_log is proper metric to monitor consumer lag?
it's increase without incoming messages, weird))
----
2019-10-18 18:07:38 UTC - Oleg Kozlov: that 's a bit different, i do want to store messages forever, but get them removed as soon (or shortly after) they are ack'd, like in a typical queue. I get that behavior by creating a shared subscription ahead of time, which is fine, like i said, it's not a blocker, just a slight complication
----
2019-10-18 18:08:13 UTC - Oleg Kozlov: btw, do subscriptions w/o connected consumers ever expire/dispappear ?
----
2019-10-18 18:09:25 UTC - Matteo Merli: By default, no. There’s a broker setting to drop subscriptions for which a consumer didn’t connect in some time. eg. 2 weeks
----
2019-10-18 18:43:14 UTC - Chris DiGiovanni: About to file a bug, though thought I would see if anyone is familiar with this issue:

When I hit the api endpoint /admin/v2/non-persistent//{tenant}/{namespace} to list non-persistent topics, the request just hangs.  If I hit the broker to 2-3 times with this request, the broker API becomes unresponsive and needs to be restarted.  This happens in our lab deployment running 2.4.1 and in our production environment running 2.3.0.  When we run the standalone instance we cannot replicate.
----
2019-10-18 18:44:04 UTC - Chris DiGiovanni: We found this while trying to use Presto on Pulsar.  Even though we don't have any non-persistent topics, Presto still wants to list non-persistent topics.
----
2019-10-18 18:45:28 UTC - Chris DiGiovanni: I should also mention I never see the GET in the logs of the broker for the request...
----
2019-10-18 18:50:15 UTC - Chris DiGiovanni: Seems related to this: <https://github.com/apache/pulsar/issues/3877>
----
2019-10-18 18:57:27 UTC - Sijie Guo: Not aware of any similar issue.

I would suggest taking following steps to investigate.

1) use `pulsar-admin topic stats` to see if there is any backlog for the replication subscription.
2) check if the broker log to see if there is any errors.
----
2019-10-18 18:59:23 UTC - Sijie Guo: yes. can you use `pulsar-admin topic stats` to get the detailed stats about produce and consume rate to see if there are messages being produced
----
2019-10-18 19:20:29 UTC - Alex: Hello, I have a question regarding schema management in Pulsar.
Does it checks for added/removed fields in JSON/Avro schemas?
How does it works for custom SchemaType?
----
2019-10-18 19:25:26 UTC - Alex: Also I can't figure out what is the purpose of SchemaInfo.schema. How it's used by Pulsar?
----
2019-10-18 19:52:29 UTC - Raman Gupta: Does anyone else have an issue with the producer `sequenceId` being only a `long` value? You can't even stuff a `uuid` into that, and it limits the utility of this to sequential producers rather than to any unique value.
----
2019-10-18 19:55:55 UTC - Matteo Merli: well, a sequence hints that the ids are going to be sequentials :slightly_smiling_face: . A UUID won’t have that property
----
2019-10-18 20:00:37 UTC - Chris DiGiovanni: Created the following bug: <https://github.com/apache/pulsar/issues/5417>
----
2019-10-18 20:05:26 UTC - Sijie Guo: it does compatibility check for Pulsar supported schemas.
----
2019-10-18 20:06:18 UTC - Sijie Guo: if you are introducing your own schema type, you can implement your own schema compatibility checker.
----
2019-10-18 20:06:56 UTC - Sijie Guo: `schema` field is used for storing schema definition. currrent all the struct schemas in Pulsar (avro, json, protobuf) is using avro specification for defining schema.
----
2019-10-18 20:13:34 UTC - Alex: @Sijie Guo so my current understanding is the following (please correct me if I'm wrong):
1. If `Schema.type` is (avro/json/protobuf), `schema` field MUST contain avro-based schema definition;
2. If `Schema.type` is NONE, I can specify any data in `schema` field and then use it in my compatibility checker.
----
2019-10-18 20:21:06 UTC - Daniel Moran: @Daniel Moran has joined the channel
----
2019-10-18 20:24:11 UTC - Daniel Moran: Hi all, my team is evaluating Pulsar as a replacement for Kafka. One thing that worries us is the lack of support for transactional processing;
----
2019-10-18 20:24:24 UTC - Daniel Moran: but I see there’s an issue for adding that functionality
----
2019-10-18 20:24:34 UTC - Daniel Moran: is there a planned date for when the txn logic will be available?
----
2019-10-18 20:46:05 UTC - David Kjerrumgaard: @Daniel Moran That is tentatively slated for the 2.5 release.
----
2019-10-18 20:47:02 UTC - Daniel Moran: is there a rough date for when that release is expected?
----
2019-10-18 20:52:55 UTC - David Kjerrumgaard: <https://github.com/apache/pulsar/milestones>
----
2019-10-18 20:53:13 UTC - David Kjerrumgaard: You can track the progress here. Currently, it is 19 days overdue :smiley:
tada : Daniel Moran
----
2019-10-18 20:59:34 UTC - Daniel Moran: thank you!
----
2019-10-18 22:48:39 UTC - Luke Lu: Spot a new error in the broker log today: `2019-10-18 21:04:55,693 bookkeeper-ml-workers-OrderedExecutor-3-0 ERROR Recursive call to appender RoutingAppender` Any explanations?
----
2019-10-18 23:05:41 UTC - Luke Lu: Hey guys, is there anyway to move an existing topic to a different namespace, while retaining all its retained data (from the retention policy)?
----
2019-10-18 23:06:54 UTC - Matteo Merli: Not directly, though you could re-publish all the data into the new topic
----
2019-10-18 23:13:21 UTC - Luke Lu: Understood. Was wondering about any secret way to change some metadata without moving data.
----
2019-10-18 23:48:26 UTC - Oleg Kozlov: thank you
----
2019-10-19 00:33:54 UTC - Alex: When subscriptions/cursors got deleted from Pulsar? Is it ok to create lots of them?
Basically I want to have a topic with disabled expiration and re-read it every time my application starts (and rebuilds it state from the whole topic content).
----
2019-10-19 01:10:57 UTC - Alex: I just figured out that I can read topic without sending acks...
----
2019-10-19 01:18:35 UTC - Sijie Guo: 1 is correct
----
2019-10-19 01:20:02 UTC - Sijie Guo: 2 is not. if Schema.type is NONE, pulsar doesn’t interpret data. I was talking if you want to introduce a new custom type, you can add a new type, add your own schema data in `schema` field and implement a compatibility checker to validate the schema.
----
2019-10-19 01:25:11 UTC - Sijie Guo: seems like a bug from log4j2
----
2019-10-19 01:26:39 UTC - Alex: @Sijie Guo thanks for explanation! It looks not that simple - I will investigate it further.
----
2019-10-19 01:34:07 UTC - Sijie Guo: yeah ping me if you need any help
----
2019-10-19 08:21:27 UTC - Retardust: It would be nice feature for smooth production refactoring:) maybe some topic aliasing?
----