You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/10/17 09:11:03 UTC
Slack digest for #general - 2019-10-17

2019-10-16 09:46:58 UTC - Alexandre DUVAL: @Matteo Merli
----
2019-10-16 10:05:27 UTC - vikash: Yes it  works  thanks  for  your  help
+1 : Vladimir Shchur
----
2019-10-16 10:25:37 UTC - Olivier Chicha: Thank you for the answer.
Yes effectively it does not sound a good idea
----
2019-10-16 10:29:12 UTC - vikash: now  getting  this  Error
----
2019-10-16 10:29:14 UTC - vikash: Could not load file or assembly 'System.IO.Pipelines, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
----
2019-10-16 10:29:38 UTC - vikash: but  i  have  added  binding  redirect
----
2019-10-16 10:29:39 UTC - vikash: &lt;dependentAssembly&gt;
        &lt;assemblyIdentity name="System.IO.Pipelines" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" /&gt;
        &lt;bindingRedirect oldVersion="0.0.0.0-4.0.1.1" newVersion="4.0.1.0" /&gt;
      &lt;/dependentAssembly&gt;
----
2019-10-16 10:33:35 UTC - Jack: thanks you @Matteo Merli,  download new version then move data and config from 2.3.0 to 2.4.0 is all step that i do ?
----
2019-10-16 10:40:46 UTC - Vladimir Shchur: can you try 4.0.0.1 ? or I'll check in half an hour
      &lt;dependentAssembly&gt;
        &lt;assemblyIdentity name="System.IO.Pipelines" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" /&gt;
        &lt;bindingRedirect oldVersion="0.0.0.0-4.0.1.0" newVersion="4.0.0.1" /&gt;
      &lt;/dependentAssembly&gt;
----
2019-10-16 11:28:34 UTC - vikash: yes  working
----
2019-10-16 12:05:50 UTC - Naby: Thanks
----
2019-10-16 12:21:37 UTC - Vladimir Shchur: Updated nuget, also one more feature included
----
2019-10-16 12:38:54 UTC - Retardust: Hi, is there option to add tags to pulsar function metrics?
----
2019-10-16 12:40:28 UTC - jamie: ooh, thanks!
----
2019-10-16 12:49:30 UTC - Retardust: also, what does
storageSize in pulsar topic mean?
I've write   messages  in persistent topic, but "storageSize": 0,
----
2019-10-16 13:00:48 UTC - Junli Antolovich: @Vladimir Shchur Since you are working on .net client for Pulsar, just wondering if you and your team has setup Pulsar in docker on windows 10?  If we are unable to install it on windows, it won't be a viable candidate for us. Any help is appreciated.
----
2019-10-16 13:01:21 UTC - Vladimir Shchur: Hi! I'm using minikube with helm chart
----
2019-10-16 13:03:39 UTC - Junli Antolovich: Well, I am not following.  what do you mean: "I'm using minikube with helm chart"?
----
2019-10-16 13:05:47 UTC - Vladimir Shchur: <https://kubernetes.io/docs/setup/learning-environment/minikube/>
----
2019-10-16 13:06:22 UTC - Vladimir Shchur: So this is minikube that I'm running on windows. And to run Pulsar there I'm using Pulsar helm chart for kubernetes
----
2019-10-16 13:07:34 UTC - Vladimir Shchur: If you want to run cluster on windows without docker and kubernetes, then I won't help you
----
2019-10-16 13:10:17 UTC - Vladimir Shchur: Another guy from my team doesn't use minikube, only docker, so it should work fine as well.
----
2019-10-16 13:10:43 UTC - Junli Antolovich: I was trying to run Pulsar on Docker and kubernetes, but I ran into "Error open RocksDB database" issue.
----
2019-10-16 13:11:16 UTC - Chris Bartholomew: @Junli Antolovich I am going to look into the issue you are seeing running using Docker on Windows. It should work. I just need to spark up a Windows machine, since I usually work in Linux.
+1 : Junli Antolovich
----
2019-10-16 13:12:54 UTC - Vladimir Shchur: @Junli Antolovich which command are you running? try mounting disc C, it might help
----
2019-10-16 13:14:32 UTC - Junli Antolovich: @Vladimir Shchur I was using this command:
----
2019-10-16 13:14:44 UTC - Junli Antolovich: `docker run -it -p 6650:6650 -p 8080:8080 -v "c:\docker\pulsar:/pulsar/data" apachepulsar/pulsar:2.4.1 bin/pulsar standalone`
----
2019-10-16 13:15:33 UTC - Chris Bartholomew: You might want to try using forward slashes in the Windows path: ```c:/docker/pulsar:/pulsar/data```
----
2019-10-16 13:18:37 UTC - Alexandre DUVAL: --custom-schema-inputs
----
2019-10-16 13:18:39 UTC - Alexandre DUVAL: ```Input Topics cannot be altered

Reason: Input Topics cannot be altered
```
----
2019-10-16 13:18:42 UTC - Alexandre DUVAL: @David Kjerrumgaard
----
2019-10-16 13:19:46 UTC - Retardust: does pulsar not store data if there is no consumers on topic? how I could change that behaviour ?
----
2019-10-16 13:20:28 UTC - Junli Antolovich: The forward and backward slash both works in windows. Trying with forward slash, and got the same error:
----
2019-10-16 13:20:41 UTC - Junli Antolovich: `13:18:19.576 [main] ERROR org.apache.pulsar.PulsarStandaloneStarter - Failed to start pulsar service.`
`java.io.IOException: Error open RocksDB database`
----
2019-10-16 13:22:41 UTC - Vladimir Shchur: I will provide the command that works for my mate later today
----
2019-10-16 13:25:42 UTC - Junli Antolovich: Thanks much for the help, @Vladimir Shchur and @Chris Bartholomew I hope to get this working that I can do a POC on it soon
----
2019-10-16 13:42:02 UTC - Vladimir Shchur: You need to set retention to change that
----
2019-10-16 14:39:05 UTC - Shishir Pandey: @Matteo Merli I want to produce data from same producer to multiple queues - does this mean I can't do that? Does it hold true for consumers too?
I was planning to test out the following set up, for my use case
Topic 1 -&gt; contains processing request
Topic 2 -&gt; contains data fetch request.

Producer -&gt; {produce message topic 2, produce delayed (up to 15 seconds) message topic 1}

Consumer 2 -&gt; {consume from Topic 2, produce message on Topic 3}

Consumer 1 -&gt;{ consume from Topic 1 (key shared subscription) and consume from Topic 3 }

Is this not feasible? In this case we have producer producing on two topics and consumer consuming from two topics.
----
2019-10-16 14:40:02 UTC - Shishir Pandey: I forgot to add - I will be using java for the producer and C++ for the consumer
----
2019-10-16 15:12:15 UTC - Sam Leung: Is there a configuration to control auto creation of topics per namespace? I can set `allowAutoTopicCreation` to false on the broker config, but I need something that can allow it for certain namespaces.
----
2019-10-16 15:17:58 UTC - Sijie Guo: Currently it is a broker setting. We can easily add it to the namespace policies so users can control it at namespace level.

Can you help creating a github issue? We can add it.
----
2019-10-16 15:19:21 UTC - Sijie Guo: @Retardust: pulsar is a subscription-based data system. the data is kept either 1) there are subscriptions or 2) there is a retention setting.
----
2019-10-16 15:20:30 UTC - Sijie Guo: storageSize means the total storage of the retained messages.
----
2019-10-16 15:21:13 UTC - Sijie Guo: Can you give an example about the tags here?
----
2019-10-16 15:34:34 UTC - Matteo Merli: Maybe I’m missing something, but why can’t you create 2 producers , (P1, P2) and use either of them depending on which topic you want to publish a message?
----
2019-10-16 15:37:07 UTC - Retardust: thanks
----
2019-10-16 15:37:55 UTC - Retardust: I mean prometheus tags
----
2019-10-16 15:38:55 UTC - Retardust: messages_parsed_count{status="failed"}
messages_parsed_count{status="success"}
----
2019-10-16 15:39:34 UTC - Retardust: -
----
2019-10-16 15:43:30 UTC - Shishir Pandey: Yes, I could do that but I just wanted to know if it was possible, I have currently coded it up as two separate producers, and two consumers but since I always have to do both the tasks I thought to self if it'd be possible to do so via the same producer by simply changing the topic to which you were producing it'd be simpler.
----
2019-10-16 15:44:55 UTC - Retardust: -
----
2019-10-16 16:02:40 UTC - Retardust: what's the difference between consumer and subscription metrics? there is no difference if I have only 1 consumer in group, am I right?
----
2019-10-16 16:13:18 UTC - Matteo Merli: correct
----
2019-10-16 16:13:34 UTC - Matteo Merli: subscription level is just aggregated across multiple consumers
----
2019-10-16 17:10:28 UTC - Sam Leung: Sure, thanks
----
2019-10-16 17:17:11 UTC - Sam Leung: <https://github.com/apache/pulsar/issues/5395>
----
2019-10-16 17:17:55 UTC - David Kjerrumgaard: Interesting.
----
2019-10-16 18:06:27 UTC - Alexandre DUVAL: :p
----
2019-10-16 18:26:43 UTC - Sam Leung: Is the javadoc at <https://pulsar.apache.org/api/client/> the wrong one to look at? It seems very out of date.
----
2019-10-16 19:12:43 UTC - Sijie Guo: I think the javadoc built script is somehow broken. so that doc was out of date. @tuteng was looking into fixing it.
----
2019-10-16 19:21:20 UTC - Thor Sigurjonsson: Is there a released redis connector nar file available somewhere?
----
2019-10-16 19:23:02 UTC - Ali Ahmed: @Thor Sigurjonsson redis is available <https://pulsar.apache.org/docs/en/io-redis/#sink>
----
2019-10-16 19:30:05 UTC - Thor Sigurjonsson: Right, I am aware of the sink code being there - and docs.. Just not able to find a released/built nar file for it.
----
2019-10-16 19:32:19 UTC - Thor Sigurjonsson: Its not here <https://pulsar.apache.org/en/download/> or in the docker pulsar-all images.
----
2019-10-16 19:33:20 UTC - Sijie Guo: I think it is added for 2.5.0
----
2019-10-16 19:33:57 UTC - Sijie Guo: the documentation changes of 2.5.0 was released as part of 2.4.1 (which is the latest release version)
----
2019-10-16 19:34:11 UTC - Sijie Guo: the documentation framework doesn’t handle well with the minor releases.
----
2019-10-16 19:36:08 UTC - Thor Sigurjonsson: I'm guessing then that 2.5.0 builds aren't available yet. Looks like the redis sink code hasn't been modified for a few months.
----
2019-10-16 19:37:25 UTC - Sam Leung: thanks
----
2019-10-16 19:37:27 UTC - Sijie Guo: actually I took it back.
----
2019-10-16 19:37:36 UTC - Sijie Guo: I think it was contributed in 2.4.0-
----
2019-10-16 19:37:46 UTC - Sijie Guo: somehow I thought it was in 2.5.0
----
2019-10-16 19:38:14 UTC - Thor Sigurjonsson: Looks like builds/releases aren't happening for all the connectors then...
<http://apache.claz.org/pulsar/pulsar-2.4.1/connectors/>
----
2019-10-16 19:39:26 UTC - Sijie Guo: yeah it is probably not added to the release list.
----
2019-10-16 19:39:33 UTC - Sijie Guo: Can you file a github issue for that?
----
2019-10-16 19:39:42 UTC - Thor Sigurjonsson: Sure :slightly_smiling_face:
----
2019-10-16 19:55:07 UTC - Thor Sigurjonsson: <https://github.com/apache/pulsar/issues/5397>
----
2019-10-16 23:54:33 UTC - Junli Antolovich: @Sijie Guo I am  seeing your comments on <https://github.com/apache/pulsar/issues/4684>
----
2019-10-17 02:09:31 UTC - Junli Antolovich: _Currently I don't think Pulsar support running on Windows very well. Because there are a few dependencies are using native code. The bindings we are shipping probably doesn't contain the native libraries built for windows._
----
2019-10-17 02:11:37 UTC - Junli Antolovich: Thanks all, so long!
----
2019-10-17 03:05:57 UTC - Gary Myers: @Gary Myers has joined the channel
----
2019-10-17 06:44:34 UTC - Vladimir Shchur: Hi @Junli Antolovich! I guess that response was not about Docker, but about running Pulsar on bare Windows. Regarding Docker, this is the command that works for my teammate:
`docker run --interactive --tty -p 31002:6650 -p 8080:8080 -v "c:/projects/runpulsar/data:/pulsar/data" apachepulsar/pulsar:2.4.1 bin/pulsar standalone`
You might want to remove existing container to start from scratch
----
2019-10-17 07:49:32 UTC - Vladimir Shchur: Hi! How powerful are Pulsar Functions? The docs say "That processing logic could be just about anything you want .. writing results to an external database.", but it requires external libraries to be loaded. Is there an ability to load external libraries to be used from inside Java or Python pulsar functions?
----