You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/08/12 09:11:05 UTC
Slack digest for #general - 2020-08-12
2020-08-11 09:57:39 UTC - Allen ONeill: *FUNCTIONS - STATE*
Hi all - I am trying to use STATE with python ... set and get.
My simple example (below) is to set a counter, set state, then read them back.
I am doing something wrong as it seems to be neither setting or getting the data ... here is my code:
```from pulsar import Function
class State(Function):
def process(self, input, context):
properties = context.get_message_properties()
run_id = properties.get("run_id", "none")
if run_id != "none":
context.incr_counter(run_id, 1)
context.put_state("run_id", run_id)
print(f'Counter result: {run_id} = {context.get_counter(run_id)}')
print(f'State result: {run_id} = {context.get_state(run_id)}')```
Any ideas?
Thanks, Allen.
----
2020-08-11 10:45:26 UTC - vikash: i am also not able to understand how to enable jmx port in standalone pulsar
----
2020-08-11 11:57:56 UTC - Raghav: Hi all,
Does anyone know how to mark a ledger directory from read-only to read-write? chmod is not working.
----
2020-08-11 12:05:31 UTC - Alexandre DUVAL:
----
2020-08-11 12:51:50 UTC - Seun: Hi everyone,
Is anyone here aware of any light-weight version of helm charts deployment of Apache pulsar? Something that doesn't consume many CPUs and memories of compute machines
----
2020-08-11 12:52:10 UTC - Caleb Epstein: ns = dns? I'd think this would be in your operating system configs (i.e. /etc/resolv.conf, /etc/nsswitch.conf on Linux)
----
2020-08-11 12:53:14 UTC - Caleb Epstein: What are helm charts? Does this help?
<https://github.com/streamnative/apache-pulsar-grafana-dashboard>
----
2020-08-11 12:54:55 UTC - Seun: Many thanks @Caleb Epstein Helm charts are k8s deployment and management tool. Simplify control of k8s clusters. I will check that out.
----
2020-08-11 12:58:32 UTC - Boluwatife Fasugba: @Boluwatife Fasugba has joined the channel
----
2020-08-11 13:11:34 UTC - Emanuel: Hi all, is there any pessimistic lock on a event enters in bookkeeper ?
----
2020-08-11 13:13:57 UTC - Thomas O'Neill: You can use the helm chart and change the replica counts if you want fewer pods. You can also change the resource requests too, but you will want to leave enough that the application can still work. All of this can be set in a values file that you use when installing and updating the helm chart. You can also disable the proxy, toolset and pulsar_manager, and the monitoring (grafana and prometheus) if you are not using them.
----
2020-08-11 13:17:28 UTC - Seun: I did all of these except deleting pulsar_manager. I setup GKE on gcp and i keep running into CPUs and memory insufficiency and Google has refused to raise compute CPus quotas. Shame! I will remove pulsar_manager and see what happens.
Many thanks @Thomas O'Neill
----
2020-08-11 13:38:12 UTC - Meyappan Ramasamy: hi team, i was unable to subscribe to future topics when using pulsar version 2.5.0. please let me know if future subscription will work in pulsar version 2.6.0
----
2020-08-11 13:39:37 UTC - Meyappan Ramasamy: hi team, sorry i missed your response. this error was showing up when the gradle build and unit test was happening in jenkins server, maybe some compatibility issue exists with other platforms, please let me know how to resolve.
----
2020-08-11 17:03:44 UTC - Rich Adams: Yeah, I was wondering about that too. But not sure if the shaded jar would fix it. I think that as long as you get the classloader from the main process the dependency injection will grab the classes that presto has loaded. But I’m not an expert in this stuff, so I may be wrong.
----
2020-08-11 18:04:24 UTC - Addison Higham: Hrm... as mentioned, that demo code is probably pretty out of date. If you could provide some logs of what log4j says it is doing (which you can get by setting ` log4j2.debug` as a system prop) then that would probably be helpful
@Axel Sirota however, your best bet is probably to just try and use this instead: <https://github.com/apache/pulsar/tree/master/pulsar-log4j2-appender> It as a log4j2 append that goes direct to Pulsar and doesn't use the kafka appender
----
2020-08-11 18:04:47 UTC - Addison Higham: here is the deps: <https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-log4j2-appender>
----
2020-08-11 18:11:46 UTC - Addison Higham: @Alexandre DUVAL AFAICT, I don't see any mechanism to change the default NS name or the creation of that namespace. The values are defined here: <https://github.com/apache/pulsar/blob/8aee353d5b000c80be5cc281e6b574bd62ba5531/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java#L42>
And here is where the namespace is created:
<https://github.com/apache/pulsar/blob/5421a4c36671823e70750995bfb1cef763ea48c1/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataSetup.java#L233>
----
2020-08-11 18:12:04 UTC - Addison Higham: that said, you should be able to delete it and change it's policies pretty easy
----
2020-08-11 18:22:22 UTC - Yarden Arane: @Yarden Arane has joined the channel
----
2020-08-11 18:43:11 UTC - Yarden Arane: *FUNCTIONS: heartbeat*
Hi all, is there any built-in heartbeat functionality for pulsar functions? Essentially I want a thread to trigger a function every second/minute/whatever.
A few possibilities would be:
1) a cron job to trigger / produce message on specific topic
2) a custom io-connector with a never ending thread triggering a function
Has anyone implemented a heartbeat before? Is there a better, more idiomatic way of achieving this functionality.
Thanks in advance.
----
2020-08-11 18:46:16 UTC - Addison Higham: @Seun you can use very small resource requests, like 256MB of heap can still allow pulsar to function, but you will have to worry about heap exceptions as you push the cluster
----
2020-08-11 18:48:09 UTC - Aaron: What type of avro encoding does Pulsar use? Binary, Single Object?
----
2020-08-11 18:52:27 UTC - Allen ONeill: I have the same use-case (almost) and havn't found a timer/cron function for Pulsar functions. My solution therefore is to make an external cron service that will send a message into a specific topic (call it the pulse/heartbeat topic, whatever), and then have a function watch for those specific messages/that topic and take that message as a trigger to do whatever it is I need done at the heartbeat time.
hth.
Allen.
----
2020-08-11 18:57:03 UTC - Allen ONeill: Cant see anything in documentation but does this indicate single object casting?
```Define Avro schemas with Schema.AVRO. The following code snippet demonstrates how to create and use Avro schema.
Producer<MyAvro> avroProducer = client.newProducer(Schema.AVRO(MyAvro.class))
.topic("some-avro-topic")
.create();```
from here:
<https://pulsar.apache.org/docs/en/client-libraries-java/#schemas>
----
2020-08-11 18:58:29 UTC - Yarden Arane: Hi Allen, thanks so much for your reply. Will likely go down the external cron service path as well.
----
2020-08-11 19:01:28 UTC - Aaron: Interesting. When forming a GenericAvroRecord it looks like Pulsar is using the binary encoding. (see line 99 <https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/generic/GenericAvroReader.java>)
----
2020-08-11 19:48:30 UTC - Aaron: To clarify, I am wondering what the encoding is when using AUTO_PRODUCE to a topic with an avro schema
----
2020-08-11 22:01:39 UTC - Jay Sun: Hi team, I am wondering how to refresh the tls cert and key for a long-lived consumer? especially for go client lib, but I am ok if you show me java code. Thanks
----
2020-08-11 22:13:22 UTC - Addison Higham: will answer in dev-go :slightly_smiling_face:
----
2020-08-11 22:18:01 UTC - Jay Sun: Thanks. In fact I am waiting there for one day already.. not sure it is being monitored..
----
2020-08-11 23:03:01 UTC - Guillaume: Hello eveyrone,
I'd like to know if there is any way to know which clients are connected to Pulsar?
Thanks
----
2020-08-11 23:17:00 UTC - Allen ONeill: if by client you mean message consumers, you can get this information by querying the admin API ... check getStats for example
<https://pulsar.apache.org/docs/en/admin-api-persistent-topics/#get-stats>
<https://pulsar.apache.org/admin-rest-api/?version=2.6.0&apiversion=v2#operation/getStats>
@Chris Bartholomew led me down this useful little path :)
----
2020-08-12 07:24:12 UTC - Hiroyuki Yamada: @Penghui Li I’m sorry for pushing, but can you give me the updates on this ?
<https://github.com/apache/pulsar/issues/7455#issuecomment-671699106>
For example, no clue at all at the moment ? or is someone investigating or woking on it ?
If no one is doing anything, I need to dig into it by myself or need to explore different approaches or different platforms.
----
2020-08-12 07:25:34 UTC - Penghui Li: Ok
----
2020-08-12 08:44:10 UTC - Dim: @Dim has joined the channel
----