You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/03/26 09:11:03 UTC
Slack digest for #general - 2020-03-26
2020-03-25 09:39:40 UTC - Pierre-Yves Lebecq: When I have a producer with an Avro schema creating a topic, I can see the topic and its schema using the CLI. When I add a function on the topic, I have the following error: `java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to com.zenaton.engine.workflowengine.messages.DispatchTaskMessage`
Whereas if I do the exact same thing but using JSONSchema.of() instead of AvroSchema.of() for the producer when it creates the topic, everything works fine when I using a function on this topic.
----
2020-03-25 09:42:37 UTC - Yosi Attias: @Gilles Barbier thanks!
I found a solution to my parallelism issue, Instead of have key (event) to list of subscriptions which can cause issues.
I will have key like `Event/SubscriptionId` to it’s subscription. this way I won’t have any concurrency issue, and I can use the failover subscription mode to handle the issue.
(subscriptions I am saying above, is my application subscriptions and not pulsar subscriptions).
The only thing missing right now, is adding range read so I can read all subscriptions given event.
----
2020-03-25 09:50:11 UTC - xue: Does pulsar function support the spring framework?
----
2020-03-25 09:51:59 UTC - xue: ```ApplicationContext applicationContext = new ClassPathXmlApplicationContext("file:G:\\dubbo-consumer.xml");```
function show error: cvc-elt.1: Cannot find the declaration of element 'beans'
----
2020-03-25 10:00:49 UTC - Dennis Yung: I found that the WebSocket client doesn't support right now, and such support doesn't seem to come soon. Is it possible to hack it through? eg inject schema version info to the message manually, and publish the schema through REST api?
----
2020-03-25 14:54:28 UTC - David Kjerrumgaard: @xue I think Spring would be a bit of overkill for a Pulsar Function IMHO. Longer term, the community needs to come up with a project similar to <https://github.com/spring-projects/spring-kafka|spring-kafka> . Pulsar functions are intended to be lightweight, simple functions that are only a few lines long.
----
2020-03-25 14:59:39 UTC - David Kjerrumgaard: @Pierre-Yves Lebecq If you are using the LocalRunner, then you can configure the input schema type as shown in this snippet.
----
2020-03-25 15:01:42 UTC - Pierre-Yves Lebecq: @David Kjerrumgaard I’m not unfortunately. Is there a way to do the same thing when using the CLI “functions create” command?
----
2020-03-25 15:04:44 UTC - David Kjerrumgaard: Yes, there is a `--custom-schema-inputs` switch you can use to specify this type of information. <https://pulsar.apache.org/docs/en/functions-cli/#create>
----
2020-03-25 15:06:25 UTC - Pierre-Yves Lebecq: I’ll have a look, thank you. :+1:
----
2020-03-25 16:26:21 UTC - Sijie Guo: @Pierre-Yves Lebecq I think there is an issue regarding avro. We are fixing that.
----
2020-03-25 16:30:16 UTC - Pierre-Yves Lebecq: @Sijie Guo All right. thanks for the heads up.
----
2020-03-25 17:19:42 UTC - Ashish Shinde: @Ashish Shinde has joined the channel
----
2020-03-25 17:25:08 UTC - Joel Cressy: Hey, I just found this comment thread on HN and i’m extremely intrigued: <https://news.ycombinator.com/item?id=21937710>
If anyone knows the OP or can get in contact with them, can we share code examples? My exact stack is AWS, EKS and hashicorp vault, so i’d love to know how they glued together auth (piggybacked on aws iam) and various other configs.
----
2020-03-25 17:42:35 UTC - Adam Feldman: Email is in their HN profile: <https://news.ycombinator.com/user?id=addisonj>
----
2020-03-25 17:48:56 UTC - Addison Higham: hi, that is me :slightly_smiling_face:
tada : Adam Feldman, Sijie Guo
----
2020-03-25 17:49:07 UTC - Addison Higham: @Joel Cressy ^^
----
2020-03-25 17:50:12 UTC - Joel Cressy: Awesome! glad to meet you
----
2020-03-25 17:51:16 UTC - Joel Cressy: i’m just beginning my research into this and am trying to gather as much info as I can
----
2020-03-25 17:51:29 UTC - Addison Higham: sure thing, let me put some details under a thread here
100 : Sijie Guo
+1 : Sijie Guo
----
2020-03-25 17:57:19 UTC - Addison Higham: So, the quick 5 minute version:
• we run on EKS, pretty much we started with the manifest definitions in the pulsar repo but have changed them quite a bit at this point, but nothing too fancy there, no real magic on an individual pulsar cluster
• We have many AWS accounts and VPCs as my company, so we expose pulsar via the pulsar proxy to other accounts using privatelink endpoints. We create the NLB for pulsar proxy just using a k8s service (see <https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html>). Then we have some terraform that simply looks up the service in k8s and then creates the privatelink endpooint.
• We use a combination of VPC peering, external DNS, and NLBs to run a global zookeeper on K8S AND to allow for geo replication. I can get into more detail on this later
----
2020-03-25 18:00:59 UTC - Addison Higham: now for auth: the basic concept is that we allow our users (via a small CLI tool and a microservice) to create an association between an IAM role and a pulsar role.
This association is basically stored in vault. Vault allows you to create policies that are "templated" based on the identity of who you are connect with. So for example, the following vault policy:
```pulsar/data/client/{{identity.entity.aliases.<id of your aws iam service auth>.name}}/creds/*```
----
2020-03-25 18:01:16 UTC - Addison Higham: we apply that policy to any roles we want to trust to any AWS accounts
----
2020-03-25 18:02:44 UTC - Addison Higham: basically, AWS roles that auth against vault can read from a location that include a unique identifier of their role. Our little microservice periodically drops creds to that same location (based on the association a user created)
----
2020-03-25 18:03:05 UTC - Joel Cressy: Are these creds a jwt that pulsar validates?
----
2020-03-25 18:04:33 UTC - Addison Higham: correct, we just use token based auth, but we use public/private key tokens so that the brokers/proxy only need the public key and private key can be limited to just signing tokens. Which we have a k8s cron job that basically just refreshes the tokens every few hours. Eventually, we want to write a vault plugin so that tokens are generated on demand, but this works for now
----
2020-03-25 18:05:47 UTC - Joel Cressy: Yeah, kinda sucks that vault doesn’t have a jwt secrets engine. You can auth to vault with jwt, but vault won’t generate/sign jwt’s for you.
----
2020-03-25 18:06:15 UTC - Addison Higham: we have a WIP plugin that we hope to open source when we get there, just sorta stalled out with lots of other priorities
----
2020-03-25 18:07:31 UTC - Addison Higham: but anyways, does that make sense? We have some tooling in our little CLI tool that can do the vault auth and fetch the token for you. That has made it easier for even teams that aren't totally up on vault yet to just get started by calling our CLI tool to fetch the token
----
2020-03-25 18:07:58 UTC - Joel Cressy: What kind of TTL is on these credentials?
----
2020-03-25 18:08:55 UTC - Addison Higham: oh and one important detail: the ID of the AWS role (or any AWS principal really) that vault uses for templating locations is the `UniqueID` which means that any tooling first needs to resolve role/user/etc to the unique ID before going to fetch the credential
----
2020-03-25 18:09:19 UTC - Addison Higham: (see this doc for details on that: <https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html#identifiers-unique-ids>)
----
2020-03-25 18:09:58 UTC - Joel Cressy: Oh, that’s like access key id for standard iam creds. When you assume role, you have one of these unique ID’s as the access key id along with secret and session/security token.
----
2020-03-25 18:10:20 UTC - Addison Higham: we let the user decide the TTL within a range, but right now, defaults to a year. We still have a lot of apps that aren't native k8s or with an existing vault integration yet. As that evolves, we want to drop the default TTL down to a day or something
----
2020-03-25 18:10:48 UTC - Joel Cressy: Awesome, yeah user configurable TTL is great because I also have a lot of things that don’t have native integrations.
----
2020-03-25 18:11:46 UTC - Joel Cressy: BTW, one of the first use cases I intend to run pulsar for is log ingest/forwarding. I will do traditional pub/sub later but current goals are to provide interfaces for log forwarders to send data (e.g. filebeat, fluentd/fluentbit, etc)
----
2020-03-25 18:11:49 UTC - Addison Higham: yeah, obviously not ideal, but since this is all over internal VPC (privatelink) and we have a plan for apps to be able to use short lived tokens, that is where we are for now
----
2020-03-25 18:12:10 UTC - Addison Higham: heh, yeah, so I actually just wrote a fluentbit plugin
----
2020-03-25 18:12:36 UTC - Addison Higham: via the golang interface, haven't rolled it out at scale yet (next week likely), it isn't open source yet though
----
2020-03-25 18:12:53 UTC - Joel Cressy: I may also need find a way to map non-aws service identities to pulsar, but with vault the sky’s the limit.
----
2020-03-25 18:14:21 UTC - Addison Higham: yeah, so our little microservice we have, it can generate creds for users after an okta login. But yes, the same pattern with vault should work relatively sanely
----
2020-03-25 18:15:19 UTC - Joel Cressy: ooo, yeah we use okta too so i’d have to do something similar. Users auth to vault with okta, so maybe something could be done there.
----
2020-03-25 18:16:58 UTC - Arthur: @Arthur has joined the channel
----
2020-03-25 18:21:51 UTC - Arthur: Hey, I try pulsar on Kubernetes with official helm with only 3 kubernetes nodes. First try, I got affinity/antiaffinity error. I change replication to 1 for zookeeper, booker and broker but now I have "ManagedLedgerException: Not enough non-faulty bookies available". Is it possible to disable antiaffinity ?
<#CJ0FMGHSM|kubernetes>
----
2020-03-25 21:01:15 UTC - Sijie Guo: you need to change the broker config map to reduce ensemble size to 1.
<https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/values.yaml#L231>
I don’t think the current helm chart in master support disabling antiaffinity.
If you are looking for a chart doing so, you can try our chart. <https://github.com/streamnative/charts>
use this values file (<https://github.com/streamnative/charts/blob/master/examples/pulsar/values-minikube.yaml>)
----
2020-03-26 00:24:34 UTC - Hiroyuki Yamada: Hi, I posted some question regarding Key_Shared behavior in the mailing list. Maybe my understanding and expectation are not correct, but it doesn't work as expected so far.
<http://mail-archives.apache.org/mod_mbox/pulsar-users/202003.mbox/browser>
Can anyone help me ?
(BTW, I've tested it with 2.5.0)
----
2020-03-26 00:39:03 UTC - Sijie Guo: Just replied.
----
2020-03-26 01:27:25 UTC - Kannan: Hi, Running Pulsar 2.5.0 with istio 1.5 on AKS(k8s cluster). Is it possible to replace pulsar proxy with istio ingress ?
----
2020-03-26 02:00:29 UTC - Hiroyuki Yamada: Thank you !
I just replied back.
Hmm, seems like it is not working as expected. So, messages with the same key go to different consumers even if there is no new consumer joining the subscription.
----
2020-03-26 02:15:03 UTC - Andy Papia: we are using Keycloak (<http://keycloak.org|keycloak.org>) for authn/authz in our system. has anyone though about creating an authentication plugin for keycloak or something standards-based like OAuth 2?
----
2020-03-26 02:18:02 UTC - Ali Ahmed: @Andy Papia no not yet but it should be simple to do here is a sample external auth plugin for pulsar
----
2020-03-26 02:18:03 UTC - Ali Ahmed: <https://github.com/CleverCloud/biscuit-pulsar>
----
2020-03-26 02:19:23 UTC - Andy Papia: thanks, that's helpful
----
2020-03-26 03:22:41 UTC - Kevin Hui: @Kevin Hui has joined the channel
----
2020-03-26 03:43:56 UTC - Sijie Guo: Interesting, @Penghui Li can you help check this?
man-bowing : Hiroyuki Yamada
----
2020-03-26 03:55:30 UTC - Evan Xu: @Evan Xu has joined the channel
----
2020-03-26 06:25:25 UTC - Amit Aggarwal: @Amit Aggarwal has joined the channel
----
2020-03-26 08:21:32 UTC - Amit Aggarwal: are there any known issues with ttlDurationDefaultInSeconds config setting in pulsar 2.5.0 ?
After setting it to 604800 (7 days), the messages were being expired immediately (5 min after producing)
----
2020-03-26 08:52:10 UTC - Sijie Guo: Can you describe the sequence and how do you verify that?
----