You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "gmiklos-ltg (via GitHub)" <gi...@apache.org> on 2023/04/13 12:49:30 UTC
[GitHub] [pulsar] gmiklos-ltg opened a new pull request, #20090: [fix][fn] Use functions classloader in TopicSchema
gmiklos-ltg opened a new pull request, #20090:
URL: https://github.com/apache/pulsar/pull/20090
Fixes #5350
### Motivation
While evaluating Pulsar Functions for our data transformation use-cases we ran into the following startup issue with our function that was using a custom SerDe class:
`2023-04-12T11:30:35,368+0200 [public/default/user-transform-new-2-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [public/default/user-transform-new-2:0] Uncaught exception in Java Instance
java.lang.RuntimeException: User class must be in class path
at org.apache.pulsar.common.util.Reflections.createInstance(Reflections.java:72) ~[org.apache.pulsar-pulsar-common-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.instance.InstanceUtils.createInstance(InstanceUtils.java:92) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.instance.InstanceUtils.initializeSerDe(InstanceUtils.java:51) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.source.TopicSchema.newSchemaInstance(TopicSchema.java:238) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.source.TopicSchema.newSchemaInstance(TopicSchema.java:246) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.source.TopicSchema.lambda$getSchema$0(TopicSchema.java:68) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
at org.apache.pulsar.functions.source.TopicSchema.getSchema(TopicSchema.java:68) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.source.PulsarSource.buildPulsarSourceConsumerConfig(PulsarSource.java:176) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.source.SingleConsumerPulsarSource.open(SingleConsumerPulsarSource.java:69) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupInput(JavaInstanceRunnable.java:774) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setup(JavaInstanceRunnable.java:226) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:259) ~[org.apache.pulsar-pulsar-functions-instance-2.11.0.jar:2.11.0]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.ClassNotFoundException: com.bridge.data.UserMessageSerDe
at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:587) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[?:?]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:467) ~[?:?]
at org.apache.pulsar.common.util.Reflections.createInstance(Reflections.java:70) ~[org.apache.pulsar-pulsar-common-2.11.0.jar:2.11.0]
... 13 more`
We made it absolutely sure that the class is included in the fat jar that we have compiled.
config.yaml looks like this:
```
tenant: public
namespace: default
name: user-transform
className: com.bridge.data.TransformUserMessage
inputs:
- "persistent://public/default/debezium.public.users"
customSerdeInputs:
"persistent://public/default/debezium.public.users": com.bridge.data.UserMessageSerDe
output: "persistent://public/default/debezium.public.users-public"
outputSerdeClassName: com.bridge.data.PublicUserSerDe
jar: "transform-function-v1.jar"
parallelism: 1
retainOrdering: true
```
The mentioned `UserMessageSerDe` class is just a very simple wrapper around Jackson to deserialize UserMessage data classes.
### Modifications
I have modified TopicSchema in such a way that it accepts a functions classloader parameter which will be used by `newSchemaInstance()` to resolve this class. This way our function can now start up without any issues.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change is already covered by existing tests, such as `TopicSchemaTest`.
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: [(https://github.com/gmiklos-ltg/pulsar/pull/2)]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gmiklos-ltg commented on pull request #20090: [fix][fn] Use functions classloader in TopicSchema
Posted by "gmiklos-ltg (via GitHub)" <gi...@apache.org>.
gmiklos-ltg commented on PR #20090:
URL: https://github.com/apache/pulsar/pull/20090#issuecomment-1511002224
> Generally, we firstly review a PR against master branch and pick it to maintained branches. Unless this is a 2.11 specific issue, please close this one and open a new PR against master branch first.
Hi @tisonkun! I have prepared a PR for master here:
https://github.com/apache/pulsar/pull/20115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tisonkun commented on pull request #20090: [fix][fn] Use functions classloader in TopicSchema
Posted by "tisonkun (via GitHub)" <gi...@apache.org>.
tisonkun commented on PR #20090:
URL: https://github.com/apache/pulsar/pull/20090#issuecomment-1507865911
Generally, we firstly review a PR against master branch and pick it to maintained branches. Unless this is a 2.11 specific issue, please close this one and open a new PR against master branch first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gmiklos-ltg closed pull request #20090: [fix][fn] Use functions classloader in TopicSchema
Posted by "gmiklos-ltg (via GitHub)" <gi...@apache.org>.
gmiklos-ltg closed pull request #20090: [fix][fn] Use functions classloader in TopicSchema
URL: https://github.com/apache/pulsar/pull/20090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org