You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/05/06 02:11:32 UTC
[GitHub] [pulsar] lanwen opened a new issue #6883: Java Pulsar function can't find SerDe class on schema init
lanwen opened a new issue #6883:
URL: https://github.com/apache/pulsar/issues/6883
**Describe the bug**
I’ve faced weird classpath issue when running in cluster mode, so I prepared a little project to show an issue
Pulsar can’t find my SerDe implementation, even its same package as the function (and it finds the function without issues)
https://gist.github.com/lanwen/769a1204b26f8392c3929c7457a980e2
project is here https://github.com/lanwen/pulsar-functions-example
and the test class
https://github.com/lanwen/pulsar-functions-example/blob/master/tests/src/test/java/ru/lanwen/pulsar/functions/IntegrationTest.java#L132
starting this test its expected to see both input and output, but only that exception is thrown. Other modes with another 3 methods work fine.
see also the screenshot
After some investigation I found that it for some reason instantiates a new schema for the input/output and at that moment something weird happens with the classloader, which couldn't load my SerDe implementation. I don't really understand why it ever goes into schema creation.
I even debugged to a direct run of my function to debug it easier, but failed to catch an origin of the issue
```
java -agentlib:jdwp='transport=dt_socket,server=y,suspend=n,address=*:5006' \
-cp '/Users/lanwen/code/github.com/lanwen/pulsar-functions-example/apache-pulsar-2.5.0/instances/java-instance.jar:/Users/lanwen/code/github.com/lanwen/pulsar-functions-example/apache-pulsar-2.5.0/instances/deps/*' \
-Dpulsar.functions.extra.dependencies.dir='/Users/lanwen/code/github.com/lanwen/pulsar-functions-example/apache-pulsar-2.5.0/instances/deps' \
-Dpulsar.functions.instance.classpath='/Users/lanwen/code/github.com/lanwen/pulsar-functions-example/apache-pulsar-2.5.0/conf:::/Users/lanwen/code/github.com/lanwen/pulsar-functions-example/apache-pulsar-2.5.0/lib/*:' \
-Dlog4j.configurationFile=java_instance_log4j2.xml \
-Dpulsar.function.log.dir=/tmp/functions/public/default/serde2 \
-Dpulsar.function.log.file=serde2-0 \
-Xmx1073741824 \
org.apache.pulsar.functions.instance.JavaInstanceMain \
--jar \
/Users/lanwen/code/github.com/lanwen/pulsar-functions-example/apache-pulsar-2.5.0/download/pulsar_functions/public/default/serde2/0/functions-all.jar \
--instance_id 0 \
--function_id cb7b9ab0-c075-4e87-875e-eaf6c0b3eaab \
--function_version c7f649ad-772d-4b4b-b812-ec3a4bacc0f4 \
--function_details '{"tenant":"public","namespace":"default","name":"serde2","className":"ru.lanwen.pulsar.functions.SerDeFunction","logTopic":"test-log-topic","autoAck":true,"parallelism":1,"source":{"subscriptionType":"FAILOVER","typeClassName":"com.fasterxml.jackson.databind.JsonNode","inputSpecs":{"test-inputs-topic":{"serdeClassName":"ru.lanwen.pulsar.functions.TreeNodeSerDe"}},"cleanupSubscription":true},"sink":{"topic":"test-output-topic","serDeClassName":"ru.lanwen.pulsar.functions.TreeNodeSerDe","typeClassName":"com.fasterxml.jackson.databind.JsonNode"},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"},"retryDetails":{"maxMessageRetries":2,"deadLetterTopic":"test-dlq-topic"},"componentType":"FUNCTION"}' \
--pulsar_serviceurl pulsar://localhost:6650 \
--max_buffered_tuples 1024 \
--port 49639 \
--metrics_port 49640 \
--state_storage_serviceurl bk://127.0.0.1:4181 \
--expected_healthcheck_interval 30 \
--secrets_provider org.apache.pulsar.functions.secretsprovider.ClearTextSecretsProvider \
--cluster_name standalone
```
**To Reproduce**
Steps to reproduce the behavior:
1. clone the project and run the test https://github.com/lanwen/pulsar-functions-example/blob/master/tests/src/test/java/ru/lanwen/pulsar/functions/IntegrationTest.java#L132 via gradle (so the gradle would build appropriate jar)
2. check the log for the output, which would show only input topic, but not the output. Test also logs a command to grab a function log from container with the exception same as in gist
3. Test will timeout after 10 min, to leave some time for debug
(its not actually a test, just a runner)
**Expected behavior**
Function should run
**Screenshots**
**Desktop (please complete the following information):**
- osx, docker, 2.5.0/2.5.1
**Additional context**
Didn't actually find any working SerDe example with a proper command
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #6883: Java Pulsar function can't find SerDe class on schema init
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #6883:
URL: https://github.com/apache/pulsar/issues/6883#issuecomment-624977869
@gaoran10 This issue is related to one of the issues you are working on. Can you also check this issue as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org