You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/06/03 02:48:10 UTC
[GitHub] [skywalking] myfairys opened a new issue #7061: when i use kafka-reporter-plugin,i have some problems about kafka producer config
myfairys opened a new issue #7061:
URL: https://github.com/apache/skywalking/issues/7061
### when i use kafka-reporter-plugin,i have some problems about kafka producer config.
when i set config : -Dskywalking.plugin.kafka.producer_config[compression.type]=snappy, the agent is also error, as follow:
```
ERROR 2021-06-03 10:27:16:442 SkywalkingAgent-8-ServiceManagementClientKafkaProducer-0 KafkaServiceManagementServiceClient : unexpected exception.
org.apache.skywalking.apm.dependencies.org.apache.kafka.common.KafkaException: java.lang.UnsatisfiedLinkError: org.apache.skywalking.apm.dependencies.org.xerial.snappy.SnappyNative.maxCompressedLength(I)I
at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.CompressionType$3.wrapForOutput(CompressionType.java:83)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.MemoryRecordsBuilder.<init>(MemoryRecordsBuilder.java:134)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.MemoryRecordsBuilder.<init>(MemoryRecordsBuilder.java:170)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:508)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:490)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:438)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.internals.RecordAccumulator.recordsBuilder(RecordAccumulator.java:253)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:229)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:965)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:886)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
at org.apache.skywalking.apm.agent.core.kafka.KafkaServiceManagementServiceClient.run(KafkaServiceManagementServiceClient.java:110)
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsatisfiedLinkError: org.apache.skywalking.apm.dependencies.org.xerial.snappy.SnappyNative.maxCompressedLength(I)I
at org.apache.skywalking.apm.dependencies.org.xerial.snappy.SnappyNative.maxCompressedLength(Native Method)
at org.apache.skywalking.apm.dependencies.org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:380)
at org.apache.skywalking.apm.dependencies.org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:99)
at org.apache.skywalking.apm.dependencies.org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:91)
at org.apache.skywalking.apm.dependencies.org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:81)
at org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.CompressionType$3.wrapForOutput(CompressionType.java:81)
... 19 more
```
when i analysis the source code of 'org.xerial.snappy.SnappyNative.maxCompressedLength(I)I' ,i can find some code
the kafka producer compression type class is fixed，when use maven-shade-plugin to resolve conflict, it's only modify import class,but can not modify these fixed className。
### Bug
I use skywalking 8.1.0, windows,and jdk8
Is this a bug or other？
Do you have any ideas？Please ask me，thanks。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #7061: CompressionType unusable in kafka-reporter-plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #7061:
URL: https://github.com/apache/skywalking/issues/7061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] myfairys commented on issue #7061: CompressionType unusable in kafka-reporter-plugin
Posted by GitBox <gi...@apache.org>.
myfairys commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853527515
> > I think it's because it uses reflection (class name in literal strings), which shade plugin can't cover that case
>
> I can see a lot of same cases in `org.apache.kafka.common.record.CompressionType`. In my mind, only solution is adding a patch plugin in the Kafka reporter for `org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.CompressionType#findConstructor`, and manipulate the input. This is hacking work, but should work.
But How to do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #7061: when i use kafka-reporter-plugin,i have some problems about kafka producer config
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853521340
I think it's because it uses reflection (class name in literal strings), which shade plugin can't cover that case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7061: CompressionType unusable in kafka-reporter-plugin
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853528002
> > > I think it's because it uses reflection (class name in literal strings), which shade plugin can't cover that case
> >
> >
> > I can see a lot of same cases in `org.apache.kafka.common.record.CompressionType`. In my mind, only solution is adding a patch plugin in the Kafka reporter for `org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.CompressionType#findConstructor`, and manipulate the input. This is hacking work, but should work.
>
> But How to do that?
What I gave is development work, not a simple setting. This requires you already know SkyWalking plugin mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7061: when i use kafka-reporter-plugin,i have some problems about kafka producer config
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853522849
> I think it's because it uses reflection (class name in literal strings), which shade plugin can't cover that case
I can see a lot of same cases in `org.apache.kafka.common.record.CompressionType`. In my mind, only solution is adding a patch plugin in the Kafka reporter for `org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.CompressionType#findConstructor`, and manipulate the input. This is hacking work, but should work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7061: when i use kafka-reporter-plugin,i have some problems about kafka producer config
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853520730
I am not sure whether we included `compression` of Kafka. @dmsolr Any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] myfairys commented on issue #7061: CompressionType unusable in kafka-reporter-plugin
Posted by GitBox <gi...@apache.org>.
myfairys commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853527515
> > I think it's because it uses reflection (class name in literal strings), which shade plugin can't cover that case
>
> I can see a lot of same cases in `org.apache.kafka.common.record.CompressionType`. In my mind, only solution is adding a patch plugin in the Kafka reporter for `org.apache.skywalking.apm.dependencies.org.apache.kafka.common.record.CompressionType#findConstructor`, and manipulate the input. This is hacking work, but should work.
But How to do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7061: when i use kafka-reporter-plugin,i have some problems about kafka producer config
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853520730
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #7061: when i use kafka-reporter-plugin,i have some problems about kafka producer config
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853521340
I think it's because it uses reflection (class name in literal strings), which shade plugin can't cover that case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] myfairys commented on issue #7061: CompressionType unusable in kafka-reporter-plugin
Posted by GitBox <gi...@apache.org>.
myfairys commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-854517716
> What I gave is development work, not a simple setting. This requires you already know SkyWalking plugin mechanism
But i'm sorry, i'm no idea do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7061: when i use kafka-reporter-plugin,i have some problems about kafka producer config
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7061:
URL: https://github.com/apache/skywalking/issues/7061#issuecomment-853521065
We added `org.apache.skywalking.apm.dependencies.` to all Kafka codes for avoiding class conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org