You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "gzlicanyi (via GitHub)" <gi...@apache.org> on 2023/05/31 08:29:38 UTC
[GitHub] [skywalking-java] gzlicanyi opened a new pull request, #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
gzlicanyi opened a new pull request, #542:
URL: https://github.com/apache/skywalking-java/pull/542
<!--
⚠️ Please make sure to read this template first, pull requests that don't accord with this template
maybe closed without notice.
Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
-->
### Fix <bug description or the bug issue number or bug issue link>
- [x] Add a unit test to verify that the fix works.
- [x] Explain briefly why the bug exists and how to fix it.
When the value of a JSON config in Kafka is a non-string type, such as {"batch.size":32768}, an error will be thrown when setting properties with the generated map.
```
ERROR 2023-05-31 14:56:57.847 SkywalkingAgent-13-org.apache.skywalking.apm.dependencies.kafkaProducerInitThread-0 KafkaProducerManager : unexpected exception.
java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String
at java.util.Map.forEach(Map.java:630)
at org.apache.skywalking.apm.agent.core.kafka.KafkaProducerManager.run(KafkaProducerManager.java:111)
```
Because this code does not deserialize the value of JSON into a String type. However, the value required by Properties is of String type.
```java
Map<String, String> config = (Map<String, String>) gson.fromJson(Kafka.PRODUCER_CONFIG_JSON, Map.class);
```
<!-- ==== 🔌 Remove this line WHEN AND ONLY WHEN you're adding a new plugin, follow the checklist 👇 ====
### Add an agent plugin to support <framework name>
- [ ] Add a test case for the new plugin, refer to [the doc](https://github.com/apache/skywalking-java/blob/main/docs/en/setup/service-agent/java-agent/Plugin-test.md)
- [ ] Add a component id in [the component-libraries.yml](https://github.com/apache/skywalking/blob/master/oap-server/server-starter/src/main/resources/component-libraries.yml)
- [ ] Add a logo in [the UI repo](https://github.com/apache/skywalking-rocketbot-ui/tree/master/src/views/components/topology/assets)
==== 🔌 Remove this line WHEN AND ONLY WHEN you're adding a new plugin, follow the checklist 👆 ==== -->
<!-- ==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👇 ====
### Improve the performance of <class or module or ...>
- [ ] Add a benchmark for the improvement, refer to [the existing ones](https://github.com/apache/skywalking-java/blob/main/apm-commons/apm-datacarrier/src/test/java/org/apache/skywalking/apm/commons/datacarrier/LinkedArrayBenchmark.java)
- [ ] The benchmark result.
```text
<Paste the benchmark results here>
```
- [ ] Links/URLs to the theory proof or discussion articles/blogs. <links/URLs here>
==== 📈 Remove this line WHEN AND ONLY WHEN you're improving the performance, follow the checklist 👆 ==== -->
<!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
### <Feature description>
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [ ] Update the documentation to include this new feature.
- [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking-java/blob/main/CHANGES.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569851089
Could you provide more details? I am not familiar with this kind of Kafka configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] gzlicanyi commented on a diff in pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "gzlicanyi (via GitHub)" <gi...@apache.org>.
gzlicanyi commented on code in PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#discussion_r1211405806
##########
apm-sniffer/optional-reporter-plugins/kafka-reporter-plugin/pom.xml:
##########
@@ -80,6 +80,10 @@
<pattern>com.google.gson</pattern>
<shadedPattern>${shade.package}.com.google.gson</shadedPattern>
</relocation>
+ <relocation>
+ <pattern>com.google.gson</pattern>
+ <shadedPattern>${shade.package}.com.google.common</shadedPattern>
+ </relocation>
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] gzlicanyi commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "gzlicanyi (via GitHub)" <gi...@apache.org>.
gzlicanyi commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569845846
> Didn't you have fixed? #538 I am confused.
When the value of a JSON config in Kafka is a non-string type, such as {"batch.size":32768}, an error will be thrown when setting properties with the generated map.
But such as {"batch.size":"32768"}, it is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569836213
Didn't you have fixed? https://github.com/apache/skywalking-java/pull/538 I am confused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] gzlicanyi commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "gzlicanyi (via GitHub)" <gi...@apache.org>.
gzlicanyi commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569828550
>
#540 hasn't fixed this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] gzlicanyi commented on a diff in pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "gzlicanyi (via GitHub)" <gi...@apache.org>.
gzlicanyi commented on code in PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#discussion_r1211399209
##########
apm-sniffer/optional-reporter-plugins/kafka-reporter-plugin/src/test/java/org/apache/skywalking/apm/agent/core/kafka/KafkaProducerManagerTest.java:
##########
@@ -54,6 +55,17 @@ public void testFormatTopicNameThenRegister() {
assertEquals(KafkaReporterPluginConfig.Plugin.Kafka.TOPIC_METRICS, value);
}
+ @Test
+ public void testSetPropertiesFromJsonConfig() {
+ KafkaProducerManager kafkaProducerManager = new KafkaProducerManager();
+ Properties properties = new Properties();
+
+ KafkaReporterPluginConfig.Plugin.Kafka.PRODUCER_CONFIG_JSON = "{\"batch.size\":32768}";
+ kafkaProducerManager.setPropertiesFromJsonConfig(properties);
+
+ assertEquals(properties.get("batch.size"), "32768");
+ }
+
Review Comment:
Show an example of how a JSON configuration can be correctly set to properties.
##########
apm-sniffer/optional-reporter-plugins/kafka-reporter-plugin/src/test/java/org/apache/skywalking/apm/agent/core/kafka/KafkaProducerManagerTest.java:
##########
@@ -54,6 +55,17 @@ public void testFormatTopicNameThenRegister() {
assertEquals(KafkaReporterPluginConfig.Plugin.Kafka.TOPIC_METRICS, value);
}
+ @Test
+ public void testSetPropertiesFromJsonConfig() {
+ KafkaProducerManager kafkaProducerManager = new KafkaProducerManager();
+ Properties properties = new Properties();
+
+ KafkaReporterPluginConfig.Plugin.Kafka.PRODUCER_CONFIG_JSON = "{\"batch.size\":32768}";
+ kafkaProducerManager.setPropertiesFromJsonConfig(properties);
+
+ assertEquals(properties.get("batch.size"), "32768");
+ }
+
Review Comment:
> What does this UT prove? The codes are not shaded ever in the UT scope, right?
right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] gzlicanyi commented on a diff in pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "gzlicanyi (via GitHub)" <gi...@apache.org>.
gzlicanyi commented on code in PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#discussion_r1211457036
##########
apm-sniffer/optional-reporter-plugins/kafka-reporter-plugin/src/test/java/org/apache/skywalking/apm/agent/core/kafka/KafkaProducerManagerTest.java:
##########
@@ -54,6 +55,17 @@ public void testFormatTopicNameThenRegister() {
assertEquals(KafkaReporterPluginConfig.Plugin.Kafka.TOPIC_METRICS, value);
}
+ @Test
+ public void testSetPropertiesFromJsonConfig() {
+ KafkaProducerManager kafkaProducerManager = new KafkaProducerManager();
+ Properties properties = new Properties();
+
+ KafkaReporterPluginConfig.Plugin.Kafka.PRODUCER_CONFIG_JSON = "{\"batch.size\":32768}";
+ kafkaProducerManager.setPropertiesFromJsonConfig(properties);
+
+ assertEquals(properties.get("batch.size"), "32768");
+ }
+
Review Comment:
> So, the point of this PR is `TypeToken`, which cast the type accordingly.
>
> But still `import com.google.common.reflect.TypeToken;` seems not from gson package. Why?
I have used the wrong TypeToken out of two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng commented on a diff in pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#discussion_r1211431950
##########
apm-sniffer/optional-reporter-plugins/kafka-reporter-plugin/src/test/java/org/apache/skywalking/apm/agent/core/kafka/KafkaProducerManagerTest.java:
##########
@@ -54,6 +55,17 @@ public void testFormatTopicNameThenRegister() {
assertEquals(KafkaReporterPluginConfig.Plugin.Kafka.TOPIC_METRICS, value);
}
+ @Test
+ public void testSetPropertiesFromJsonConfig() {
+ KafkaProducerManager kafkaProducerManager = new KafkaProducerManager();
+ Properties properties = new Properties();
+
+ KafkaReporterPluginConfig.Plugin.Kafka.PRODUCER_CONFIG_JSON = "{\"batch.size\":32768}";
+ kafkaProducerManager.setPropertiesFromJsonConfig(properties);
+
+ assertEquals(properties.get("batch.size"), "32768");
+ }
+
Review Comment:
So, the point of this PR is `TypeToken`, which cast the type accordingly.
But still `import com.google.common.reflect.TypeToken;` seems not from gson package. Why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] gzlicanyi commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "gzlicanyi (via GitHub)" <gi...@apache.org>.
gzlicanyi commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569859202
> Could you provide more details? I am not familiar with this kind of Kafka configuration.
Kafka's batch.size configuration refers to the size of messages sent by the producer in one batch, i.e., the batch size. By default, the batch.size is 16KB. If the size of a message is smaller than batch.size, the producer will wait for a certain amount of time to try to aggregate more messages before sending them together, in order to improve producer efficiency.
Typically, the batch size can be adjusted based on the size of messages sent by the producer to achieve optimal performance. If the messages are small, the batch size can be decreased, and if the messages are large, the batch size can be increased. However, it should be noted that if the batch size is set too large, it can increase the time it takes for messages to be transmitted over the network and increase the risk of message loss.
batch.size is usually configured as a numeric value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng merged pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #542:
URL: https://github.com/apache/skywalking-java/pull/542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng commented on a diff in pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#discussion_r1211397684
##########
apm-sniffer/optional-reporter-plugins/kafka-reporter-plugin/pom.xml:
##########
@@ -80,6 +80,10 @@
<pattern>com.google.gson</pattern>
<shadedPattern>${shade.package}.com.google.gson</shadedPattern>
</relocation>
+ <relocation>
+ <pattern>com.google.gson</pattern>
+ <shadedPattern>${shade.package}.com.google.common</shadedPattern>
+ </relocation>
Review Comment:
`com.google.gson` -> `com.google.common`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng closed pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
URL: https://github.com/apache/skywalking-java/pull/542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569899881
Please fix the conflicts as another PR gets merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng commented on a diff in pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#discussion_r1211393958
##########
apm-sniffer/optional-reporter-plugins/kafka-reporter-plugin/src/test/java/org/apache/skywalking/apm/agent/core/kafka/KafkaProducerManagerTest.java:
##########
@@ -54,6 +55,17 @@ public void testFormatTopicNameThenRegister() {
assertEquals(KafkaReporterPluginConfig.Plugin.Kafka.TOPIC_METRICS, value);
}
+ @Test
+ public void testSetPropertiesFromJsonConfig() {
+ KafkaProducerManager kafkaProducerManager = new KafkaProducerManager();
+ Properties properties = new Properties();
+
+ KafkaReporterPluginConfig.Plugin.Kafka.PRODUCER_CONFIG_JSON = "{\"batch.size\":32768}";
+ kafkaProducerManager.setPropertiesFromJsonConfig(properties);
+
+ assertEquals(properties.get("batch.size"), "32768");
+ }
+
Review Comment:
What does this UT prove? The codes are not shaded ever in the UT scope, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] wu-sheng commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569812237
I think this is duplicated with #540?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-java] gzlicanyi commented on pull request #542: Fix deserialization of kafka producer json config in the kafka-reporter-plugin.
Posted by "gzlicanyi (via GitHub)" <gi...@apache.org>.
gzlicanyi commented on PR #542:
URL: https://github.com/apache/skywalking-java/pull/542#issuecomment-1569842239
> Didn't you have fixed? #538 I am confused.
It's not the same issue. #538 fixed the Gson dependency issue, while #542 fixed the Gson deserialization problem, but both addressed the Kafka JSON config issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org