You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/08/31 10:30:28 UTC
[GitHub] [dolphinscheduler] ruanwenjun opened a new issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
ruanwenjun opened a new issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072
**Describe the bug**
This method is hope to deserialize a JSON to Map.
https://github.com/apache/dolphinscheduler/blob/e866d1be86464d812551e8c38ba60767a204c82e/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/JSONUtils.java#L249-L251
But the generics type in the Map is not what we expected.
We can write a simple test case:
```java
String sqlResult = "{\"id\":[],\"test1\":\"6\"}";
Map<String, List> param = toMap(sqlResult, String.class, List.class);
System.out.println(param);
```
It will execute success, and print
```
{id=[], test1=6}
```
The value type of `id` is `List`, the value type of `test1` is `String`.
But If we loop the map, it will throw a exception, because the value in this map has different type
```java
String sqlResult = "{\"id\":[],\"test1\":\"6\"}";
Map<String, List> param = toMap(sqlResult, String.class, List.class);
param.forEach((key, value) -> System.out.println(value));
```
```
[]
Exception in thread "main" java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.apache.dolphinscheduler.common.utils.JSONUtils.main(JSONUtils.java:256)
```
I don't find out the deep reason, but I think we should remove this method.
**Which version of Dolphin Scheduler:**
-[dev]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou edited a comment on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou edited a comment on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-921359935
> @EricPyZhou Great, assign to you.
Hi, just spent time setting up the dev environment for the task. One problem occurred during running MasterServer is that the error says _plugin dir not exists ! lib/plugin/registry_
I therefore changed the registry.plugin.dir to dolphinscheduler-dist/target/dolphinscheduler-dist-1.3.6-SNAPSHOT/lib/plugin/registry/zookeeper according to the instruction.
But the terminal freezes when starting the MasterServer again
Here is my registry.properties setting just for convience:
Anything I can do to try to debug?
BTW: I didnt do this line because there is no existing zookeeper.preperties file in the module _Modify the service module try to change the zookeeper.quorum part of the zookeeper.properties file zookeeper.quorum=localhost:2181_
UPDATE:
By Adding stdout and comment out appender-ref that does not its corresponding appender, the Master Server starts
<root level="INFO">
<appender-ref ref="STDOUT"/> <!-- 添加控制台输出 -->
<!-- <appender-ref ref="TASKLOGFILE"/>-->
<!-- <appender-ref ref="WORKERLOGFILE"/>-->
<!-- <appender-ref ref="SKYWALKING-LOG"/>-->
</root>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-922634505
> @EricPyZhou The second way is not a good practice, in most times, we don't use Map<String, Object>, it lost the compiled type check. The first way is good, and the `JSONUtils.parseObject` is use the first way. You need to remove the `toMap` method in `JSONUtils`.
Correct,and I think I also need to modify code that currently uses `toMap` method to `JSONUtils.parseObject`, is that okay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-909110054
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-922683281
@EricPyZhou Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou edited a comment on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou edited a comment on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-921359935
> @EricPyZhou Great, assign to you.
Hi, just spent time setting up the dev environment for the task. One problem occurred during running MasterServer is that the error says _plugin dir not exists ! lib/plugin/registry_
I therefore changed the registry.plugin.dir to dolphinscheduler-dist/target/dolphinscheduler-dist-1.3.6-SNAPSHOT/lib/plugin/registry/zookeeper according to the instruction.
But the terminal freezes when starting the MasterServer again
Here is my registry.properties setting just for convience:
Anything I can do to try to debug?
BTW: I didnt do this line because there is no existing zookeeper.preperties file in the module _Modify the service module try to change the zookeeper.quorum part of the zookeeper.properties file zookeeper.quorum=localhost:2181_
UPDATE:
By Adding stdout and comment out appender-ref that does not its corresponding appender, the Master Server starts
<root level="INFO">
<appender-ref ref="STDOUT"/> <!-- 添加控制台输出 -->
<!-- <appender-ref ref="TASKLOGFILE"/>-->
<!-- <appender-ref ref="WORKERLOGFILE"/>-->
<!-- <appender-ref ref="SKYWALKING-LOG"/>-->
</root>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou edited a comment on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou edited a comment on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-922606441
@ruanwenjun @zhuangchong By my own test method:
```
@Test
public void simpleTest() {
String sqlResult = "{\"id\":[],\"test1\":\"7\"}";
// first way
Map<String, Object> param = JSONUtils.parseObject(sqlResult, new TypeReference<Map<String, Object>>() {});
System.out.println(param + " " +param.getClass().getSimpleName());
param.forEach((key, value) -> System.out.println(value + " " + value.getClass().getSimpleName()));
// second way
Map<String, Object> param2 = JSONUtils.toMap(sqlResult, String.class, Object.class);
System.out.println(param2 + " " +param2.getClass().getSimpleName());
param2.forEach((key, value) -> System.out.println(value + " " + value.getClass().getSimpleName()));
}
```
There is no difference in between when dealing with such hashmap, and I saw alot of code written as the **second** way, for example in [ExecutorServiceImpl.java](https://github.com/apache/dolphinscheduler/blob/69a153c5f5c9d91b4eececbcfbf9e2fb407f89c4/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java#L273)
Shall I remove this method (toMap(String json, Class<K> classK, Class<V> classV)) and replace it with parseObject as in the **first** way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-909110054
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun edited a comment on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
ruanwenjun edited a comment on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-922618198
@EricPyZhou The second way is not a good practice, in most times, we don't use Map<String, Object>, it lost the compiled type check. The first way is good, and the `JSONUtils.parseObject` is use the first way. You need to remove the `toMap` method in `JSONUtils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-921359935
> @EricPyZhou Great, assign to you.
Hi, just spent time setting up the dev environment for the task. One problem occurred during running MasterServer is that the error says _plugin dir not exists ! lib/plugin/registry_
I therefore changed the registry.plugin.dir to dolphinscheduler-dist/target/dolphinscheduler-dist-1.3.6-SNAPSHOT/lib/plugin/registry/zookeeper according to the instruction.
But the terminal freezes when starting the MasterServer again
Here is my registry.properties setting just for convience:
Anything I can do to try to debug?
BTW: I didnt do this line because there is no existing zookeeper.preperties file in the module _Modify the service module try to change the zookeeper.quorum part of the zookeeper.properties file zookeeper.quorum=localhost:2181_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-923601468
@ruanwenjun Please have a look at this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-911435640
> @ruanwenjun
>
> It is necessary to remove this method and use parseObject(String json, TypeReference type) instead.
Ok, this can be a good first issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-909110054
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-922618198
@EricPyZhou The second way is not a good practice, in most times, we don't use Map<String, Object>, it lost the compiled type check. The first way is good, and the `JSONUtils.parseObject` is use the first way. You need to remove the toMap method in JSONUtils.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun closed issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
ruanwenjun closed issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-921399499
the env setup is done. will examine a way to first reproduce the bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-922606441
@ruanwenjun @zhuangchong By my own test method:
```
@Test
public void simpleTest() {
String sqlResult = "{\"id\":[],\"test1\":\"7\"}";
// first way
Map<String, Object> param = JSONUtils.parseObject(sqlResult, new TypeReference<Map<String, Object>>() {});
System.out.println(param + " " +param.getClass().getSimpleName());
param.forEach((key, value) -> System.out.println(value + " " + value.getClass().getSimpleName()));
// second way
Map<String, Object> param2 = JSONUtils.toMap(sqlResult, String.class, Object.class);
System.out.println(param2 + " " +param2.getClass().getSimpleName());
param.forEach((key, value) -> System.out.println(value + " " + value.getClass().getSimpleName()));
}
```
There is no difference in between when dealing with such hashmap, and I saw alot of code written as the **second** way, for example in [ExecutorServiceImpl.java](https://github.com/apache/dolphinscheduler/blob/69a153c5f5c9d91b4eececbcfbf9e2fb407f89c4/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java#L273)
Shall I remove this method (toMap(String json, Class<K> classK, Class<V> classV)) and replace it with parseObject as in the **first** way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhuangchong commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
zhuangchong commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-911429546
@ruanwenjun
It is necessary to remove this method and use parseObject(String json, TypeReference<T> type) instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricPyZhou commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
EricPyZhou commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-921024176
if currently, no one is working on this issue, Can I take a look at this one? More details on the fix will be provided by the end of this week (Sept, 19th, EDT) since there is already a suggestion fix in the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #6072: [Bug][common] Generics type is invalid in JsonUtils.toMap
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #6072:
URL: https://github.com/apache/dolphinscheduler/issues/6072#issuecomment-921344273
@EricPyZhou Great, assign to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org