You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/07/18 11:39:50 UTC
[GitHub] [incubator-dolphinscheduler] BoYiZhang opened a new issue #3240: [Bug][worker] Cannot find a (Map) Key deserializer for type
BoYiZhang opened a new issue #3240:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3240
When I execute SQL query task, JSON parsing error
error info :
````
com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
Cannot find a (Map) Key deserializer for type [simple type, class org.apache.dolphinscheduler.dao.entity.UdfFunc]
`````
show code :
TaskExecutionContext taskExecutionContext = JSONUtils.parseObject(contextJson, TaskExecutionContext.class);
test unit :
```
@Test
public void testJson(){
String contextJson = "{\"taskInstanceId\":2,\"taskName\":\"SQL-QUERY\",\"startTime\":\"2020-07-18 19:18:58\",\"taskType\":\"SQL\",\"host\":null,\"executePath\":\"/tmp/dolphinscheduler/exec/process/2/17/2/2\",\"logPath\":null,\"taskJson\":\"{\\\"id\\\":\\\"tasks-5736\\\",\\\"name\\\":\\\"SQL-QUERY\\\",\\\"desc\\\":null,\\\"type\\\":\\\"SQL\\\",\\\"runFlag\\\":\\\"NORMAL\\\",\\\"loc\\\":null,\\\"maxRetryTimes\\\":0,\\\"retryInterval\\\":1,\\\"params\\\":{\\\"type\\\":\\\"MYSQL\\\",\\\"datasource\\\":3,\\\"sql\\\":\\\"SELECT * FROM person\\\",\\\"udfs\\\":\\\"\\\",\\\"sqlType\\\":\\\"0\\\",\\\"title\\\":\\\"SQL-QUERY\\\",\\\"receivers\\\":\\\"zhangboyi_mx@163.com\\\",\\\"receiversCc\\\":\\\"\\\",\\\"showType\\\":\\\"TABLE\\\",\\\"localParams\\\":[],\\\"connParams\\\":\\\"\\\",\\\"preStatements\\\":[],\\\"postStatements\\\":[]},\\\"preTasks\\\":[],\\\"extras\\\":null,\\\"depList\\\":[],\\\"dependence\\\":{},\\\"conditionResult\\\":{\\\"successNode\\\":[\\\"\\\"],\\\"failedNode
\\\":[\\\"\\\"]},\\\"taskInstancePriority\\\":\\\"MEDIUM\\\",\\\"workerGroup\\\":\\\"default\\\",\\\"workerGroupId\\\":null,\\\"timeout\\\":{\\\"strategy\\\":\\\"\\\",\\\"interval\\\":null,\\\"enable\\\":false},\\\"conditionsTask\\\":false,\\\"forbidden\\\":false,\\\"taskTimeoutParameter\\\":{\\\"enable\\\":false,\\\"strategy\\\":null,\\\"interval\\\":0}}\",\"processId\":0,\"appIds\":null,\"processInstanceId\":2,\"scheduleTime\":null,\"globalParams\":null,\"executorId\":2,\"cmdTypeIfComplement\":0,\"tenantCode\":\"sysadmin\",\"queue\":\"default\",\"processDefineId\":17,\"projectId\":2,\"taskParams\":null,\"envFile\":null,\"definedParams\":null,\"taskAppId\":null,\"taskTimeoutStrategy\":0,\"taskTimeout\":0,\"workerGroup\":\"default\",\"resources\":{},\"sqlTaskExecutionContext\":{\"warningGroupId\":0,\"connectionParams\":\"{\\\"type\\\":null,\\\"address\\\":\\\"jdbc:mysql://127.0.0.1:3306\\\",\\\"database\\\":\\\"test\\\",\\\"jdbcUrl\\\":\\\"jdbc:mysql://127.0.0.1:3306/test\\\",\\\"us
er\\\":\\\"root\\\",\\\"password\\\":\\\"IUAjJCVeJipyb290\\\"}\",\"udfFuncTenantCodeMap\":null},\"dataxTaskExecutionContext\":{\"dataSourceId\":0,\"sourcetype\":0,\"sourceConnectionParams\":null,\"dataTargetId\":0,\"targetType\":0,\"targetConnectionParams\":null},\"dependenceTaskExecutionContext\":null,\"sqoopTaskExecutionContext\":{\"dataSourceId\":0,\"sourcetype\":0,\"sourceConnectionParams\":null,\"dataTargetId\":0,\"targetType\":0,\"targetConnectionParams\":null},\"procedureTaskExecutionContext\":{\"connectionParams\":null}}" ;
TaskExecutionContext taskExecutionContext = JSONUtils.parseObject(contextJson, TaskExecutionContext.class);
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yangyichao-mango commented on issue #3240: [Bug][worker] Cannot find a (Map) Key deserializer for type
Posted by GitBox <gi...@apache.org>.
yangyichao-mango commented on issue #3240:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3240#issuecomment-660576523
> When I execute SQL query task, JSON parsing error
>
> error info :
>
> ```
>
> com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
> Cannot find a (Map) Key deserializer for type [simple type, class org.apache.dolphinscheduler.dao.entity.UdfFunc]
>
> ```
>
> 
>
> show code :
>
> TaskExecutionContext taskExecutionContext = JSONUtils.parseObject(contextJson, TaskExecutionContext.class);
> 
>
> contextJson:
>
> ```
>
> {
> "taskInstanceId":2,
> "taskName":"SQL-QUERY",
> "startTime":"2020-07-18 19:18:58",
> "taskType":"SQL",
> "host":null,
> "executePath":"/tmp/dolphinscheduler/exec/process/2/17/2/2",
> "logPath":null,
> "taskJson":"{"id":"tasks-5736","name":"SQL-QUERY","desc":null,"type":"SQL","runFlag":"NORMAL","loc":null,"maxRetryTimes":0,"retryInterval":1,"params":{"type":"MYSQL","datasource":3,"sql":"SELECT * FROM person","udfs":"","sqlType":"0","title":"SQL-QUERY","receivers":"zhangboyi_mx@163.com","receiversCc":"","showType":"TABLE","localParams":[],"connParams":"","preStatements":[],"postStatements":[]},"preTasks":[],"extras":null,"depList":[],"dependence":{},"conditionResult":{"successNode":[""],"failedNode":[""]},"taskInstancePriority":"MEDIUM","workerGroup":"default","workerGroupId":null,"timeout":{"strategy":"","interval":null,"enable":false},"conditionsTask":false,"forbidden":false,"taskTimeoutParameter":{"enable":false,"strategy":null,"interval":0}}",
> "processId":0,
> "appIds":null,
> "processInstanceId":2,
> "scheduleTime":null,
> "globalParams":null,
> "executorId":2,
> "cmdTypeIfComplement":0,
> "tenantCode":"sysadmin",
> "queue":"default",
> "processDefineId":17,
> "projectId":2,
> "taskParams":null,
> "envFile":null,
> "definedParams":null,
> "taskAppId":null,
> "taskTimeoutStrategy":0,
> "taskTimeout":0,
> "workerGroup":"default",
> "resources":{
>
> },
> "sqlTaskExecutionContext":{
> "warningGroupId":0,
> "connectionParams":"{"type":null,"address":"jdbc:mysql://127.0.0.1:3306","database":"test","jdbcUrl":"jdbc:mysql://127.0.0.1:3306/test","user":"root","password":"IUAjJCVeJipyb290"}",
> "udfFuncTenantCodeMap":null
> },
> "dataxTaskExecutionContext":{
> "dataSourceId":0,
> "sourcetype":0,
> "sourceConnectionParams":null,
> "dataTargetId":0,
> "targetType":0,
> "targetConnectionParams":null
> },
> "dependenceTaskExecutionContext":null,
> "sqoopTaskExecutionContext":{
> "dataSourceId":0,
> "sourcetype":0,
> "sourceConnectionParams":null,
> "dataTargetId":0,
> "targetType":0,
> "targetConnectionParams":null
> },
> "procedureTaskExecutionContext":{
> "connectionParams":null
> }
> }
> ```
>
> test unit :
>
> ```
>
> public class TaskExecuteProcessorTest {
>
> @Test
> public void testJson(){
> String contextJson = "{\n" +
> " \"taskInstanceId\":2,\n" +
> " \"taskName\":\"SQL-QUERY\",\n" +
> " \"startTime\":\"2020-07-18 19:18:58\",\n" +
> " \"taskType\":\"SQL\",\n" +
> " \"host\":null,\n" +
> " \"executePath\":\"/tmp/dolphinscheduler/exec/process/2/17/2/2\",\n" +
> " \"logPath\":null,\n" +
> " \"taskJson\":\"{\"id\":\"tasks-5736\",\"name\":\"SQL-QUERY\",\"desc\":null,\"type\":\"SQL\",\"runFlag\":\"NORMAL\",\"loc\":null,\"maxRetryTimes\":0,\"retryInterval\":1,\"params\":{\"type\":\"MYSQL\",\"datasource\":3,\"sql\":\"SELECT * FROM person\",\"udfs\":\"\",\"sqlType\":\"0\",\"title\":\"SQL-QUERY\",\"receivers\":\"zhangboyi_mx@163.com\",\"receiversCc\":\"\",\"showType\":\"TABLE\",\"localParams\":[],\"connParams\":\"\",\"preStatements\":[],\"postStatements\":[]},\"preTasks\":[],\"extras\":null,\"depList\":[],\"dependence\":{},\"conditionResult\":{\"successNode\":[\"\"],\"failedNode\":[\"\"]},\"taskInstancePriority\":\"MEDIUM\",\"workerGroup\":\"default\",\"workerGroupId\":null,\"timeout\":{\"strategy\":\"\",\"interval\":null,\"enable\":false},\"conditionsTask\":false,\"forbidden\":false,\"taskTimeoutParameter\":{\"enable\":false,\"strategy\":null,\"interval\":0}}\",\n" +
> " \"processId\":0,\n" +
> " \"appIds\":null,\n" +
> " \"processInstanceId\":2,\n" +
> " \"scheduleTime\":null,\n" +
> " \"globalParams\":null,\n" +
> " \"executorId\":2,\n" +
> " \"cmdTypeIfComplement\":0,\n" +
> " \"tenantCode\":\"sysadmin\",\n" +
> " \"queue\":\"default\",\n" +
> " \"processDefineId\":17,\n" +
> " \"projectId\":2,\n" +
> " \"taskParams\":null,\n" +
> " \"envFile\":null,\n" +
> " \"definedParams\":null,\n" +
> " \"taskAppId\":null,\n" +
> " \"taskTimeoutStrategy\":0,\n" +
> " \"taskTimeout\":0,\n" +
> " \"workerGroup\":\"default\",\n" +
> " \"resources\":{\n" +
> "\n" +
> " },\n" +
> " \"sqlTaskExecutionContext\":{\n" +
> " \"warningGroupId\":0,\n" +
> " \"connectionParams\":\"{\"type\":null,\"address\":\"jdbc:mysql://127.0.0.1:3306\",\"database\":\"test\",\"jdbcUrl\":\"jdbc:mysql://127.0.0.1:3306/test\",\"user\":\"root\",\"password\":\"IUAjJCVeJipyb290\"}\",\n" +
> " \"udfFuncTenantCodeMap\":null\n" +
> " },\n" +
> " \"dataxTaskExecutionContext\":{\n" +
> " \"dataSourceId\":0,\n" +
> " \"sourcetype\":0,\n" +
> " \"sourceConnectionParams\":null,\n" +
> " \"dataTargetId\":0,\n" +
> " \"targetType\":0,\n" +
> " \"targetConnectionParams\":null\n" +
> " },\n" +
> " \"dependenceTaskExecutionContext\":null,\n" +
> " \"sqoopTaskExecutionContext\":{\n" +
> " \"dataSourceId\":0,\n" +
> " \"sourcetype\":0,\n" +
> " \"sourceConnectionParams\":null,\n" +
> " \"dataTargetId\":0,\n" +
> " \"targetType\":0,\n" +
> " \"targetConnectionParams\":null\n" +
> " },\n" +
> " \"procedureTaskExecutionContext\":{\n" +
> " \"connectionParams\":null\n" +
> " }\n" +
> "}" ;
>
> TaskExecutionContext taskExecutionContext = JSONUtils.parseObject(contextJson, TaskExecutionContext.class);
>
> assertTrue(taskExecutionContext != null);
> }
> }
> ```
>
> error info :
>
> ```
> Connected to the target VM, address: '127.0.0.1:53429', transport: 'socket'
> 19:43:31.405 [main] ERROR org.apache.dolphinscheduler.common.utils.JSONUtils - parse object exception!
> com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot find a (Map) Key deserializer for type [simple type, class org.apache.dolphinscheduler.dao.entity.UdfFunc]
> at [Source: (String)"{
> "taskInstanceId":2,
> "taskName":"SQL-QUERY",
> "startTime":"2020-07-18 19:18:58",
> "taskType":"SQL",
> "host":null,
> "executePath":"/tmp/dolphinscheduler/exec/process/2/17/2/2",
> "logPath":null,
> "taskJson":"{"id":"tasks-5736","name":"SQL-QUERY","desc":null,"type":"SQL","runFlag":"NORMAL","loc":null,"maxRetryTimes":0,"retryInterval":1,"params":{"type":"MYSQL","datasource":3,"sql":"SELECT * FROM person","udfs":"","sqlType":"0","title":"SQL-QUERY","receivers":"zhangboyi_mx"[truncated 1787 chars]; line: 1, column: 1]
> at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
> at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1452)
> at com.fasterxml.jackson.databind.deser.DeserializerCache._handleUnknownKeyDeserializer(DeserializerCache.java:599)
> at com.fasterxml.jackson.databind.deser.DeserializerCache.findKeyDeserializer(DeserializerCache.java:168)
> at com.fasterxml.jackson.databind.DeserializationContext.findKeyDeserializer(DeserializationContext.java:500)
> at com.fasterxml.jackson.databind.deser.std.MapDeserializer.createContextual(MapDeserializer.java:248)
> at com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:651)
> at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:484)
> at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
> at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
> at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
> at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:467)
> at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:473)
> at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
> at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
> at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
> at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
> at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4190)
> at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4009)
> at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3004)
> at org.apache.dolphinscheduler.common.utils.JSONUtils.parseObject(JSONUtils.java:108)
> at org.apache.dolphinscheduler.server.worker.processor.TaskExecuteProcessorTest.testJson(TaskExecuteProcessorTest.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> Disconnected from the target VM, address: '127.0.0.1:53429', transport: 'socket'
>
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at org.apache.dolphinscheduler.server.worker.processor.TaskExecuteProcessorTest.testJson(TaskExecuteProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> ```
Hi,
Which version and branch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] lenboo closed issue #3240: [Bug][worker] Cannot find a (Map) Key deserializer for type
Posted by GitBox <gi...@apache.org>.
lenboo closed issue #3240:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3240
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] hsupu commented on issue #3240: [Bug][worker] Cannot find a (Map) Key deserializer for type
Posted by GitBox <gi...@apache.org>.
hsupu commented on issue #3240:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3240#issuecomment-660579775
+1
same exception on shell task with 1.3.2-SNAPSHOT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] liukunyuan commented on issue #3240: [Bug][worker] Cannot find a (Map) Key deserializer for type
Posted by GitBox <gi...@apache.org>.
liukunyuan commented on issue #3240:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3240#issuecomment-660634635
+1
dev branch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CycloneBoy commented on issue #3240: [Bug][worker] Cannot find a (Map) Key deserializer for type
Posted by GitBox <gi...@apache.org>.
CycloneBoy commented on issue #3240:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3240#issuecomment-660643122
> The problem is that when Jackson serializes the Map, it will parse the object corresponding to the Map's Key into a string by default; but this string format is not in the Json format, by adding the serialization parsing class and deserialization of the custom Map key Parsing class, can realize the Json serialization/deserialization of CategoryEntity object．
The problem is described in the following link： https://stackoverflow.com/questions/11246748/deserializing-non-string-map-keys-with-jackson
Due to serialization： TaskExecutionContext－＞SQLTaskExecutionContext －＞Map<UdfFunc,String> udfFuncTenantCodeMap．The key of the Map need to serialization/deserialization .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org