You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "Firefly4H (via GitHub)" <gi...@apache.org> on 2023/04/11 00:44:01 UTC

[GitHub] [dolphinscheduler] Firefly4H opened a new issue, #13903: Why is the task execution process killed

Firefly4H opened a new issue, #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What happened
   
   
   When the amount of synchronized MySQL mysql data is too large and there is 800w of data, the datax process is killed, and my environment must appear. In addition, if the dolphinscheduler, datax, and target database are on the same machine, modifying the jvm parameters of the dolphinscheduler will not work. System details: Centos8.2.2004, CPU: 32 cores
   Memory: 64G
   
   ![08ed7291bdf344e41e1eec3549b6fd3](https://user-images.githubusercontent.com/32868079/231025974-422e9fbf-49dd-48fd-91ba-ff25d42ac6f4.png)
   ![4bf192ee808d279d517abd1ca4ee5a7](https://user-images.githubusercontent.com/32868079/231025977-5a35f55d-e0a4-4644-b0f1-514f98e474ee.png)
   
   
   ### What you expected to happen
   
   数据量过大比现
   
   ### How to reproduce
   
   数据量过大必现
   
   ### Anything else
   
   _No response_
   
   ### Version
   
   3.1.x
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] fuchanghai commented on issue #13903: Why is the task execution process killed

Posted by "fuchanghai (via GitHub)" <gi...@apache.org>.
fuchanghai commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1510585299

   > `{ "reader": { "parameter": { "password": "*******", "column": [ "RowKey", "DataId", "DataType", "ChannelIndex", "FlowName", "SubFlowName", "ChannelTag1", "ChannelTag2", "CaptureType", "CaptureAngle", "Inputer", "DeviceSN", "DeviceIndex", "StartEventTime", "EventTime", "CaptureTime", "LPRClass", "LPR", "LPRColor", "VehicleClass", "VehicleType" ], "connection": [ { "jdbcUrl": [ "jdbc:mysql://170.31.11.193:3306/enshi_its" ], "table": [ "`VehiclePassCPT_t`" ] } ], "splitPk": "", "username": "root" }, "name": "mysqlreader" }, "writer": { "parameter": { "path": "/vision01/user/hive/warehouse/dcm_first.db/ods_vehiclepasscpt_t_vv5/dt=$[yyyyMMdd]", "fileName": "ods_vehiclepasscpt_t_vv5", "hadoopConfig": { "dfs.namenode.rpc-address.maxvision01.nn2": "170.31.11.202:8020", "dfs.namenode.rpc-address.maxvision01.nn1": "170.31.11.201:8020", "dfs.client.failover.proxy.provider.maxvision01": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "dfs.nameservices": "maxvis
 ion01", "dfs.ha.namenodes.maxvision01": "nn1,nn2" }, "column": [ { "name": "rowkey", "type": "string" }, { "name": "dataid", "type": "string" }, { "name": "datatype", "type": "int" }, { "name": "channelindex", "type": "int" }, { "name": "flowname", "type": "string" }, { "name": "subflowname", "type": "string" }, { "name": "channeltag1", "type": "string" }, { "name": "channeltag2", "type": "string" }, { "name": "capturetype", "type": "int" }, { "name": "captureangle", "type": "int" }, { "name": "inputer", "type": "string" }, { "name": "devicesn", "type": "string" }, { "name": "deviceindex", "type": "int" }, { "name": "starteventtime", "type": "string" }, { "name": "eventtime", "type": "string" }, { "name": "capturetime", "type": "string" }, { "name": "lprclass", "type": "int" }, { "name": "lpr", "type": "string" }, { "name": "lprcolor", "type": "int" }, { "name": "vehicleclass", "type": "int" }, { "name": "vehicletype", "type": "string" } ], "defaultFS": "hdfs://vision01", "writeMode
 ": "truncate", "fieldDelimiter": "|", "fileType": "text" }, "name": "hdfswriter" } }`
   
   can you show ``` setting``` 
   
   ![image](https://user-images.githubusercontent.com/33984497/232360606-14598ecb-51d5-4c40-913b-890904e1ea42.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] Firefly4H commented on issue #13903: Why is the task execution process killed

Posted by "Firefly4H (via GitHub)" <gi...@apache.org>.
Firefly4H commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1510746113

   > > `{ "reader": { "parameter": { "password": "*******", "column": [ "RowKey", "DataId", "DataType", "ChannelIndex", "FlowName", "SubFlowName", "ChannelTag1", "ChannelTag2", "CaptureType", "CaptureAngle", "Inputer", "DeviceSN", "DeviceIndex", "StartEventTime", "EventTime", "CaptureTime", "LPRClass", "LPR", "LPRColor", "VehicleClass", "VehicleType" ], "connection": [ { "jdbcUrl": [ "jdbc:mysql://170.31.11.193:3306/enshi_its" ], "table": [ "`VehiclePassCPT_t`" ] } ], "splitPk": "", "username": "root" }, "name": "mysqlreader" }, "writer": { "parameter": { "path": "/vision01/user/hive/warehouse/dcm_first.db/ods_vehiclepasscpt_t_vv5/dt=$[yyyyMMdd]", "fileName": "ods_vehiclepasscpt_t_vv5", "hadoopConfig": { "dfs.namenode.rpc-address.maxvision01.nn2": "170.31.11.202:8020", "dfs.namenode.rpc-address.maxvision01.nn1": "170.31.11.201:8020", "dfs.client.failover.proxy.provider.maxvision01": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "dfs.nameservices": "maxv
 ision01", "dfs.ha.namenodes.maxvision01": "nn1,nn2" }, "column": [ { "name": "rowkey", "type": "string" }, { "name": "dataid", "type": "string" }, { "name": "datatype", "type": "int" }, { "name": "channelindex", "type": "int" }, { "name": "flowname", "type": "string" }, { "name": "subflowname", "type": "string" }, { "name": "channeltag1", "type": "string" }, { "name": "channeltag2", "type": "string" }, { "name": "capturetype", "type": "int" }, { "name": "captureangle", "type": "int" }, { "name": "inputer", "type": "string" }, { "name": "devicesn", "type": "string" }, { "name": "deviceindex", "type": "int" }, { "name": "starteventtime", "type": "string" }, { "name": "eventtime", "type": "string" }, { "name": "capturetime", "type": "string" }, { "name": "lprclass", "type": "int" }, { "name": "lpr", "type": "string" }, { "name": "lprcolor", "type": "int" }, { "name": "vehicleclass", "type": "int" }, { "name": "vehicletype", "type": "string" } ], "defaultFS": "hdfs://vision01", "writeMo
 de": "truncate", "fieldDelimiter": "|", "fileType": "text" }, "name": "hdfswriter" } }`
   > 
   > can you show ` setting`
   > 
   > ![image](https://user-images.githubusercontent.com/33984497/232360606-14598ecb-51d5-4c40-913b-890904e1ea42.png)
   
   ![{23e83464-a544-455b-8183-8ef877a8192f}](https://user-images.githubusercontent.com/32868079/232394640-6890ded1-4812-4258-9fca-9ba67e7df0ef.png)
   The test found that there is a problem with the mysql database and mysql driver version used by dolphinscheduler. After reducing the mysql version to a lower level, it can be executed normally, which is the mysql I used before. It is version 8.0.3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] github-actions[bot] closed issue #13903: Why is the task execution process killed

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13903: Why is the task execution process killed
URL: https://github.com/apache/dolphinscheduler/issues/13903


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13903: Why is the task execution process killed

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1556049811

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] fuchanghai commented on issue #13903: Why is the task execution process killed

Posted by "fuchanghai (via GitHub)" <gi...@apache.org>.
fuchanghai commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1510576714

   It appears to have received a kill request from another service,not because of OOM @SbloodyS WDYT?
   ![image](https://user-images.githubusercontent.com/33984497/232358796-c9117167-95a3-4194-8267-ca380067d7fa.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] sqxinquan commented on issue #13903: Why is the task execution process killed

Posted by "sqxinquan (via GitHub)" <gi...@apache.org>.
sqxinquan commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1509437588

   This is a datax configuration problem, which I have also encountered.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] liu-hai commented on issue #13903: Why is the task execution process killed

Posted by "liu-hai (via GitHub)" <gi...@apache.org>.
liu-hai commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1502550482

   I also encountered the same bug, hope to fix soon, datax task node more than 2G of data will be killed halfway through the first run
   Here is my datax node configuration:
   [
   ![image](https://user-images.githubusercontent.com/22786504/231029551-5fa9d58b-60f7-4b0b-a2d1-922d8a64ca90.png)
   ](url)
   [
   ![image](https://user-images.githubusercontent.com/22786504/231029749-7d1a7945-e7e8-4950-bb7b-2fd3e5ecde34.png)
   ](url)
   
   job.json configuration :
   `{
       "reader": {
           "parameter": {
               "password": "*******",
               "column": [
                   "RowKey",
                   "DataId",
                   "DataType",
                   "ChannelIndex",
                   "FlowName",
                   "SubFlowName",
                   "ChannelTag1",
                   "ChannelTag2",
                   "CaptureType",
                   "CaptureAngle",
                   "Inputer",
                   "DeviceSN",
                   "DeviceIndex",
                   "StartEventTime",
                   "EventTime",
                   "CaptureTime",
                   "LPRClass",
                   "LPR",
                   "LPRColor",
                   "VehicleClass",
                   "VehicleType"
               ],
               "connection": [
                   {
                       "jdbcUrl": [
                           "jdbc:mysql://170.31.11.193:3306/enshi_its"
                       ],
                       "table": [
                           "`VehiclePassCPT_t`"
                       ]
                   }
               ],
               "splitPk": "",
               "username": "root"
           },
           "name": "mysqlreader"
       },
       "writer": {
           "parameter": {
               "path": "/vision01/user/hive/warehouse/dcm_first.db/ods_vehiclepasscpt_t_vv5/dt=$[yyyyMMdd]",
               "fileName": "ods_vehiclepasscpt_t_vv5",
               "hadoopConfig": {
                   "dfs.namenode.rpc-address.maxvision01.nn2": "170.31.11.202:8020",
                   "dfs.namenode.rpc-address.maxvision01.nn1": "170.31.11.201:8020",
                   "dfs.client.failover.proxy.provider.maxvision01": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
                   "dfs.nameservices": "maxvision01",
                   "dfs.ha.namenodes.maxvision01": "nn1,nn2"
               },
               "column": [
                   {
                       "name": "rowkey",
                       "type": "string"
                   },
                   {
                       "name": "dataid",
                       "type": "string"
                   },
                   {
                       "name": "datatype",
                       "type": "int"
                   },
                   {
                       "name": "channelindex",
                       "type": "int"
                   },
                   {
                       "name": "flowname",
                       "type": "string"
                   },
                   {
                       "name": "subflowname",
                       "type": "string"
                   },
                   {
                       "name": "channeltag1",
                       "type": "string"
                   },
                   {
                       "name": "channeltag2",
                       "type": "string"
                   },
                   {
                       "name": "capturetype",
                       "type": "int"
                   },
                   {
                       "name": "captureangle",
                       "type": "int"
                   },
                   {
                       "name": "inputer",
                       "type": "string"
                   },
                   {
                       "name": "devicesn",
                       "type": "string"
                   },
                   {
                       "name": "deviceindex",
                       "type": "int"
                   },
                   {
                       "name": "starteventtime",
                       "type": "string"
                   },
                   {
                       "name": "eventtime",
                       "type": "string"
                   },
                   {
                       "name": "capturetime",
                       "type": "string"
                   },
                   {
                       "name": "lprclass",
                       "type": "int"
                   },
                   {
                       "name": "lpr",
                       "type": "string"
                   },
                   {
                       "name": "lprcolor",
                       "type": "int"
                   },
                   {
                       "name": "vehicleclass",
                       "type": "int"
                   },
                   {
                       "name": "vehicletype",
                       "type": "string"
                   }
               ],
               "defaultFS": "hdfs://vision01",
               "writeMode": "truncate",
               "fieldDelimiter": "|",
               "fileType": "text"
           },
           "name": "hdfswriter"
       }
   }`
   
   The following is the node run log, you can see that it was killed in the middle of the run,
   不知道什么原因被杀死了,因为节点配置了失败重试次数为1,第一次失败之后重试又可以成功,只要mysql表的数据量大一点这个问题就必然出现,我这边是将mysql数据往hive写
   [
   ![image](https://user-images.githubusercontent.com/22786504/231030326-9b59642d-492a-40b0-b9ec-66c61e23259c.png)
   ](url)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] Firefly4H commented on issue #13903: Why is the task execution process killed

Posted by "Firefly4H (via GitHub)" <gi...@apache.org>.
Firefly4H commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1510542447

   > This is a datax configuration problem, which I have also encountered.
   
   May I ask what the correct configuration is


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] Firefly4H commented on issue #13903: Why is the task execution process killed

Posted by "Firefly4H (via GitHub)" <gi...@apache.org>.
Firefly4H commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1515743337

   > > > > `{ "reader": { "parameter": { "password": "*******", "column": [ "RowKey", "DataId", "DataType", "ChannelIndex", "FlowName", "SubFlowName", "ChannelTag1", "ChannelTag2", "CaptureType", "CaptureAngle", "Inputer", "DeviceSN", "DeviceIndex", "StartEventTime", "EventTime", "CaptureTime", "LPRClass", "LPR", "LPRColor", "VehicleClass", "VehicleType" ], "connection": [ { "jdbcUrl": [ "jdbc:mysql://170.31.11.193:3306/enshi_its" ], "table": [ "`VehiclePassCPT_t`" ] } ], "splitPk": "", "username": "root" }, "name": "mysqlreader" }, "writer": { "parameter": { "path": "/vision01/user/hive/warehouse/dcm_first.db/ods_vehiclepasscpt_t_vv5/dt=$[yyyyMMdd]", "fileName": "ods_vehiclepasscpt_t_vv5", "hadoopConfig": { "dfs.namenode.rpc-address.maxvision01.nn2": "170.31.11.202:8020", "dfs.namenode.rpc-address.maxvision01.nn1": "170.31.11.201:8020", "dfs.client.failover.proxy.provider.maxvision01": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "dfs.nameservices": "
 maxvision01", "dfs.ha.namenodes.maxvision01": "nn1,nn2" }, "column": [ { "name": "rowkey", "type": "string" }, { "name": "dataid", "type": "string" }, { "name": "datatype", "type": "int" }, { "name": "channelindex", "type": "int" }, { "name": "flowname", "type": "string" }, { "name": "subflowname", "type": "string" }, { "name": "channeltag1", "type": "string" }, { "name": "channeltag2", "type": "string" }, { "name": "capturetype", "type": "int" }, { "name": "captureangle", "type": "int" }, { "name": "inputer", "type": "string" }, { "name": "devicesn", "type": "string" }, { "name": "deviceindex", "type": "int" }, { "name": "starteventtime", "type": "string" }, { "name": "eventtime", "type": "string" }, { "name": "capturetime", "type": "string" }, { "name": "lprclass", "type": "int" }, { "name": "lpr", "type": "string" }, { "name": "lprcolor", "type": "int" }, { "name": "vehicleclass", "type": "int" }, { "name": "vehicletype", "type": "string" } ], "defaultFS": "hdfs://vision01", "wri
 teMode": "truncate", "fieldDelimiter": "|", "fileType": "text" }, "name": "hdfswriter" } }`
   > > > 
   > > > 
   > > > can you show ` setting`
   > > > ![image](https://user-images.githubusercontent.com/33984497/232360606-14598ecb-51d5-4c40-913b-890904e1ea42.png)
   > > 
   > > 
   > > ![{23e83464-a544-455b-8183-8ef877a8192f}](https://user-images.githubusercontent.com/32868079/232394640-6890ded1-4812-4258-9fca-9ba67e7df0ef.png)
   > > The test found a problem with the mysql database and mysql driver version used by dolphinsscheduler. After lowering the mysql version, it can be executed normally. The problematic mysql version is 8.0.32.
   > 
   > Hi @Firefly4H I think this is because the jar package of version 5.x is used by default in the `DataX` component,not because of DS <img alt="image" width="1423" src="https://user-images.githubusercontent.com/33984497/233016316-e9172680-1a9e-489c-9728-3beafd2af38c.png">
   
   I don't think so. Using datax alone to run tasks is not a problem, as it can occur when used together with DS


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13903: Why is the task execution process killed

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1502524706

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What happened
   
   
   When the amount of synchronized MySQL mysql data is too large and there is 800w of data, the datax process is killed, and my environment must appear. In addition, if the dolphinscheduler, datax, and target database are on the same machine, modifying the jvm parameters of the dolphinscheduler will not work. System details: Centos8.2.2004, CPU: 32 cores
   Memory: 64G
   
   ![08ed7291bdf344e41e1eec3549b6fd3](https://user-images.githubusercontent.com/32868079/231025974-422e9fbf-49dd-48fd-91ba-ff25d42ac6f4.png)
   ![4bf192ee808d279d517abd1ca4ee5a7](https://user-images.githubusercontent.com/32868079/231025977-5a35f55d-e0a4-4644-b0f1-514f98e474ee.png)
   
   
   ### What you expected to happen
   
   数据量过大比现
   
   ### How to reproduce
   
   数据量过大必现
   
   ### Anything else
   
   _No response_
   
   ### Version
   
   3.1.x
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] fuchanghai commented on issue #13903: Why is the task execution process killed

Posted by "fuchanghai (via GitHub)" <gi...@apache.org>.
fuchanghai commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1514339786

   > > > `{ "reader": { "parameter": { "password": "*******", "column": [ "RowKey", "DataId", "DataType", "ChannelIndex", "FlowName", "SubFlowName", "ChannelTag1", "ChannelTag2", "CaptureType", "CaptureAngle", "Inputer", "DeviceSN", "DeviceIndex", "StartEventTime", "EventTime", "CaptureTime", "LPRClass", "LPR", "LPRColor", "VehicleClass", "VehicleType" ], "connection": [ { "jdbcUrl": [ "jdbc:mysql://170.31.11.193:3306/enshi_its" ], "table": [ "`VehiclePassCPT_t`" ] } ], "splitPk": "", "username": "root" }, "name": "mysqlreader" }, "writer": { "parameter": { "path": "/vision01/user/hive/warehouse/dcm_first.db/ods_vehiclepasscpt_t_vv5/dt=$[yyyyMMdd]", "fileName": "ods_vehiclepasscpt_t_vv5", "hadoopConfig": { "dfs.namenode.rpc-address.maxvision01.nn2": "170.31.11.202:8020", "dfs.namenode.rpc-address.maxvision01.nn1": "170.31.11.201:8020", "dfs.client.failover.proxy.provider.maxvision01": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "dfs.nameservices": "ma
 xvision01", "dfs.ha.namenodes.maxvision01": "nn1,nn2" }, "column": [ { "name": "rowkey", "type": "string" }, { "name": "dataid", "type": "string" }, { "name": "datatype", "type": "int" }, { "name": "channelindex", "type": "int" }, { "name": "flowname", "type": "string" }, { "name": "subflowname", "type": "string" }, { "name": "channeltag1", "type": "string" }, { "name": "channeltag2", "type": "string" }, { "name": "capturetype", "type": "int" }, { "name": "captureangle", "type": "int" }, { "name": "inputer", "type": "string" }, { "name": "devicesn", "type": "string" }, { "name": "deviceindex", "type": "int" }, { "name": "starteventtime", "type": "string" }, { "name": "eventtime", "type": "string" }, { "name": "capturetime", "type": "string" }, { "name": "lprclass", "type": "int" }, { "name": "lpr", "type": "string" }, { "name": "lprcolor", "type": "int" }, { "name": "vehicleclass", "type": "int" }, { "name": "vehicletype", "type": "string" } ], "defaultFS": "hdfs://vision01", "write
 Mode": "truncate", "fieldDelimiter": "|", "fileType": "text" }, "name": "hdfswriter" } }`
   > > 
   > > 
   > > can you show ` setting`
   > > ![image](https://user-images.githubusercontent.com/33984497/232360606-14598ecb-51d5-4c40-913b-890904e1ea42.png)
   > 
   > ![{23e83464-a544-455b-8183-8ef877a8192f}](https://user-images.githubusercontent.com/32868079/232394640-6890ded1-4812-4258-9fca-9ba67e7df0ef.png)
   > 
   > The test found a problem with the mysql database and mysql driver version used by dolphinsscheduler. After lowering the mysql version, it can be executed normally. The problematic mysql version is 8.0.32.
   
   I think this is because the jar package of version 5.x is used by default in the ```DataX``` component,not because of DS
   <img width="1423" alt="image" src="https://user-images.githubusercontent.com/33984497/233016316-e9172680-1a9e-489c-9728-3beafd2af38c.png">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] Firefly4H commented on issue #13903: Why is the task execution process killed

Posted by "Firefly4H (via GitHub)" <gi...@apache.org>.
Firefly4H commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1504835489

   > exit code 137代表datax中的OOM。你可以检查datax的配置。
   
   1. Modifying the datax memory and dolphinscheduler memory does not take effect.
   
   2. There is no problem running only datax.
   
   3. There is no problem running datax tasks in Dolphinscheduler standalone mode.
   
   Simply using the dolphinscheduler cluster will appear, and the cluster configuration is as follows:
   ![1681286409875](https://user-images.githubusercontent.com/32868079/231392006-11dc564b-d895-4946-a2c3-772850b25366.jpg)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13903: Why is the task execution process killed

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1567617476

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13903: Why is the task execution process killed

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1502524839

   Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
   * In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
   * If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on issue #13903: Why is the task execution process killed

Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #13903:
URL: https://github.com/apache/dolphinscheduler/issues/13903#issuecomment-1502998760

   Exit code 137 represents OOM in datax. You can check the config of datax.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org