You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "Flutemr (via GitHub)" <gi...@apache.org> on 2023/03/02 02:09:17 UTC
[GitHub] [dolphinscheduler] Flutemr opened a new issue, #13658: Datax synchronizes mysql data, The task fails after about 2 hours each time
Flutemr opened a new issue, #13658:
URL: https://github.com/apache/dolphinscheduler/issues/13658
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
datax synchronizes mysql data. The task fails after about 2 hours each time. However, I used a dolphinscheduler version 2.0.7 that did not fail and ran for over 3 hours, using the same datax version.
3.1.4 and 3.1.3 version fail log and run times:
this 2.0.7 version,and can run succeed:
### What you expected to happen
How to solve
### How to reproduce
no
### Anything else
no
### Version
3.1.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] fuchanghai commented on issue #13658: [Bug]Datax synchronizes mysql data, The task fails after about 2 hours each time
Posted by "fuchanghai (via GitHub)" <gi...@apache.org>.
fuchanghai commented on issue #13658:
URL: https://github.com/apache/dolphinscheduler/issues/13658#issuecomment-1451589949
@Flutemr Can you show the contents of the datax script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13658: [Bug]Datax synchronizes mysql data, The task fails after about 2 hours each time
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13658:
URL: https://github.com/apache/dolphinscheduler/issues/13658#issuecomment-1513946410
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13658: [Bug]Datax synchronizes mysql data, The task fails after about 2 hours each time
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13658:
URL: https://github.com/apache/dolphinscheduler/issues/13658#issuecomment-1500737963
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Flutemr commented on issue #13658: [Bug]Datax synchronizes mysql data, The task fails after about 2 hours each time
Posted by "Flutemr (via GitHub)" <gi...@apache.org>.
Flutemr commented on issue #13658:
URL: https://github.com/apache/dolphinscheduler/issues/13658#issuecomment-1453239225
> @Flutemr Can you show the contents of the datax script
The following is my script, the table data volume is relatively large, synchronization time is long:
{
"job": {
"setting": {
"speed": {
"byte": 5242880
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "name",
"password": "pass",
"column": [
"id",
"capture_type",
"capture_time",
"create_time",
"image_path",
"community_id",
"data_id",
"point_id",
"del_flag",
"file_size",
"big_img"
],
"splitPk": "",
"where":"str_to_date(capture_time,'%Y-%m-%d')='${bizdate}'",
"connection": [
{
"table": [
"tb_zaxq_capture"
],
"jdbcUrl": [
"jdbc:mysql://ip:3306/databases?useSSL=false&serverTimezone=GMT%2B8"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://ns1",
"fileType": "orc",
"path": "/user/hive/warehouse/za_ods.db/ods_za_tb_zaxq_capture/dt=${bizdate1}",
"fileName":"ods_za_tb_zaxq_capture",
"writeMode": "append",
"fieldDelimiter": "\u0001",
"column": [
{"name":"id","type":"string"},
{"name":"capture_type","type":"string"},
{"name":"capture_time","type":"string"},
{"name":"create_time","type":"string"},
{"name":"image_path","type":"string"},
{"name":"community_id","type":"string"},
{"name":"data_id","type":"string"},
{"name":"point_id","type":"string"},
{"name":"del_flag","type":"string"},
{"name":"file_size","type":"string"}
{"name":"big_img","type":"string"}
]
}
}
}
]
},
"core":{
"transport": {
"channel": {
"speed": {
"byte": 1048576
}
}
}
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13658: Datax synchronizes mysql data, The task fails after about 2 hours each time
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13658:
URL: https://github.com/apache/dolphinscheduler/issues/13658#issuecomment-1451168543
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] closed issue #13658: [Bug]Datax synchronizes mysql data, The task fails after about 2 hours each time
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13658: [Bug]Datax synchronizes mysql data, The task fails after about 2 hours each time
URL: https://github.com/apache/dolphinscheduler/issues/13658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org