You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/09/08 03:26:35 UTC
[GitHub] [incubator-dolphinscheduler] JinyLeeChina opened a new pull request #3689: fix #3688, Optimization of table building statements
JinyLeeChina opened a new pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689
1. When executing MySQL table creation statement, it will report that the index length is not enough.
this will happen: Index column size too large. The maximum column size is 767 bytes.
2. The length of the t_ds_task_instance.app_link is not enough. When hive SQL is complex and yarn will start multiple applications
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yangyichao-mango commented on a change in pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
yangyichao-mango commented on a change in pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#discussion_r485029335
##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -248,7 +248,7 @@ CREATE TABLE `QRTZ_TRIGGERS` (
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_STATE`),
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE_GRP` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_GROUP`,`TRIGGER_STATE`),
CONSTRAINT `QRTZ_TRIGGERS_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `QRTZ_JOB_DETAILS` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
Why need to use `utf8` to replace `utd8mb4`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] sonarcloud[bot] commented on pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#issuecomment-688600514
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/proje
ct/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=3689&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=3689&metric=coverage&view=list) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo.png' alt='No Duplication information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=3689&metric=duplicated_lines_density&view=list) No Duplication information
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] lgcareer commented on a change in pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
lgcareer commented on a change in pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#discussion_r496458886
##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -248,7 +248,7 @@ CREATE TABLE `QRTZ_TRIGGERS` (
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_STATE`),
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE_GRP` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_GROUP`,`TRIGGER_STATE`),
CONSTRAINT `QRTZ_TRIGGERS_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `QRTZ_JOB_DETAILS` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
I will send a email in the dev list,we can discuss it which charset we need to choose.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] felix-thinkingdata commented on a change in pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
felix-thinkingdata commented on a change in pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#discussion_r485297560
##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -248,7 +248,7 @@ CREATE TABLE `QRTZ_TRIGGERS` (
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_STATE`),
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE_GRP` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_GROUP`,`TRIGGER_STATE`),
CONSTRAINT `QRTZ_TRIGGERS_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `QRTZ_JOB_DETAILS` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
https://www.cnblogs.com/cuiqq/p/11045487.html
I read this post and didn't quite understand the need to change the coding format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] lgcareer commented on a change in pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
lgcareer commented on a change in pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#discussion_r496457845
##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -248,7 +248,7 @@ CREATE TABLE `QRTZ_TRIGGERS` (
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_STATE`),
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE_GRP` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_GROUP`,`TRIGGER_STATE`),
CONSTRAINT `QRTZ_TRIGGERS_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `QRTZ_JOB_DETAILS` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
Hi,we should discuss first,the utf8mb4 is suggested to use in mysql.You can see the article below.
https://medium.com/@adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434
I saw the error you pointed `Index column size too large. The maximum column size is 767 bytes.`The another article give a method to resolve it.
https://www.jianshu.com/p/5c0cce24189b
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong commented on pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
dailidong commented on pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#issuecomment-721105492
> @JinyLeeChina @yangyichao-mango @ALL
> I sent a email to discuss that which charst should we choose in mysql,We can discuss in the email thread below.
> https://lists.apache.org/thread.html/r6a1ebb8bf5767571cdabfc625cad62611f9b04d9c06671668e9ebef7%40%3Cdev.dolphinscheduler.apache.org%3E
the conclusion is to use utf8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong merged pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
dailidong merged pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] JinyLeeChina commented on a change in pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
JinyLeeChina commented on a change in pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#discussion_r485567005
##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -248,7 +248,7 @@ CREATE TABLE `QRTZ_TRIGGERS` (
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_STATE`),
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE_GRP` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_GROUP`,`TRIGGER_STATE`),
CONSTRAINT `QRTZ_TRIGGERS_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `QRTZ_JOB_DETAILS` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
Version 5.6, I think there are many users of this version of MySQL, we should be compatible
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] JinyLeeChina commented on a change in pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
JinyLeeChina commented on a change in pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#discussion_r485302693
##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -248,7 +248,7 @@ CREATE TABLE `QRTZ_TRIGGERS` (
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_STATE`),
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE_GRP` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_GROUP`,`TRIGGER_STATE`),
CONSTRAINT `QRTZ_TRIGGERS_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `QRTZ_JOB_DETAILS` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
I checked the qrtz related tables and will not write 4-byte Unicode. Utf8mb4 encoding will cause the index of qrtz related tables to exceed the default value of MySQL, which will cause many users to fail to create tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] lgcareer commented on pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
lgcareer commented on pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#issuecomment-700529314
@JinyLeeChina @yangyichao-mango @all
I sent a email to discuss that which charst should we choose in mysql,We can discuss in the email thread below.
https://lists.apache.org/thread.html/r6a1ebb8bf5767571cdabfc625cad62611f9b04d9c06671668e9ebef7%40%3Cdev.dolphinscheduler.apache.org%3E
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] yangyichao-mango commented on a change in pull request #3689: fix #3688, Optimization of table building statements
Posted by GitBox <gi...@apache.org>.
yangyichao-mango commented on a change in pull request #3689:
URL: https://github.com/apache/incubator-dolphinscheduler/pull/3689#discussion_r485337724
##########
File path: sql/dolphinscheduler_mysql.sql
##########
@@ -248,7 +248,7 @@ CREATE TABLE `QRTZ_TRIGGERS` (
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_STATE`),
KEY `IDX_QRTZ_T_NFT_ST_MISFIRE_GRP` (`SCHED_NAME`,`MISFIRE_INSTR`,`NEXT_FIRE_TIME`,`TRIGGER_GROUP`,`TRIGGER_STATE`),
CONSTRAINT `QRTZ_TRIGGERS_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `QRTZ_JOB_DETAILS` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Review comment:
> I checked the qrtz related tables and will not write 4-byte Unicode. Utf8mb4 encoding will cause the index of qrtz related tables to exceed the default value of MySQL, which will cause many users to fail to create tables.
>
> 
>
>
Which version of your mysql?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org