You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2023/05/18 15:39:53 UTC
[GitHub] [hadoop] slfan1989 opened a new pull request, #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
slfan1989 opened a new pull request, #5673:
URL: https://github.com/apache/hadoop/pull/5673
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
JIRA: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209399807
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
Thank you very much for helping to review this pr!
`YARN-8337` describes a scenario where we use the Mysql database, and when multiple sessions call the sp_addApplicationHomeSubCluster stored procedure at the same time, a deadlock may occur.
MySQL Gap lock is a type of gap lock used to protect a range of gaps instead of specific data rows. We can look at the following picture.
<img width="684" alt="image" src="https://github.com/apache/hadoop/assets/55643692/bef31a2d-20b7-40b8-a007-0b3d07bffc65">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209399807
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
Thank you very much for helping to review this pr!
`YARN-8337` describes a scenario where we use the Mysql database, and when multiple sessions call the sp_addApplicationHomeSubCluster stored procedure at the same time, a deadlock may occur.
MySQL Gap lock is a type of gap lock used to protect a range of gaps instead of specific data rows. We can look at the following picture.
<img width="684" alt="image" src="https://github.com/apache/hadoop/assets/55643692/bef31a2d-20b7-40b8-a007-0b3d07bffc65">
When we need to write data to MySQL, for example `application_1526528662010_001201`.
- Step1, we need to apply for a gap lock there is a gap lock between the previous record(`application_1526528662010_001199`) and infinity. Prevent other transactions from inserting new records in this range.
- Step2, After applying for the gap lock, `application_1526528662010_001201` will apply for the row lock and complete the record writing.
If we have 3 different Sessions applying to insert different records at the same time, SessionA writes `application_1526528662010_001201`, SessionB writes `application_1526528662010_001202`, and SessionC writes `application_1526528662010_001203`. They may hold different gap locks, but need to acquire row locks at the same time to continue execution, resulting in a deadlock.
But fortunately, there is a timeout period for different sessions, and the deadlock condition will be terminated. These three applications can finally write successfully, but this will affect the performance of data writing.
`INSERT IGNORE INTO` allows to ignore errors in the insert process and continue to perform subsequent operations when errors are encountered. This means that `INSERT IGNORE INTO` can more easily handle duplicate data or insertion errors, and can improve the throughput of insert operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1210495371
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
Thank you very much for your question! I will try to find a better way to handle this code to make it run more smoothly. In the process of submitting applications, we have added retry logic so that even if this part of the code fails, we will not miss application information. I will set up a complete Mysql test environment and submit a large number of apps in a multi-session situation to confirm the performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209399807
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
Thank you very much for helping to review this pr!
`YARN-8337` describes a scenario where we use the Mysql database, and when multiple sessions call the sp_addApplicationHomeSubCluster stored procedure at the same time, a deadlock may occur.
MySQL Gap lock is a type of gap lock used to protect a range of gaps instead of specific data rows. We can look at the following picture.
<img width="684" alt="image" src="https://github.com/apache/hadoop/assets/55643692/bef31a2d-20b7-40b8-a007-0b3d07bffc65">
When we need to write data to MySQL, for example `application_1526528662010_001201`, there is a Gap lock between the previous record(`application_1526528662010_001199`) and infinity. Prevent other transactions from inserting new records in this range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209421573
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestSQLFederationStateStore.java:
##########
@@ -104,7 +105,7 @@ protected FederationStateStore createStateStore() {
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_PASSWORD,
DATABASE_PASSWORD);
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_URL,
- DATABASE_URL + System.currentTimeMillis());
+ DATABASE_URL + System.currentTimeMillis() + MYSQL_COMPATIBILITY);
Review Comment:
INSERT IGNORE INTO is Mysql syntax, not applicable to other databases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#issuecomment-1553747564
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 30m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 42s | | trunk passed |
| +1 :green_heart: | compile | 7m 33s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 6m 45s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 10s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 27s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 12m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 6m 43s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 6m 43s | | the patch passed |
| +1 :green_heart: | compile | 6m 28s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 6m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 53s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 18s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 12m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 233m 26s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 3m 27s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 419m 29s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5673 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 1ec97b3db018 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c76882f0d4617bd8c7f2fc047a2c28f8151f03f9 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/1/testReport/ |
| Max. process+thread count | 2773 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209399807
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
Thank you very much for helping to review this pr!
`YARN-8337` describes a scenario where we use the Mysql database, and when multiple sessions call the sp_addApplicationHomeSubCluster stored procedure at the same time, a deadlock may occur.
MySQL Gap lock is a type of gap lock used to protect a range of gaps instead of specific data rows. We can look at the following picture.
<img width="684" alt="image" src="https://github.com/apache/hadoop/assets/55643692/bef31a2d-20b7-40b8-a007-0b3d07bffc65">
When we need to write data to MySQL, for example `application_1526528662010_001201`.
- Step1, we need to apply for a gap lock there is a gap lock between the previous record(`application_1526528662010_001199`) and infinity. Prevent other transactions from inserting new records in this range.
- Step2, After applying for the gap lock, `application_1526528662010_001201` will apply for the row lock and complete the record writing.
If we have 3 different Sessions applying to insert different records at the same time, SessionA writes `application_1526528662010_001201`, SessionB writes `application_1526528662010_001202`, and SessionC writes `application_1526528662010_001203`. They may hold different gap locks, but need to acquire row locks at the same time to continue execution, resulting in a deadlock.
But fortunately, there is a timeout period for different sessions, and the deadlock condition will be terminated. These three applications can finally write successfully, but this will affect the performance of data writing.
`INSERT IGNORE INTO` allows to ignore errors in the insert process and continue to perform subsequent operations when errors are encountered. This means that `INSERT IGNORE INTO` can more easily handle duplicate data or insertion errors, and can improve the throughput of insert operations.
`INSERT INTO` is used to insert new rows into the table. If an inserted row overlaps an existing row, `INSERT INTO `will result in an error and the entire transaction will be rolled back. Therefore, `INSERT INTO` performs stricter verification of the correctness of the inserted data, but may have an impact on concurrency.
Our scenario is that multiple Routers write data at the same time, and write different application data. `INSERT IGNORE INTO` should be more appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209399807
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
Thank you very much for helping to review this pr!
`YARN-8337` describes a scenario where we use the Mysql database, and when multiple sessions call the sp_addApplicationHomeSubCluster stored procedure at the same time, a deadlock may occur.
MySQL Gap lock is a type of gap lock used to protect a range of gaps instead of specific data rows. We can look at the following picture.
<img width="684" alt="image" src="https://github.com/apache/hadoop/assets/55643692/bef31a2d-20b7-40b8-a007-0b3d07bffc65">
When we need to write data to MySQL, for example `application_1526528662010_001201`.
- Step1, we need to apply for a gap lock there is a gap lock between the previous record(`application_1526528662010_001199`) and infinity. Prevent other transactions from inserting new records in this range.
- Step2, After applying for the gap lock, `application_1526528662010_001201` will apply for the row lock and complete the record writing.
If we have 3 different Sessions applying to insert different records at the same time, SessionA writes `application_1526528662010_001201`, SessionB writes `application_1526528662010_001202`, and SessionC writes `application_1526528662010_001203`. They may hold different gap locks, but need to acquire row locks at the same time to continue execution, resulting in a deadlock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "goiri (via GitHub)" <gi...@apache.org>.
goiri commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209479728
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestSQLFederationStateStore.java:
##########
@@ -104,7 +105,7 @@ protected FederationStateStore createStateStore() {
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_PASSWORD,
DATABASE_PASSWORD);
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_URL,
- DATABASE_URL + System.currentTimeMillis());
+ DATABASE_URL + System.currentTimeMillis() + MYSQL_COMPATIBILITY);
Review Comment:
I meant the MYSQL_COMPATIBILITY flag.
This test is run for other SQL implementations right? How do they handle that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1210511646
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestSQLFederationStateStore.java:
##########
@@ -104,7 +105,7 @@ protected FederationStateStore createStateStore() {
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_PASSWORD,
DATABASE_PASSWORD);
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_URL,
- DATABASE_URL + System.currentTimeMillis());
+ DATABASE_URL + System.currentTimeMillis() + MYSQL_COMPATIBILITY);
Review Comment:
I will refactor this part of the code to be compatible with other SQL implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "goiri (via GitHub)" <gi...@apache.org>.
goiri commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1209479472
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
I wasn't aware of synchronization issues with MySQL.
Isn't there some explicit atomic that would help mitigate this?
What happens if we ignore the insertion here? Can we miss apps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "goiri (via GitHub)" <gi...@apache.org>.
goiri commented on code in PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#discussion_r1208050097
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestSQLFederationStateStore.java:
##########
@@ -104,7 +105,7 @@ protected FederationStateStore createStateStore() {
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_PASSWORD,
DATABASE_PASSWORD);
conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_URL,
- DATABASE_URL + System.currentTimeMillis());
+ DATABASE_URL + System.currentTimeMillis() + MYSQL_COMPATIBILITY);
Review Comment:
Is this ignored by others?
##########
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreStoredProcs.sql:
##########
@@ -92,12 +92,9 @@ CREATE PROCEDURE sp_addApplicationHomeSubCluster(
IN applicationContext_IN BLOB,
OUT storedHomeSubCluster_OUT varchar(256), OUT rowCount_OUT int)
BEGIN
- INSERT INTO applicationsHomeSubCluster
- (applicationId, homeSubCluster, createTime, applicationContext)
- (SELECT applicationId_IN, homeSubCluster_IN, NOW(), applicationContext_IN
- FROM applicationsHomeSubCluster
- WHERE applicationId = applicationId_IN
- HAVING COUNT(*) = 0 );
+ INSERT IGNORE INTO applicationsHomeSubCluster(
Review Comment:
What does IGNORE do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#issuecomment-1562051228
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 22s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 43s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 0s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 91m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5673 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6e54ede4dbf3 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 73cd29d5163418c8905e9328d2df90924a3f6d75 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/2/testReport/ |
| Max. process+thread count | 700 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#issuecomment-1567368755
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 23s | | trunk passed |
| +1 :green_heart: | compile | 6m 55s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 6m 24s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 57s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 11m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 35s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 54s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 56s | | the patch passed |
| +1 :green_heart: | compile | 6m 21s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 6m 21s | | the patch passed |
| +1 :green_heart: | compile | 6m 22s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 6m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 39s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 8s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 11m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 55s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 231m 51s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 3m 31s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not generate ASF License warnings. |
| | | 394m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5673 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 75a553a2e66b 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68bd56d27a876afa1d3d99e3beed53c946908fe2 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/5/testReport/ |
| Max. process+thread count | 2703 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#issuecomment-1562273496
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 31s | | trunk passed |
| +1 :green_heart: | compile | 6m 59s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 6m 22s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 56s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 11m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 34s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 53s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 57s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 0s | | the patch passed |
| +1 :green_heart: | compile | 6m 21s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 6m 21s | | the patch passed |
| +1 :green_heart: | compile | 6m 18s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 6m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 6s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 11m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 232m 18s | [/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt) | hadoop-yarn in the patch passed. |
| -1 :x: | unit | 3m 34s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt) | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not generate ASF License warnings. |
| | | 395m 28s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| | hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| | hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5673 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux dda9905d34af 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8df96ba0634fda64dba0164f93aa10a11ffd6149 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/3/testReport/ |
| Max. process+thread count | 2628 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5673: YARN-8337. [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5673:
URL: https://github.com/apache/hadoop/pull/5673#issuecomment-1563524594
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 20m 17s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 56s | | trunk passed |
| +1 :green_heart: | compile | 7m 32s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 6m 47s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 58s | | trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 13s | | trunk passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 12m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 10s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 3s | | the patch passed |
| +1 :green_heart: | compile | 6m 57s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 6m 57s | | the patch passed |
| +1 :green_heart: | compile | 6m 43s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 6m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 42s | | the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 6s | | the patch passed with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 12m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 12s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 236m 21s | | hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 3m 18s | | hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not generate ASF License warnings. |
| | | 416m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5673 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 226cabfa6723 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 347076e3192a4eafd355f89a51cdc8e007e07e77 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/4/testReport/ |
| Max. process+thread count | 1901 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5673/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org