You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/01/01 09:50:22 UTC
[GitHub] [airflow] potiuk opened a new pull request #6988: [AIRFLOW-2516]
Fix mysql deadlocks
potiuk opened a new pull request #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988
Deadlocks were occuring in mysql when task_instance was modified
by two queries at the same time. One query used state as selection
criteria and updated it in the same query where second query just
updated the state for the same table. The first query locked state
index first and primary index afterwards, the second query locked
primary index first and state afterwards - leading to deadlocks.
This change splits the first query into two independent ones.
First query makes select FOR UPDATE and selects all the task
instances to act on (this will lock primary index only)
and second updates all affected task instances.
Note that performance impact for that is neglectable because this
query is only run once every scheduler loop and the second part
of it (looping through task instances) will only happen in case
there are some manually modified DagRun states - so it is only
run to correct some wrong states of DagRun. This should happen
very infrequently.
- [x] Description above provides context of the change
- [x] Commit message contains [\[AIRFLOW-XXXX\]](https://issues.apache.org/jira/browse/AIRFLOW-XXXX) or `[AIRFLOW-XXXX]` for document-only changes
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #6988: [AIRFLOW-2516]
Fix mysql deadlocks
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-570069391
Not yet @kaxil -> that's why it's still Draft . But I provided the users with patched versions of jobs.py/scheduled_job.py for 1.9, 1.10.6, 1.10.3 and asked them to test it. See https://issues.apache.org/jira/browse/AIRFLOW-2516?focusedCommentId=17006364&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17006364 and https://issues.apache.org/jira/browse/AIRFLOW-4498?focusedCommentId=17006370&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17006370
They tested it before for 1.9 and 1.10.6 so I expect they will come back after NewYear's holidays.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6988: [AIRFLOW-2516] Fix
mysql deadlocks
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-570045289
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=h1) Report
> Merging [#6988](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6d2b4b721f823310b52fb2ea20fd3d23261a109d?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6988 +/- ##
==========================================
- Coverage 84.85% 84.56% -0.29%
==========================================
Files 679 679
Lines 38536 38539 +3
==========================================
- Hits 32698 32591 -107
- Misses 5838 5948 +110
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.32% <100%> (+0.04%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=footer). Last update [6d2b4b7...8084255](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6988: [AIRFLOW-2516] Fix mysql
deadlocks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-573744078
@kaxil @ashb @mik-laj @nuclearpinguin -> It's 10 days without the deadlock for our customer https://issues.apache.org/jira/browse/AIRFLOW-2516 so it looks like the problem is solved. Please approve and i will merge it and cherry-pick to 1.10.8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #6988: [AIRFLOW-2516] Fix
mysql deadlocks
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6988: [AIRFLOW-2516] Fix mysql
deadlocks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-570069391
Not yet @kaxil -> that's why it's still Draft . But I provided the users with patched versions of jobs.py/scheduled_jobs.py for 1.9, 1.10.6, 1.10.3 and asked them to test it. See https://issues.apache.org/jira/browse/AIRFLOW-2516?focusedCommentId=17006364&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17006364 and https://issues.apache.org/jira/browse/AIRFLOW-4498?focusedCommentId=17006370&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17006370
They tested it before for 1.9 and 1.10.6 so I expect they will come back after NewYear's holidays.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6988: [AIRFLOW-2516] Fix mysql
deadlocks
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-570069471
:crossed_fingers:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6988: [AIRFLOW-2516] Fix mysql
deadlocks
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-570069454
And I keep my fingers crossed that it's going to help 🤞
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6988:
[AIRFLOW-2516] Fix mysql deadlocks
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-570045289
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@f4d3e5e`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6988 +/- ##
=========================================
Coverage ? 85.03%
=========================================
Files ? 707
Lines ? 39361
Branches ? 0
=========================================
Hits ? 33472
Misses ? 5889
Partials ? 0
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <100%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=footer). Last update [f4d3e5e...706b8cc](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6988:
[AIRFLOW-2516] Fix mysql deadlocks
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6988: [AIRFLOW-2516] Fix mysql deadlocks
URL: https://github.com/apache/airflow/pull/6988#issuecomment-570045289
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@f4d3e5e`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6988 +/- ##
=========================================
Coverage ? 85.03%
=========================================
Files ? 707
Lines ? 39361
Branches ? 0
=========================================
Hits ? 33472
Misses ? 5889
Partials ? 0
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6988/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <100%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=footer). Last update [f4d3e5e...706b8cc](https://codecov.io/gh/apache/airflow/pull/6988?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services