You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/04 12:41:38 UTC
[GitHub] [airflow] Khrol opened a new pull request #7622: [AIRFLOW-6983]
Configure connection pool disabling in CLI
Khrol opened a new pull request #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI
URL: https://github.com/apache/airflow/pull/7622
Tasks are executed in CLI mode when connection pool to the database is disabled.
`settings.configure_orm(disable_connection_pool=True)`
While one task is run, multiple DB communications are happening while a separate connection is allocated for each of them. It results in DB failures.
Default behavior might be ok but it worth making it configurable.
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Khrol commented on issue #7622: [AIRFLOW-6983] Configure
connection pool disabling in CLI
Posted by GitBox <gi...@apache.org>.
Khrol commented on issue #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI
URL: https://github.com/apache/airflow/pull/7622#issuecomment-598117215
I've pushed a wider explanation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Khrol commented on issue #7622: [AIRFLOW-6983] Configure
connection pool disabling in CLI
Posted by GitBox <gi...@apache.org>.
Khrol commented on issue #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI
URL: https://github.com/apache/airflow/pull/7622#issuecomment-608408394
> Rather than disabling the pooling, I'd like to try using a different Pool implementation all together.
Sounds reasonable. I'll try something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7622: [AIRFLOW-6983]
Configure connection pool disabling in CLI
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI
URL: https://github.com/apache/airflow/pull/7622#issuecomment-598118289
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=h1) Report
> Merging [#7622](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6cc9d817fa98204887f81c962e81c159bf57aeb2&el=desc) will **decrease** coverage by `0.32%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7622 +/- ##
==========================================
- Coverage 86.98% 86.65% -0.33%
==========================================
Files 904 904
Lines 43728 43728
==========================================
- Hits 38037 37894 -143
- Misses 5691 5834 +143
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `72.94% <100.00%> (ø)` | |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.33% <0.00%> (-0.39%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.53% <0.00%> (-0.15%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=footer). Last update [6cc9d81...2b614f8](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7622: [AIRFLOW-6983] Configure
connection pool disabling in CLI
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI
URL: https://github.com/apache/airflow/pull/7622#issuecomment-608407827
I want to take a look at solving the underlying problem in a different way:
- Rather than disabling the pooling, I'd like to try using a different Pool implementation all together.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7622:
[AIRFLOW-6983] Configure connection pool disabling in CLI
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI
URL: https://github.com/apache/airflow/pull/7622#issuecomment-598118289
# [Codecov](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=h1) Report
> Merging [#7622](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6cc9d817fa98204887f81c962e81c159bf57aeb2&el=desc) will **decrease** coverage by `0.32%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #7622 +/- ##
==========================================
- Coverage 86.98% 86.65% -0.33%
==========================================
Files 904 904
Lines 43728 43728
==========================================
- Hits 38037 37894 -143
- Misses 5691 5834 +143
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `72.94% <100.00%> (ø)` | |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0.00%> (-45.66%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.33% <0.00%> (-0.39%)` | :arrow_down: |
| [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7622/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `90.53% <0.00%> (-0.15%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=footer). Last update [6cc9d81...2b614f8](https://codecov.io/gh/apache/airflow/pull/7622?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7622: [AIRFLOW-6983] Configure
connection pool disabling in CLI
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7622: [AIRFLOW-6983] Configure connection pool disabling in CLI
URL: https://github.com/apache/airflow/pull/7622#issuecomment-608496536
https://docs.sqlalchemy.org/en/13/core/pooling.html?highlight=pool#sqlalchemy.pool.SingletonThreadPool was the one I was thinking of
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services