You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/20 15:02:58 UTC
[GitHub] [airflow] ANiteckiP opened a new pull request #7473: [AIRFLOW-6852]
Force reschedule mode for sensors when running on SequentialExecutor
ANiteckiP opened a new pull request #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473
Force reschedule mode for sensors when running on SequentialExecutor to avoid deadlocks.
---
Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
<sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7473:
[AIRFLOW-6852] Force reschedule mode for sensors when running on
SequentialExecutor
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#discussion_r383259139
##########
File path: airflow/models/taskinstance.py
##########
@@ -909,7 +909,7 @@ def _run_raw_task(
# with single process executor, thus we change the mode to`reschedule`
# to allow parallel task being scheduled and executed
if isinstance(task_copy, BaseSensorOperator) and \
- conf.get('core', 'executor') == "DebugExecutor":
+ conf.get('core', 'executor') in ["DebugExecutor", "SequentialExecutor"]:
self.log.warning("DebugExecutor changes sensor mode to 'reschedule'.")
Review comment:
This log needs updating anyway :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7473: [AIRFLOW-6852]
Force reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-589942661
I am not convinced we should do this change. The `SequentialExecutor` can be used by users in production (I hope I'm wrong!) and this may cause some problems. @potiuk @ash any thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] stale[bot] closed pull request #7473: [AIRFLOW-6852]
Force reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
stale[bot] closed pull request #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ANiteckiP commented on issue #7473: [AIRFLOW-6852] Force
reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
ANiteckiP commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-589162333
@nuclearpinguin This change would effectively prohibit using poke-only sensors together with SequentialExecutor - SequentialExecutor would force the reschedule mode and the code you linked would cause DAG parsing to fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #7473: [AIRFLOW-6852] Force
reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-611563110
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #7473: [AIRFLOW-6852] Force
reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-590315367
The SequentialExecutor _stops heartbeating_ when running a task, so this is very very unlikely to be used in production, as you'd constantly get "The scheduler is not running" notices in the UI.
I'm not even sure we should have Sequential executor anymore. LocalExecutor with a single process would be much much better.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7473: [AIRFLOW-6852] Force
reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-590327691
> The SequentialExecutor _stops heartbeating_ when running a task, so this is very very unlikely to be used in production, as you'd constantly get "The scheduler is not running" notices in the UI.
>
> I'm not even sure we should have Sequential executor anymore. LocalExecutor with a single process would be much much better.
SequentialExecutor is currently the only one allowed for SQLIte
```
def _validate(self):
if (
self.get("core", "executor") not in ('DebugExecutor', 'SequentialExecutor') and
"sqlite" in self.get('core', 'sql_alchemy_conn')):
raise AirflowConfigException(
"error: cannot use sqlite with the {}".format(
self.get('core', 'executor')))
```
But indeed, if we change it to enforce the LocalExecutor and count ==1 this might work and we can get rid of the Sequential executor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #7473: [AIRFLOW-6852]
Force reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-589096888
Can you check how this will work with the following change?
https://github.com/apache/airflow/pull/7197
Stateful sensors should not be used in reschedule mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ANiteckiP edited a comment on issue #7473: [AIRFLOW-6852]
Force reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
ANiteckiP edited a comment on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-589162333
@nuclearpinguin This change would effectively prohibit using poke-only sensors together with SequentialExecutor - SequentialExecutor would force the reschedule mode and the code in PR you linked would cause DAG parsing to fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7473: [AIRFLOW-6852] Force
reschedule mode for sensors when running on SequentialExecutor
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7473: [AIRFLOW-6852] Force reschedule mode for sensors when running on SequentialExecutor
URL: https://github.com/apache/airflow/pull/7473#issuecomment-590327959
And I've heard anecdotal evidence of people using sqlite in production. Unluckily we haven't asked for the database in the survey :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services