You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/12/20 18:57:11 UTC
[GitHub] [airflow] lucafuji opened a new pull request #6870: [AIRFLOW-578]
Fix check return code
lucafuji opened a new pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870
### Jira
- [x] My PR addresses the following [AIRFLOW-578](https://issues.apache.org/jira/browse/AIRFLOW-578) issues and references them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-578
### Description
Check the return code of the spawned process of LocalTaskJob.
### Tests
- [x] My PR adds the following unit tests
tests.test_jobs:LocalTaskJobTest.test_localtaskjob_invalid_return_code
### Code Quality
- [x] Passes `flake8`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372536473
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
You haven't addressed or responded to this comment. Just marking it as resolved without replying is not cool.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-575816844
I just added python operator calls os._exit(1). My code does catch it and change the status to FAILED. without my code. The failure is not caught.
**However**, we probably don't want use to call os._exit() in Python operator. This is because:
Python operator runs in the same process with task instance. Therefore when calling os._exit, some post execution code in task instance is not executed. This will leave task instance in RUNNING state forever.
Here is the code of pose execution of task_instance:
https://github.com/apache/airflow/blob/master/airflow/models/taskinstance.py#L965
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **increase** coverage by `0.88%`.
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
+ Coverage 85.52% 86.4% +0.88%
=========================================
Files 758 882 +124
Lines 39932 41541 +1609
=========================================
+ Hits 34150 35892 +1742
+ Misses 5782 5649 -133
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.8% <100%> (+1.06%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.15% <100%> (-1.05%)` | :arrow_down: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [...rflow/providers/apache/cassandra/sensors/record.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9zZW5zb3JzL3JlY29yZC5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [...irflow/providers/apache/cassandra/sensors/table.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9zZW5zb3JzL3RhYmxlLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: |
| [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-93.97%)` | :arrow_down: |
| [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `0% <0%> (-91.94%)` | :arrow_down: |
| ... and [979 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...6ea63bd](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.05% 84.78% -0.27%
==========================================
Files 680 680
Lines 38753 38762 +9
==========================================
- Hits 32960 32863 -97
- Misses 5793 5899 +106
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372871648
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
i.e. if we're checking the exit code shouldn't we have noticed that the task state hasn't updated and mark it as failed as a result? (with a log message so the reason for the failure shows up in the task logs)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.69%`.
> The diff coverage is `80.6%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 85.12% +0.69%
==========================================
Files 679 710 +31
Lines 38514 39488 +974
==========================================
+ Hits 32517 33615 +1098
+ Misses 5997 5873 -124
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `100% <ø> (+22.39%)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5) | `100% <ø> (+17.5%)` | :arrow_up: |
| [airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=) | `100% <ø> (+7.31%)` | :arrow_up: |
| ... and [343 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...3f8ec6f](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r366037407
##########
File path: airflow/utils/state.py
##########
@@ -122,3 +122,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
Review comment:
UP_FOR_RESCHEDULE is actually added per your comment 24 days ago.
https://github.com/apache/airflow/pull/6870/files/ffaf10177cae9d8a3b99d10da33d685f79d3a64e#discussion_r360559835
So please let me know what's your thought so I can make a final change.
IMO, UP_FOR_RESCHEDULE should not be treated as a unsuccessful state as you describe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372648584
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
TBH JobState isn't used anyway (so long as it's not still running).
It's open to interpretation when a job should be marked as failed. If the task ran to completion then the job did what it was told to. Is that a failure? What about when the task is in "up_for_retry" -- what state should the Job be in?
My first though is that the job should be marked as success if the task is in any state other than "running" at the end, and only failure if the Task didn't update it's state.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r381447186
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
do you mean I need to check the state of the job in the unittest? Just added one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372698153
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
^^ @ashb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.35%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.78% +0.35%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32863 +346
+ Misses 5997 5899 -98
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372871218
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
Ah okay.
But with your new change shouldn't the on_failure have set the TI to failed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372637657
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
Review comment:
removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.55% +0.13%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32777 +260
+ Misses 5997 5985 -12
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372534650
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
Review comment:
Not the only case -- this could happen if there was some startup problem too, or if the task died _hard_ (segfault)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372647002
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
Back to your question, we need to set the job state as FAILED as well. This is handled in BaseJob.run by throwing an exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-609911766
@lucafuji that is the same Jira we've been talking about, just migrated to GitHub issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-578] Fix
check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-571248436
@ashb Would you mind take another look? thanks much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dimberman commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
dimberman commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-605654353
@ashb can we get another lookover on this one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji edited a comment on issue #6870: [AIRFLOW-0578]
Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-575732495
> Im not sure your test as it stands covers the new code - the BashOperator handles return code for that command already.
>
> I think what you need to test is a Python operator that calls `os._exit(1)` - but the real way of testing would be to run the test without calling your new on_failure, see it fail, then run it again with it enabled
On https://github.com/apache/airflow/blob/master/airflow/operators/bash_operator.py#L137, bash operator will throw an exception if the return code is non zero.
However, on https://github.com/apache/airflow/blob/master/airflow/task/task_runner/standard_task_runner.py#L85, it will catch this exception and return a return code of 1. This is what this PR trying to solve. This return code is not handled properly by local_task_job. And my test case did cover this.
But yep, I can still add another task with Python Operator to see whether it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.05% 84.78% -0.27%
==========================================
Files 680 680
Lines 38753 38762 +9
==========================================
- Hits 32960 32863 -97
- Misses 5793 5899 +106
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-579877067
> bash operator will throw an exception if the return code is non zero.
But in this case the BashOperator has already marked the task as failed, no?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r367626457
##########
File path: airflow/utils/state.py
##########
@@ -122,3 +122,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
Review comment:
Past me was right, and this is all good here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r381445504
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,14 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ self.task_instance.refresh_from_db()
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if (not self.terminating) or self.task_instance.state == State.FAILED:
Review comment:
1. "self.terminating" means job been killed externally but not "state="failed". It's set in heartbeat_callback:L137. Basically it means whenever "ti.state != State.RUNNING", it's terminating.
There are two cases this will happen
a. explicitly calling terminate of a StandardTaskRunner, then the return code is -9 and the task_instance.state is not failed. In such case, we should not treat non zero exit code as failure.
b. the task instance is explicitly set as failed, in this case, we should treat non zero exit code as failure.
2. As mentioned above, "job failure" is handled in base_job.py:run:L230. If exception is not thrown here, job state will not be marked as failure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372639124
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
The task state is usually set in another process, therefore the final state is not visible to the local task job. We need to make sure we get the correct state before moving forward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.55% +0.13%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32777 +260
+ Misses 5997 5985 -12
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #6870: [AIRFLOW-578] Fix check
return code
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-568583467
There should be no changes to base job for this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r373143875
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
> i.e. if we're checking the exit code shouldn't we have noticed that the task state hasn't updated and mark it as failed as a result? (with a log message so the reason for the failure shows up in the task logs)
It's not a failure, the exit code is zero. Maybe we can add another check that
if we get a return code and the state is not in State.finished(), we set the task state accordingly. I can make that change as well but it seems its out of the scope of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `93.75%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.52% 85.24% -0.28%
==========================================
Files 758 758
Lines 39932 39947 +15
==========================================
- Hits 34150 34052 -98
- Misses 5782 5895 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.8% <100%> (+1.06%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.72% <80%> (-0.48%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...d79295b](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r366310378
##########
File path: airflow/utils/state.py
##########
@@ -122,3 +122,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
Review comment:
:man_facepalming:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 85.52% 85.65% +0.13%
==========================================
Files 758 862 +104
Lines 39932 40499 +567
==========================================
+ Hits 34150 34689 +539
- Misses 5782 5810 +28
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.8% <100%> (+1.06%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.72% <80%> (-0.48%)` | :arrow_down: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: |
| [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-93.97%)` | :arrow_down: |
| [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `0% <0%> (-91.94%)` | :arrow_down: |
| [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| ... and [535 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...0977fe8](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.05% 84.78% -0.27%
==========================================
Files 680 680
Lines 38753 38762 +9
==========================================
- Hits 32960 32863 -97
- Misses 5793 5899 +106
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji edited a comment on issue #6870: [AIRFLOW-0578]
Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-579966341
> But in this case the BashOperator has already marked the task as failed, no?
There will be states for two different entities.
One for the task and one for the job that running that task
The task is marked as failed but the job is not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372663638
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
Oh, yep, I mean os._exit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-578] [WIP] Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-578] [WIP] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r360589111
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
This PR is actually from previously a closed PR, in previous PR one of the reviewer recommend me to add an on_failure callback in base job.
Here is the comment
https://github.com/apache/airflow/pull/1122#issuecomment-193252883.
I'm fine with either way, please let me know you thoughts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.35%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.78% +0.35%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32863 +346
+ Misses 5997 5899 -98
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **decrease** coverage by `0.11%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.52% 85.4% -0.12%
=========================================
Files 758 862 +104
Lines 39932 40499 +567
=========================================
+ Hits 34150 34590 +440
- Misses 5782 5909 +127
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.8% <100%> (+1.06%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.72% <80%> (-0.48%)` | :arrow_down: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: |
| [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-93.97%)` | :arrow_down: |
| [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `0% <0%> (-91.94%)` | :arrow_down: |
| [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: |
| ... and [538 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...0977fe8](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.55% +0.13%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32777 +260
+ Misses 5997 5985 -12
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6870: [AIRFLOW-578] Fix
check return code
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b7c11e449e528f2f8b6f030160d3eefd643e6c2f?src=pr&el=desc) will **decrease** coverage by `0.71%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 84.56% 83.85% -0.72%
==========================================
Files 680 679 -1
Lines 38454 38514 +60
==========================================
- Hits 32520 32296 -224
- Misses 5934 6218 +284
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.02% <87.5%> (-0.31%)` | :arrow_down: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `49.66% <0%> (-38.93%)` | :arrow_down: |
| ... and [36 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [b7c11e4...4b7b977](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-611241580
> @lucafuji that is the same Jira we've been talking about, just migrated to GitHub issue.
Ok, but the point here is JobState cannot be ignored. We need to set the correct JobState so that k8s operator knows whether it's failed or not
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372669508
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
Review comment:
Not sure what I got you,
in those cases I think we should mark them as failure as well? Anyway it means we have some problems in running the task
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r381445504
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,14 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ self.task_instance.refresh_from_db()
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if (not self.terminating) or self.task_instance.state == State.FAILED:
Review comment:
1. "self.terminating" means job been killed externally but not "state="failed". It's set in heartbeat_callback:L137. Basically it means whenever "ti.state != State.RUNNING", it's terminating.
There are two cases this will happen
a. explicitly calling terminate of a StandardTaskRunner, then the return code is -9 and the task_instance.state is not failed. In such case, we should not treat non zero exit code as failure.
b. the task instance is explicitly set as failed, in this case, we should treat zero exit code as failure.
2. As mentioned above, "job failure" is handled in base_job.py:run:L230. If exception is not thrown here, job state will not be marked as failure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-578] Fix
check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569113796
> There should be no changes to base job for this PR
Got it, remove all changes to base jobs. PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372639124
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
The task state is usually set in another process, therefore the final state is not visible to the local task job. We need to make sure we get the correct state before moving forward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `93.75%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.52% 85.24% -0.28%
==========================================
Files 758 758
Lines 39932 39947 +15
==========================================
- Hits 34150 34052 -98
- Misses 5782 5895 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.8% <100%> (+1.06%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.72% <80%> (-0.48%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...d79295b](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372640054
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
good point. I need to call refresh from db before proceed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372673255
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
> TBH JobState isn't used anyway (so long as it's not still running).
>
> It's open to interpretation when a job should be marked as failed. If the task ran to completion then the job did what it was told to. Is that a failure? What about when the task is in "up_for_retry" -- what state should the Job be in?
>
> My first though is that the job should be marked as success if the task is in any state other than "running" at the end, and only failure if the Task didn't update it's state.
Hmm, I see your point but I think we should better to be consistent about interpreting the meaning of Job.state across or job types?
e.g, for BackfillJob, if one of the underlying tasks fail, it will be marked as failure as well
on backfill_job.py:L792
"""
err = self._collect_errors(ti_status=ti_status, session=session)
if err:
raise AirflowException(err)
"""
So I think it might be better to have job state reflects the state of underlying task instance
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji edited a comment on issue #6870: [AIRFLOW-0578]
Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-575732495
> Im not sure your test as it stands covers the new code - the BashOperator handles return code for that command already.
>
> I think what you need to test is a Python operator that calls `os._exit(1)` - but the real way of testing would be to run the test without calling your new on_failure, see it fail, then run it again with it enabled
On https://github.com/apache/airflow/blob/master/airflow/operators/bash_operator.py#L137, bash operator will throw an exception if the return code is non zero.
However, on https://github.com/apache/airflow/blob/master/airflow/task/task_runner/standard_task_runner.py#L85, it will catch this exception and return a return code of 1. This is what this PR trying to solve. This return code is not handled properly by local_task_job. And my test case did cover this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-578] [WIP] Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-578] [WIP] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r360589111
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
This PR is actually from previously a closed PR, in previous PR one of the reviewer @mistercrunch recommend me to add an on_failure callback in base job.
Here is the comment
https://github.com/apache/airflow/pull/1122#issuecomment-193252883.
I'm fine with either way, please let me know you thoughts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.05% 84.78% -0.27%
==========================================
Files 680 680
Lines 38753 38762 +9
==========================================
- Hits 32960 32863 -97
- Misses 5793 5899 +106
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.35%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.78% +0.35%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32863 +346
+ Misses 5997 5899 -98
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `93.75%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.52% 85.24% -0.28%
==========================================
Files 758 758
Lines 39932 39946 +14
==========================================
- Hits 34150 34051 -99
- Misses 5782 5895 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.69% <100%> (+0.95%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.72% <80%> (-0.48%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...2f88aa5](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r366037407
##########
File path: airflow/utils/state.py
##########
@@ -122,3 +122,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
Review comment:
UP_FOR_RESCHEDULE is actually added per your comment 24 days ago.
"Up for reschedule too."
So please let me know what's your thought so I can make a final change.
IMO, UP_FOR_RESCHEDULE should not be treated as a unsuccessful state as you describe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.55% +0.13%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32777 +260
+ Misses 5997 5985 -12
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r365983674
##########
File path: airflow/utils/state.py
##########
@@ -122,3 +122,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
Review comment:
Not sure reschedule should be in this list -- it's used by sensors to retry but not tie up a slot. But this is basically still "running" otherwise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-578] [WIP]
Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-578] [WIP] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-568104204
> I agreev with the spirit of the change but not the specifics.
>
> I don't think this change should touch base job at all. The issue with not checking return codes is restricted to Local Job
This PR is actually from previously a closed PR, in previous PR one of the reviewer @mistercrunch recommend me to add an on_failure callback in base job.
Here is the comment
#1122 (comment).
I'm fine with either way, please let me know you thoughts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-575284775
@ashb ^^
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-575732495
> Im not sure your test as it stands covers the new code - the BashOperator handles return code for that command already.
>
> I think what you need to test is a Python operator that calls `os._exit(1)` - but the real way of testing would be to run the test without calling your new on_failure, see it fail, then run it again with it enabled
On https://github.com/apache/airflow/blob/master/airflow/operators/bash_operator.py#L137, bash operator will throw an exception if the return code is non zero.
However, on https://github.com/apache/airflow/blob/master/airflow/task/task_runner/standard_task_runner.py#L85, it will catch this exception and return a return code of 1. This is what this PR trying to solve. This return code is not handled properly by local_task_job.
But yep, I can still add another task with Python Operator to see whether it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r366082819
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +83,46 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_localtaskjob_invalid_return_code(self):
+ dag = DAG(
+ 'test_localtaskjob_invalid_return_code',
+ start_date=DEFAULT_DATE,
+ default_args={'owner': 'airflow'})
+
+ task = BashOperator(
+ task_id='test_bash',
+ bash_command='exit 1',
+ dag=dag,
+ owner='airflow')
+ session = settings.Session()
+
+ dag.clear()
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ ti = dr.get_task_instance(task_id=task.task_id, session=session)
+ ti.state = State.RUNNING
+ ti.hostname = get_hostname()
+ ti.pid = 1
+ session.commit()
+
+ ti_run = TI(task=task, execution_date=DEFAULT_DATE)
+ job1 = LocalTaskJob(task_instance=ti_run,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with patch.object(BaseTaskRunner, 'start',
+ return_value=None) as mock_method:
+ job1.run()
+ mock_method.assert_not_called()
Review comment:
sorry, it's not..Just rewrite the tests to run the new code. Previously I wrote in this way is because it's the local_task_job which calling the actual task's execution but now it's the task_runner.
Thanks for catching this! Nice finding.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-578] [WIP] Fix check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-578] [WIP] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r360559544
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
Throwing an exception here is heavier weight than we need. Just call on_fail here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r366495363
##########
File path: airflow/utils/state.py
##########
@@ -122,3 +122,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
Review comment:
@ashb Maybe it's because it's already resolved. Here is a screenshot
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372669508
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
Review comment:
Not sure what I got you,
in those cases I think we should mark them as failure as well? Anyway they means we have some problems in running the task then we should raise an exception here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-578] [WIP] Fix check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-578] [WIP] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r360589111
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
This PR is actually from previously a closed PR, in previous PR one of the reviewer @mistercrunch recommend me to add an on_failure callback in base job.
Here is the comment
https://github.com/apache/airflow/pull/1122#issuecomment-193252883.
I'm fine with either way, please let me know you thoughts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-578] [WIP] Fix check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-578] [WIP] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r360559835
##########
File path: airflow/utils/state.py
##########
@@ -116,3 +116,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY]
Review comment:
Up for reschedule too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-579966341
> But in this case the BashOperator has already marked the task as failed, no?
There will be two states. One for the task and one for the job that running that task,
The task is marked as failed but the job is not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r376487438
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,14 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ self.task_instance.refresh_from_db()
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if (not self.terminating) or self.task_instance.state == State.FAILED:
Review comment:
How does `state="failed"` relate to job been killed externally?
I'm also not that happy about having very similar conditions here and inside on_failure. And again: Why throw an exception only to have it caught later, why not just call `self.on_failure`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.35%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.78% +0.35%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32863 +346
+ Misses 5997 5899 -98
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372647306
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
Ah.
Ignoring `os._exit()` for now, `sys.exit` could be handled in the python operator by catching SystemExit exception. (That would be a separate PR though)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.35%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.78% +0.35%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32863 +346
+ Misses 5997 5899 -98
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.35%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.78% +0.35%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32863 +346
+ Misses 5997 5899 -98
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r366341721
##########
File path: airflow/utils/state.py
##########
@@ -122,3 +122,10 @@ def unfinished(cls):
cls.UP_FOR_RETRY,
cls.UP_FOR_RESCHEDULE
]
+
+ @classmethod
+ def unsuccessful(cls):
+ """
+ A list of states indicating that a task completed unsuccessfully.
+ """
+ return [cls.FAILED, cls.UP_FOR_RETRY, cls.UP_FOR_RESCHEDULE]
Review comment:
@lucianoviola That link doesn't take me to any discussion (anymore)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **decrease** coverage by `0.27%`.
> The diff coverage is `93.75%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.52% 85.24% -0.28%
==========================================
Files 758 758
Lines 39932 39946 +14
==========================================
- Hits 34150 34051 -99
- Misses 5782 5895 +113
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.69% <100%> (+0.95%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.72% <80%> (-0.48%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...2f88aa5](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372535482
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
Do we need to do `self.task_instance.refresh_from_db()` before this check? Otherwise it will just look at the last value in memory, ignoring anything that the subprocess may have set, I think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r376488504
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
You aren't checking the state of the job anywhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3730c24c41470cd331c5109539ee2fa0c9f4e74a?src=pr&el=desc) will **increase** coverage by `0.36%`.
> The diff coverage is `94.11%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 85.52% 85.88% +0.36%
==========================================
Files 758 862 +104
Lines 39932 40499 +567
==========================================
+ Hits 34150 34782 +632
+ Misses 5782 5717 -65
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90.8% <100%> (+1.06%)` | :arrow_up: |
| [airflow/jobs/base\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2Jhc2Vfam9iLnB5) | `91.72% <80%> (-0.48%)` | :arrow_down: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/contrib/operators/snowflake\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbm93Zmxha2Vfb3BlcmF0b3IucHk=) | `0% <0%> (-95.84%)` | :arrow_down: |
| [airflow/operators/s3\_to\_hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvczNfdG9faGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-93.97%)` | :arrow_down: |
| [airflow/contrib/hooks/grpc\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2dycGNfaG9vay5weQ==) | `0% <0%> (-91.94%)` | :arrow_down: |
| [airflow/contrib/hooks/snowflake\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Nub3dmbGFrZV9ob29rLnB5) | `0% <0%> (-81.14%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| ... and [533 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [3730c24...0977fe8](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `81.54%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.55% +0.13%
==========================================
Files 679 680 +1
Lines 38514 38762 +248
==========================================
+ Hits 32517 32777 +260
+ Misses 5997 5985 -12
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | :arrow_up: |
| [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: |
| ... and [231 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372645710
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,10 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ if return_code != 0:
+ msg = ("LocalTaskJob process exited with non zero "
+ "status {}".format(return_code))
+ raise AirflowException(msg)
Review comment:
No I did not resolve it... I can see the conversation is still here. I think it's just github mark it as outdated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4b7b9770890d3acc7bfd93c5fae577b4f63427fe?src=pr&el=desc) will **increase** coverage by `0.23%`.
> The diff coverage is `80.6%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
+ Coverage 84.42% 84.66% +0.23%
==========================================
Files 679 710 +31
Lines 38514 39488 +974
==========================================
+ Hits 32517 33432 +915
- Misses 5997 6056 +59
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/contrib/hooks/qubole\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3F1Ym9sZV9ob29rLnB5) | `52.67% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: |
| [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: |
| [airflow/hooks/hive\_hooks.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9oaXZlX2hvb2tzLnB5) | `100% <ø> (+22.39%)` | :arrow_up: |
| [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: |
| [airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5) | `100% <ø> (+17.5%)` | :arrow_up: |
| [airflow/contrib/operators/spark\_submit\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zcGFya19zdWJtaXRfb3BlcmF0b3IucHk=) | `100% <ø> (+7.31%)` | :arrow_up: |
| ... and [352 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [4b7b977...3f8ec6f](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b7c11e449e528f2f8b6f030160d3eefd643e6c2f?src=pr&el=desc) will **decrease** coverage by `0.13%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 84.56% 84.42% -0.14%
==========================================
Files 680 679 -1
Lines 38454 38514 +60
==========================================
- Hits 32520 32517 -3
- Misses 5934 5997 +63
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.02% <87.5%> (-0.31%)` | :arrow_down: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `88.59% <0%> (ø)` | :arrow_up: |
| ... and [36 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [b7c11e4...4b7b977](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-575817766
Besides that, I kind of remember why Max recommend me to change base_job.
In airflow, it's base_job.run call base_job.execute and each sub class implements its own _execute function.
And it's _execute's job to execute the actual command, but state management should be done in base_job.run, after _execute is done.
The includes managing the state of the job and task instance
We should not change the state in _execute.
Therefore it might be better to implement a on_failure callback to be called after _execute. We can make default behaviour just do nothing and only implement specific logic in local_task_job.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372608341
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
Additionally though: why do we care about calling on-failure/raising when the task_instance is already set to failure.
You are overloading two subtly different concepts here:
1. The task executed to completion, but it was marked as a failure.
2. The task process died without managing to tidy up.
Case 1 is already handled and the TI ends up in a "terminal" state successfully. It's Case 2 is the only place we need to change. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
=========================================
- Coverage 85.05% 84.55% -0.5%
=========================================
Files 680 680
Lines 38753 38762 +9
=========================================
- Hits 32960 32777 -183
- Misses 5793 5985 +192
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `90% <0%> (-6.67%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r373142387
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
> Ah okay.
>
> But with your new change shouldn't the on_failure have set the TI to failed?
In my change on_failure will only be triggered when the return code is non zero. Here we called os._exit(0)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b7c11e449e528f2f8b6f030160d3eefd643e6c2f?src=pr&el=desc) will **decrease** coverage by `0.71%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 84.56% 83.85% -0.72%
==========================================
Files 680 679 -1
Lines 38454 38514 +60
==========================================
- Hits 32520 32296 -224
- Misses 5934 6218 +284
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.02% <87.5%> (-0.31%)` | :arrow_down: |
| [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `49.66% <0%> (-38.93%)` | :arrow_down: |
| ... and [36 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [b7c11e4...4b7b977](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372642576
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
+ # there is one case we should not treat non zero return
+ # code as failed: the job has been killed externally.
+ if return_code != 0 and ((not self.terminating) or
+ self.task_instance.state ==
Review comment:
> Additionally though: why do we care about calling on-failure/raising when the task_instance is already set to failure.
>
> You are overloading two subtly different concepts here:
>
> 1. The task executed to completion, but it was marked as a failure.
> 2. The task process died without managing to tidy up.
>
> Case 1 is already handled and the TI ends up in a "terminal" state successfully. It's Case 2 is the only place we need to change. What do you think?
In theory there is only one case, the process exited with non zero exit code. We need to set the correct states for task and job.
1. The task state may or may not be set correctly, that's what on_failure is handling.
2. The job state is definitely not set as FAILED, so that's why raise another exception to let BaseJob to handle this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6870: [AIRFLOW-578]
Fix check return code
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-569120114
# [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=h1) Report
> Merging [#6870](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/033e8638f71178d40092bf16eb7c1feede4b7b89?src=pr&el=desc) will **decrease** coverage by `0.26%`.
> The diff coverage is `90%`.
[](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6870 +/- ##
==========================================
- Coverage 85.05% 84.78% -0.27%
==========================================
Files 680 680
Lines 38753 38762 +9
==========================================
- Hits 32960 32863 -97
- Misses 5793 5899 +106
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.55% <100%> (+0.25%)` | :arrow_up: |
| [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.41% <87.5%> (-0.34%)` | :arrow_down: |
| [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: |
| [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: |
| [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: |
| ... and [4 more](https://codecov.io/gh/apache/airflow/pull/6870/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=footer). Last update [033e863...ffaf101](https://codecov.io/gh/apache/airflow/pull/6870?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372534336
##########
File path: airflow/jobs/local_task_job.py
##########
@@ -95,6 +95,15 @@ def signal_handler(signum, frame):
# Monitor the task to see if it's done
return_code = self.task_runner.return_code()
if return_code is not None:
+ print(self.task_instance.state)
Review comment:
This is a debug line and shouldn't be committed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on issue #6870: [AIRFLOW-0578] Check
return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on issue #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#issuecomment-609906756
@ashb Please take a look at this issue
https://github.com/apache/airflow/issues/7962
This is exactly why we cannot ignore the exit code of a JOB
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] lucafuji commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
lucafuji commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372644672
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
That's the problem of python operator.....if you call sys.exit(0) from the python operator. It will not do any post processing like setting the state to succeed which is done in python operator.
So for this case I have following suggestions
1. disallow calling sys.exit in python operator, this can be done by a code scan before running.
2. running the python operator in another process and having the task state management in its own process.
I think it will be another PR if we really want to fix this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-0578] Check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-0578] Check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r372608786
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +81,110 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_invalid_return_code_with_bash_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_bash_operator'
+ invalid_task_id = 'test_invalid_return_code_bash_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.SUCCESS)
+ self.assertIsNotNone(valid_ti.pid)
+
+ def test_invalid_return_code_with_python_operator(self):
+ with create_session() as session:
+ dagbag = models.DagBag(
+ dag_folder=TEST_DAG_FOLDER,
+ include_examples=False,
+ )
+ dag_id = 'test_invalid_return_code'
+ valid_task_id = 'test_valid_return_code_python_operator'
+ invalid_task_id = 'test_invalid_return_code_python_operator'
+ dag = dagbag.get_dag(dag_id)
+ invalid_task = dag.get_task(invalid_task_id)
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ invalid_ti = TI(task=invalid_task, execution_date=DEFAULT_DATE)
+ invalid_ti.refresh_from_db()
+ invalid_ti.state = State.RUNNING
+ invalid_ti.hostname = get_hostname()
+ session.commit()
+
+ invalid_job = LocalTaskJob(task_instance=invalid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with self.assertRaisesRegex(AirflowException,
+ "LocalTaskJob process exited with non zero status"):
+ invalid_job.run()
+
+ invalid_ti = dr.get_task_instance(task_id=invalid_task.task_id,
+ session=session)
+ # task that returns invalid return code should fail.
+ self.assertEqual(invalid_ti.state, State.FAILED)
+ self.assertIsNotNone(invalid_ti.pid)
+
+ valid_task = dag.get_task(valid_task_id)
+ valid_ti = TI(task=valid_task, execution_date=DEFAULT_DATE)
+ valid_ti.refresh_from_db()
+ valid_ti.state = State.RUNNING
+ valid_ti.hostname = get_hostname()
+ session.commit()
+
+ valid_job = LocalTaskJob(task_instance=valid_ti,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ valid_job.run()
+ valid_ti = dr.get_task_instance(task_id=valid_task.task_id,
+ session=session)
+ # task that returns valid return code should succeed.
+ self.assertEqual(valid_ti.state, State.RUNNING)
Review comment:
If the job has run why is this TI still in running state? It should be succeeded, no?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6870:
[AIRFLOW-578] Fix check return code
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #6870: [AIRFLOW-578] Fix check return code
URL: https://github.com/apache/airflow/pull/6870#discussion_r365987002
##########
File path: tests/jobs/test_local_task_job.py
##########
@@ -81,6 +83,46 @@ def test_localtaskjob_essential_attr(self):
check_result_2 = [getattr(job1, attr) is not None for attr in essential_attr]
self.assertTrue(all(check_result_2))
+ def test_localtaskjob_invalid_return_code(self):
+ dag = DAG(
+ 'test_localtaskjob_invalid_return_code',
+ start_date=DEFAULT_DATE,
+ default_args={'owner': 'airflow'})
+
+ task = BashOperator(
+ task_id='test_bash',
+ bash_command='exit 1',
+ dag=dag,
+ owner='airflow')
+ session = settings.Session()
+
+ dag.clear()
+ dr = dag.create_dagrun(run_id="test",
+ state=State.SUCCESS,
+ execution_date=DEFAULT_DATE,
+ start_date=DEFAULT_DATE,
+ session=session)
+ ti = dr.get_task_instance(task_id=task.task_id, session=session)
+ ti.state = State.RUNNING
+ ti.hostname = get_hostname()
+ ti.pid = 1
+ session.commit()
+
+ ti_run = TI(task=task, execution_date=DEFAULT_DATE)
+ job1 = LocalTaskJob(task_instance=ti_run,
+ ignore_ti_state=True,
+ executor=SequentialExecutor())
+ with patch.object(BaseTaskRunner, 'start',
+ return_value=None) as mock_method:
+ job1.run()
+ mock_method.assert_not_called()
Review comment:
Is this test actually testing the new code? By mocking the `start` method is any of the code actually running?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services