You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/22 01:26:25 UTC
[GitHub] [airflow] uranusjr opened a new pull request #18421: Correctly select DagRun.execution_date from db
uranusjr opened a new pull request #18421:
URL: https://github.com/apache/airflow/pull/18421
Fix #3413. The importance of test coverage and type annotationsā¦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713776338
##########
File path: airflow/models/dag.py
##########
@@ -1269,16 +1269,17 @@ def get_task_instances_before(
``base_date``, or more if there are manual task runs between the
requested period, which does not count toward ``num``.
"""
- min_date = (
- session.query(DagRun)
+ min_date: Optional[datetime] = (
+ session.query(DagRun.execution_date)
.filter(
DagRun.dag_id == self.dag_id,
DagRun.execution_date <= base_date,
DagRun.run_type != DagRunType.MANUAL,
)
.order_by(DagRun.execution_date.desc())
.offset(num)
- .first()
+ .limit(1)
+ .scalar()
Review comment:
I think `first()` would require unpacking the returning tuple manually? This feels slightly easier to read for me than
```python
min_date_row = session.query(...).first()
if min_date_row is None:
...
min_date = min_date_row[0]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713776338
##########
File path: airflow/models/dag.py
##########
@@ -1269,16 +1269,17 @@ def get_task_instances_before(
``base_date``, or more if there are manual task runs between the
requested period, which does not count toward ``num``.
"""
- min_date = (
- session.query(DagRun)
+ min_date: Optional[datetime] = (
+ session.query(DagRun.execution_date)
.filter(
DagRun.dag_id == self.dag_id,
DagRun.execution_date <= base_date,
DagRun.run_type != DagRunType.MANUAL,
)
.order_by(DagRun.execution_date.desc())
.offset(num)
- .first()
+ .limit(1)
+ .scalar()
Review comment:
I think `first()` would require unpacking the returning tuple manually? This feels slightly easier to read for me than
```python
min_date_row = session.query(...)
if min_date_row is None:
...
min_date = min_date_row[0]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713661446
##########
File path: airflow/models/dag.py
##########
@@ -1269,16 +1269,17 @@ def get_task_instances_before(
``base_date``, or more if there are manual task runs between the
requested period, which does not count toward ``num``.
"""
- min_date = (
- session.query(DagRun)
+ min_date: Optional[datetime] = (
+ session.query(DagRun.execution_date)
.filter(
DagRun.dag_id == self.dag_id,
DagRun.execution_date <= base_date,
DagRun.run_type != DagRunType.MANUAL,
)
.order_by(DagRun.execution_date.desc())
.offset(num)
- .first()
+ .limit(1)
+ .scalar()
Review comment:
```suggestion
.scalar()
```
I think using only `.scalar()` is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713769717
##########
File path: tests/www/views/test_views_dagrun.py
##########
@@ -61,16 +61,16 @@ def init_blank_dagrun():
This really shouldn't be needed, but tests elsewhere leave the db dirty.
"""
with create_session() as session:
- session.query(DagRun).delete()
session.query(TaskInstance).delete()
+ session.query(DagRun).delete()
Review comment:
I just did, was about to push š
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #18421:
URL: https://github.com/apache/airflow/pull/18421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713752452
##########
File path: airflow/models/dag.py
##########
@@ -1269,16 +1269,17 @@ def get_task_instances_before(
``base_date``, or more if there are manual task runs between the
requested period, which does not count toward ``num``.
"""
- min_date = (
- session.query(DagRun)
+ min_date: Optional[datetime] = (
+ session.query(DagRun.execution_date)
.filter(
DagRun.dag_id == self.dag_id,
DagRun.execution_date <= base_date,
DagRun.run_type != DagRunType.MANUAL,
)
.order_by(DagRun.execution_date.desc())
.offset(num)
- .first()
+ .limit(1)
+ .scalar()
Review comment:
Docs say `scalar()` fails if thereās more than one row
https://docs.sqlalchemy.org/en/14/orm/query.html#sqlalchemy.orm.Query.scalar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713783497
##########
File path: tests/www/views/test_views_dagrun.py
##########
@@ -60,17 +60,12 @@ def init_blank_dagrun():
This really shouldn't be needed, but tests elsewhere leave the db dirty.
"""
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TaskInstance).delete()
+ clear_db_runs()
@pytest.fixture(autouse=True)
def reset_dagrun():
- yield
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TaskInstance).delete()
+ clear_db_runs()
Review comment:
I think the `yield` is necessary for this fixture?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713525205
##########
File path: airflow/models/dag.py
##########
@@ -1269,8 +1269,8 @@ def get_task_instances_before(
``base_date``, or more if there are manual task runs between the
requested period, which does not count toward ``num``.
"""
- min_date = (
- session.query(DagRun)
+ min_date: Optional[datetime] = (
+ session.query(DagRun.execution_date)
Review comment:
This line is the only relevant fix; the rest are type annotations I added for debugging, they should be useful in the future as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#issuecomment-924929163
Uh, the cascading fix totally works for me, not sure why the CI is failing⦠HALP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713796860
##########
File path: tests/www/views/test_views_dagrun.py
##########
@@ -60,17 +60,12 @@ def init_blank_dagrun():
This really shouldn't be needed, but tests elsewhere leave the db dirty.
"""
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TaskInstance).delete()
+ clear_db_runs()
@pytest.fixture(autouse=True)
def reset_dagrun():
- yield
- with create_session() as session:
- session.query(DagRun).delete()
- session.query(TaskInstance).delete()
+ clear_db_runs()
Review comment:
Oops, thanks for spotting this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713777469
##########
File path: tests/www/views/test_views_dagrun.py
##########
@@ -61,16 +61,16 @@ def init_blank_dagrun():
This really shouldn't be needed, but tests elsewhere leave the db dirty.
"""
with create_session() as session:
- session.query(DagRun).delete()
session.query(TaskInstance).delete()
+ session.query(DagRun).delete()
Review comment:
I also took the chance to just use `clear_db_runs` in the fixture as well. A few more db calls (but should not actually remove anything) for easier maintenance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713766682
##########
File path: airflow/models/dag.py
##########
@@ -1269,16 +1269,17 @@ def get_task_instances_before(
``base_date``, or more if there are manual task runs between the
requested period, which does not count toward ``num``.
"""
- min_date = (
- session.query(DagRun)
+ min_date: Optional[datetime] = (
+ session.query(DagRun.execution_date)
.filter(
DagRun.dag_id == self.dag_id,
DagRun.execution_date <= base_date,
DagRun.run_type != DagRunType.MANUAL,
)
.order_by(DagRun.execution_date.desc())
.offset(num)
- .first()
+ .limit(1)
+ .scalar()
Review comment:
Curious why you went with `limit` instead of `first` ?
https://docs.sqlalchemy.org/en/14/orm/query.html#sqlalchemy.orm.Query.first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#issuecomment-924929163
Uh, the cascading fix totally works for me, not sure why the CI is failing⦠HALP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713768808
##########
File path: tests/www/views/test_views_dagrun.py
##########
@@ -61,16 +61,16 @@ def init_blank_dagrun():
This really shouldn't be needed, but tests elsewhere leave the db dirty.
"""
with create_session() as session:
- session.query(DagRun).delete()
session.query(TaskInstance).delete()
+ session.query(DagRun).delete()
Review comment:
Should we also update https://github.com/apache/airflow/blob/92e775962dd74bcc350bdb0445e9729b706a2f2f/tests/test_utils/db.py#L47-L48
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#issuecomment-924760246
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18421: Correctly select DagRun.execution_date from db
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18421:
URL: https://github.com/apache/airflow/pull/18421#discussion_r713525205
##########
File path: airflow/models/dag.py
##########
@@ -1269,8 +1269,8 @@ def get_task_instances_before(
``base_date``, or more if there are manual task runs between the
requested period, which does not count toward ``num``.
"""
- min_date = (
- session.query(DagRun)
+ min_date: Optional[datetime] = (
+ session.query(DagRun.execution_date)
Review comment:
This line is the only relevant fix; the rest are type annotations I added for debugging, they should be useful in the future as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org