You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/08 04:07:14 UTC
[GitHub] [airflow] jedcunningham opened a new pull request, #24912: More typing in SchedulerJob and TaskInstance
jedcunningham opened a new pull request, #24912:
URL: https://github.com/apache/airflow/pull/24912
I noticed these were missing some typing. This doesn't bring us to complete coverage in TaskInstance, but gets all the easy ones out of the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916531138
##########
airflow/jobs/scheduler_job.py:
##########
@@ -1194,7 +1195,7 @@ def _schedule_dag_run(
return callback_to_run
@provide_session
- def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session=None):
+ def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session: Session = NEW_SESSION) -> None:
Review Comment:
I think the `provide_session` should be removed here. It's not used and it's not proper to create a new session inside the scheduler loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916534930
##########
airflow/models/taskinstance.py:
##########
@@ -197,7 +199,7 @@ def clear_task_instances(
if dag and dag.has_task(task_id):
task = dag.get_task(task_id)
ti.refresh_from_task(task)
- task_retries = task.retries
+ task_retries = task.retries or 0
Review Comment:
It wouldn’t work, and honestly I don’t even know what `None` means here. I think this is exposing a more fundamental bug…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #24912:
URL: https://github.com/apache/airflow/pull/24912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916535365
##########
airflow/jobs/scheduler_job.py:
##########
@@ -1194,7 +1195,7 @@ def _schedule_dag_run(
return callback_to_run
@provide_session
- def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session=None):
+ def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session: Session = NEW_SESSION) -> None:
Review Comment:
Good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916446383
##########
airflow/jobs/scheduler_job.py:
##########
@@ -790,7 +794,7 @@ def _update_dag_run_state_for_paused_dags(self):
dag = SerializedDagModel.get_dag(dag_id)
if dag is None:
continue
- dag_runs = DagRun.find(dag_id=dag_id, state=State.RUNNING)
+ dag_runs = DagRun.find(dag_id=dag_id, state=DagRunState.RUNNING)
Review Comment:
Also here, using the wrong one.
##########
airflow/models/taskinstance.py:
##########
@@ -790,11 +791,11 @@ def log_url(self):
)
@property
- def mark_success_url(self):
+ def mark_success_url(self) -> str:
"""URL to mark TI success"""
base_url = conf.get('webserver', 'BASE_URL')
- return base_url + (
- "/confirm"
+ return (
+ f"{base_url}/confirm"
Review Comment:
This avoids the "Optional" aspect of `conf.get`, plus I think it's easier to read.
##########
airflow/jobs/scheduler_job.py:
##########
@@ -444,9 +448,9 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session =
dag_id,
task_instance,
)
- session.query(TI).filter(TI.dag_id == dag_id, TI.state == State.SCHEDULED).update(
- {TI.state: State.FAILED}, synchronize_session='fetch'
- )
+ session.query(TI).filter(
+ TI.dag_id == dag_id, TI.state == TaskInstanceState.SCHEDULED
+ ).update({TI.state: TaskInstanceState.FAILED}, synchronize_session='fetch')
Review Comment:
Technically just a typing change, but we were using the wrong one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916460586
##########
airflow/models/taskinstance.py:
##########
@@ -163,12 +165,12 @@ def set_current_context(context: Context) -> Generator[Context, None, None]:
def clear_task_instances(
- tis,
- session,
+ tis: List["TaskInstance"],
+ session: Session,
activate_dag_runs=None,
dag=None,
Review Comment:
Interesting, never considered using typing to hint at things being deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916469538
##########
airflow/models/taskinstance.py:
##########
@@ -197,7 +199,7 @@ def clear_task_instances(
if dag and dag.has_task(task_id):
task = dag.get_task(task_id)
ti.refresh_from_task(task)
- task_retries = task.retries
+ task_retries = task.retries or 0
Review Comment:
I also had to add this, as `task.retries` is `Optional[int]`. Not sure it would have worked if it was None, but I'll look closer tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916450273
##########
airflow/models/taskinstance.py:
##########
@@ -163,12 +165,12 @@ def set_current_context(context: Context) -> Generator[Context, None, None]:
def clear_task_instances(
- tis,
- session,
+ tis: List["TaskInstance"],
+ session: Session,
activate_dag_runs=None,
dag=None,
Review Comment:
```suggestion
activate_dag_runs: None = None,
dag: Optional[DAG] = None,
```
##########
airflow/models/taskinstance.py:
##########
@@ -790,11 +791,11 @@ def log_url(self):
)
@property
- def mark_success_url(self):
+ def mark_success_url(self) -> str:
"""URL to mark TI success"""
base_url = conf.get('webserver', 'BASE_URL')
- return base_url + (
- "/confirm"
+ return (
+ f"{base_url}/confirm"
Review Comment:
Should we use `get_mandatory_value` instead?
##########
airflow/jobs/scheduler_job.py:
##########
@@ -1194,7 +1195,7 @@ def _schedule_dag_run(
return callback_to_run
@provide_session
- def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session=None):
+ def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session: Session = None) -> None:
Review Comment:
Use `NEW_SESSION` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #24912:
URL: https://github.com/apache/airflow/pull/24912#issuecomment-1178653639
I just pushed a commit to remove unnecessary `provide_session` decorators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24912:
URL: https://github.com/apache/airflow/pull/24912#issuecomment-1181500409
Conflicts need to be resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916470253
##########
airflow/models/taskinstance.py:
##########
@@ -790,11 +791,11 @@ def log_url(self):
)
@property
- def mark_success_url(self):
+ def mark_success_url(self) -> str:
"""URL to mark TI success"""
base_url = conf.get('webserver', 'BASE_URL')
- return base_url + (
- "/confirm"
+ return (
+ f"{base_url}/confirm"
Review Comment:
We could, and it'll be functionally equivalent because we have a default. That said, I do still like the fstring approach better anyways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
jedcunningham closed pull request #24912: More typing in SchedulerJob and TaskInstance
URL: https://github.com/apache/airflow/pull/24912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24912: More typing in SchedulerJob and TaskInstance
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #24912:
URL: https://github.com/apache/airflow/pull/24912#discussion_r916456470
##########
airflow/jobs/scheduler_job.py:
##########
@@ -1194,7 +1195,7 @@ def _schedule_dag_run(
return callback_to_run
@provide_session
- def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session=None):
+ def _verify_integrity_if_dag_changed(self, dag_run: DagRun, session: Session = None) -> None:
Review Comment:
I've used `NEW_SESSION` across the board, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org