You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2024/11/02 03:24:59 UTC
[PR] AIP-72: Add endpoints for updating TI state & TI Heartbeat [airflow]
kaxil opened a new pull request, #43602:
URL: https://github.com/apache/airflow/pull/43602
Part of https://github.com/apache/airflow/issues/43586
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828492203
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,232 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils import timezone
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+DEFAULT_START_DATE = timezone.parse("2024-10-31T11:00:00Z")
+DEFAULT_END_DATE = timezone.parse("2024-10-31T12:00:00Z")
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_running",
+ state=State.QUEUED,
+ session=session,
+ )
+
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ # Refresh the Task Instance from the database so that we can check the updated values
+ session.refresh(ti)
+ assert ti.state == State.RUNNING
+ assert ti.hostname == "random-hostname"
+ assert ti.unixname == "random-unixname"
+ assert ti.pid == 100
+ assert ti.start_date.isoformat() == "2024-10-31T12:00:00+00:00"
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ ("state", "end_date", "expected_state", "expected_end_date"),
+ [
+ # For SUCCESS & FAILED states, end_date is required
+ (State.SUCCESS, DEFAULT_END_DATE, State.SUCCESS, DEFAULT_END_DATE),
+ (State.FAILED, DEFAULT_END_DATE, State.FAILED, DEFAULT_END_DATE),
+ # When state is SKIPPED, end_date is optional because
+ # 1) a user can run a task
+ # and raise AirflowSkipException within the task without specifying the end_date
Review Comment:
The user isn't responsible for sending this request, the execution code in task SDK is, so we could choose to enforce an end date for this case if we want/think it's cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #43602:
URL: https://github.com/apache/airflow/pull/43602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #43602:
URL: https://github.com/apache/airflow/pull/43602#issuecomment-2457025627
> I have a (maybe/probably) crazy idea that might make our tests nicer, and easier to parallelize in future:
>
> If we use dependency overrides, we can make the session inside our routers use the same session https://fastapi.tiangolo.com/advanced/testing-dependencies/#use-the-appdependency_overrides-attribute
>
> But for now: lets set up this dep override and change the `commit()` in tests to `flush()` (possibly putting the flush inside the dag_maker fixture itself?)
Couldn't get it work easily, going to punt it to later!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828562594
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,232 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils import timezone
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+DEFAULT_START_DATE = timezone.parse("2024-10-31T11:00:00Z")
+DEFAULT_END_DATE = timezone.parse("2024-10-31T12:00:00Z")
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_running",
+ state=State.QUEUED,
+ session=session,
+ )
+
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ # Refresh the Task Instance from the database so that we can check the updated values
+ session.refresh(ti)
+ assert ti.state == State.RUNNING
+ assert ti.hostname == "random-hostname"
+ assert ti.unixname == "random-unixname"
+ assert ti.pid == 100
+ assert ti.start_date.isoformat() == "2024-10-31T12:00:00+00:00"
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ ("state", "end_date", "expected_state", "expected_end_date"),
+ [
+ # For SUCCESS & FAILED states, end_date is required
+ (State.SUCCESS, DEFAULT_END_DATE, State.SUCCESS, DEFAULT_END_DATE),
+ (State.FAILED, DEFAULT_END_DATE, State.FAILED, DEFAULT_END_DATE),
+ # When state is SKIPPED, end_date is optional because
+ # 1) a user can run a task
+ # and raise AirflowSkipException within the task without specifying the end_date
+ # 2) a task can be skipped by the scheduler without specifying the end_date
+ (State.SKIPPED, DEFAULT_END_DATE, State.SKIPPED, DEFAULT_END_DATE),
+ (State.SKIPPED, None, State.SKIPPED, None),
Review Comment:
Just simplified it and removed this in https://github.com/apache/airflow/pull/43602/commits/fa4f2228a2a057d7252e2d1dda64e3575f4b8b94
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,232 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils import timezone
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+DEFAULT_START_DATE = timezone.parse("2024-10-31T11:00:00Z")
+DEFAULT_END_DATE = timezone.parse("2024-10-31T12:00:00Z")
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_running",
+ state=State.QUEUED,
+ session=session,
+ )
+
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ # Refresh the Task Instance from the database so that we can check the updated values
+ session.refresh(ti)
+ assert ti.state == State.RUNNING
+ assert ti.hostname == "random-hostname"
+ assert ti.unixname == "random-unixname"
+ assert ti.pid == 100
+ assert ti.start_date.isoformat() == "2024-10-31T12:00:00+00:00"
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ ("state", "end_date", "expected_state", "expected_end_date"),
+ [
+ # For SUCCESS & FAILED states, end_date is required
+ (State.SUCCESS, DEFAULT_END_DATE, State.SUCCESS, DEFAULT_END_DATE),
+ (State.FAILED, DEFAULT_END_DATE, State.FAILED, DEFAULT_END_DATE),
+ # When state is SKIPPED, end_date is optional because
+ # 1) a user can run a task
+ # and raise AirflowSkipException within the task without specifying the end_date
Review Comment:
Simplified it and removed this in https://github.com/apache/airflow/pull/43602/commits/fa4f2228a2a057d7252e2d1dda64e3575f4b8b94
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828563405
##########
airflow/api_fastapi/execution_api/routes/task_instance.py:
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from uuid import UUID
+
+from fastapi import Body, Depends, HTTPException, status
+from sqlalchemy import update
+from sqlalchemy.exc import NoResultFound, SQLAlchemyError
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import select
+from typing_extensions import Annotated
+
+from airflow.api_fastapi.common.db.common import get_session
+from airflow.api_fastapi.common.router import AirflowRouter
+from airflow.api_fastapi.execution_api import schemas
+from airflow.models.taskinstance import TaskInstance as TI
+from airflow.utils.state import State
+
+# TODO: Add dependency on JWT token
+ti_router = AirflowRouter(
+ prefix="/task_instance",
+ tags=["Task Instance"],
+)
+
+
+log = logging.getLogger(__name__)
+
+
+@ti_router.patch(
+ "/{task_instance_id}/state",
+ status_code=status.HTTP_204_NO_CONTENT,
+ # TODO: Add Operation ID to control the function name in the OpenAPI spec
+ # TODO: Do we need to use create_openapi_http_exception_doc here?
+ responses={
+ status.HTTP_404_NOT_FOUND: {"description": "Task Instance not found"},
+ status.HTTP_409_CONFLICT: {"description": "The TI is already in the requested state"},
+ },
+)
+async def ti_update_state(
+ task_instance_id: UUID,
+ ti_patch_payload: Annotated[schemas.TIStateUpdate, Body()],
+ session: Annotated[Session, Depends(get_session)],
+):
+ """
+ Update the state of a TaskInstance.
+
+ Not all state transitions are valid, and transitioning to some states required extra information to be
+ passed along. (Check our the schemas for details, the rendered docs might not reflect this accurately)
+ """
+ # We only use UUID above for validation purposes
+ ti_id_str = str(task_instance_id)
+
+ old = select(TI.state).where(TI.id == ti_id_str).with_for_update()
+ try:
+ (previous_state,) = session.execute(old).one()
+ except NoResultFound:
+ log.error("Task Instance %s not found", ti_id_str)
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found",
+ },
+ )
+
+ # We exclude_unset to avoid updating fields that are not set in the payload
+ data = ti_patch_payload.model_dump(exclude_unset=True)
+
+ query = update(TI).where(TI.id == ti_id_str).values(data)
+
+ if isinstance(ti_patch_payload, schemas.TIEnterRunningPayload):
+ if previous_state != State.QUEUED:
+ log.warning(
+ "Can not start Task Instance ('%s') in invalid state: %s",
+ ti_id_str,
+ previous_state,
+ )
+
+ # TODO: Pass a RFC 9457 compliant error message in "detail" field
+ # https://datatracker.ietf.org/doc/html/rfc9457
+ # to provide more information about the error
+ # FastAPI will automatically convert this to a JSON response
+ # This might be added in FastAPI in https://github.com/fastapi/fastapi/issues/10370
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail={
+ "reason": "invalid_state",
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": previous_state,
+ },
+ )
+ log.info("Task with %s state started on %s ", previous_state, ti_patch_payload.hostname)
+ # Ensure there is no end date set.
+ query = query.values(
+ end_date=None,
+ hostname=ti_patch_payload.hostname,
+ unixname=ti_patch_payload.unixname,
+ pid=ti_patch_payload.pid,
+ state=State.RUNNING,
+ )
+ elif isinstance(ti_patch_payload, schemas.TITerminalStatePayload):
+ # If state is skipped, we need to set the end_date to the start_date if it is not set
+ end_date = ti_patch_payload.end_date
+ if ti_patch_payload.state == State.SKIPPED and not ti_patch_payload.end_date:
+ query = query.values(end_date=TI.start_date, duration=0)
+ elif ti_patch_payload.state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ # We do not update any other fields for these states
+ # as we do not expect it to even reach the worker.
+ # The scheduler should handle these states.
+ pass
+ else:
+ if not end_date:
Review Comment:
Simplified it and removed this in https://github.com/apache/airflow/pull/43602/commits/fa4f2228a2a057d7252e2d1dda64e3575f4b8b94
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1827983453
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
Review Comment:
Worth asserting the other columns are set too?
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
+ [
+ (State.SUCCESS, "2024-10-31T12:00:00Z", State.SUCCESS),
+ (State.FAILED, "2024-10-31T12:05:00Z", State.FAILED),
+ (State.SKIPPED, "2024-10-31T12:05:00Z", State.SKIPPED),
+ (State.SKIPPED, None, State.SKIPPED),
+ (State.UPSTREAM_FAILED, None, State.UPSTREAM_FAILED),
+ (State.REMOVED, None, State.REMOVED),
+ ],
+ )
+ def test_ti_update_state_to_terminal(
+ self, client, session, create_task_instance, state, end_date, expected_state
+ ):
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_terminal",
+ state=State.RUNNING,
+ )
+ session.commit()
Review Comment:
Do we need the commit here? Should we be passing the session in to `create_task_instance` too?
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
+ [
+ (State.SUCCESS, "2024-10-31T12:00:00Z", State.SUCCESS),
+ (State.FAILED, "2024-10-31T12:05:00Z", State.FAILED),
+ (State.SKIPPED, "2024-10-31T12:05:00Z", State.SKIPPED),
+ (State.SKIPPED, None, State.SKIPPED),
+ (State.UPSTREAM_FAILED, None, State.UPSTREAM_FAILED),
+ (State.REMOVED, None, State.REMOVED),
+ ],
+ )
+ def test_ti_update_state_to_terminal(
+ self, client, session, create_task_instance, state, end_date, expected_state
+ ):
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_terminal",
+ state=State.RUNNING,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ payload = {"state": state}
+ if end_date:
+ payload["end_date"] = end_date
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json=payload,
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == expected_state
+
+ if expected_state == State.SKIPPED and not end_date:
+ assert (
+ session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) == ti.start_date
+ )
+ elif expected_state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ assert session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) is None
Review Comment:
```suggestion
ti = session.get(TaskInstance, ti.id)
assert ti.state == expected_state
assert ti.end_date.isoformat() == end_date
```
I think? It's a bit hard to grok what this test is asserting with the if/else in the tests, I'd rarther we made it fully table driven.
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
Review Comment:
Nit: create a file-local test fixture for this?
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
+ [
+ (State.SUCCESS, "2024-10-31T12:00:00Z", State.SUCCESS),
+ (State.FAILED, "2024-10-31T12:05:00Z", State.FAILED),
+ (State.SKIPPED, "2024-10-31T12:05:00Z", State.SKIPPED),
+ (State.SKIPPED, None, State.SKIPPED),
+ (State.UPSTREAM_FAILED, None, State.UPSTREAM_FAILED),
+ (State.REMOVED, None, State.REMOVED),
+ ],
+ )
+ def test_ti_update_state_to_terminal(
+ self, client, session, create_task_instance, state, end_date, expected_state
+ ):
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_terminal",
+ state=State.RUNNING,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ payload = {"state": state}
+ if end_date:
+ payload["end_date"] = end_date
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json=payload,
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == expected_state
+
+ if expected_state == State.SKIPPED and not end_date:
+ assert (
+ session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) == ti.start_date
+ )
+ elif expected_state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ assert session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) is None
+
+ def test_ti_update_state_not_found(self, client, session):
+ """
+ Test that a 404 error is returned when the Task Instance does not exist.
+ """
+ task_instance_id = "0182e924-0f1e-77e6-ab50-e977118bc139"
+
+ # Assert that the Task Instance does not exist
Review Comment:
```suggestion
# Pre-condition: the Task Instance does not exist
```
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
Review Comment:
Nit
```suggestion
("state", "end_date", "expected_state"),
```
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
+ [
+ (State.SUCCESS, "2024-10-31T12:00:00Z", State.SUCCESS),
+ (State.FAILED, "2024-10-31T12:05:00Z", State.FAILED),
+ (State.SKIPPED, "2024-10-31T12:05:00Z", State.SKIPPED),
+ (State.SKIPPED, None, State.SKIPPED),
+ (State.UPSTREAM_FAILED, None, State.UPSTREAM_FAILED),
+ (State.REMOVED, None, State.REMOVED),
+ ],
+ )
+ def test_ti_update_state_to_terminal(
+ self, client, session, create_task_instance, state, end_date, expected_state
+ ):
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_terminal",
+ state=State.RUNNING,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ payload = {"state": state}
+ if end_date:
+ payload["end_date"] = end_date
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json=payload,
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == expected_state
+
+ if expected_state == State.SKIPPED and not end_date:
+ assert (
+ session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) == ti.start_date
+ )
+ elif expected_state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ assert session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) is None
+
+ def test_ti_update_state_not_found(self, client, session):
+ """
+ Test that a 404 error is returned when the Task Instance does not exist.
+ """
+ task_instance_id = "0182e924-0f1e-77e6-ab50-e977118bc139"
+
+ # Assert that the Task Instance does not exist
+ assert session.scalar(select(TaskInstance.id).where(TaskInstance.id == task_instance_id)) is None
+ session.commit()
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828487510
##########
airflow/api_fastapi/execution_api/routes/task_instance.py:
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from uuid import UUID
+
+from fastapi import Body, Depends, HTTPException, status
+from sqlalchemy import update
+from sqlalchemy.exc import NoResultFound, SQLAlchemyError
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import select
+from typing_extensions import Annotated
+
+from airflow.api_fastapi.common.db.common import get_session
+from airflow.api_fastapi.common.router import AirflowRouter
+from airflow.api_fastapi.execution_api import schemas
+from airflow.models.taskinstance import TaskInstance as TI
+from airflow.utils.state import State
+
+# TODO: Add dependency on JWT token
+ti_router = AirflowRouter(
+ prefix="/task_instance",
+ tags=["Task Instance"],
+)
+
+
+log = logging.getLogger(__name__)
+
+
+@ti_router.patch(
+ "/{task_instance_id}/state",
+ status_code=status.HTTP_204_NO_CONTENT,
+ # TODO: Add Operation ID to control the function name in the OpenAPI spec
+ # TODO: Do we need to use create_openapi_http_exception_doc here?
+ responses={
+ status.HTTP_404_NOT_FOUND: {"description": "Task Instance not found"},
+ status.HTTP_409_CONFLICT: {"description": "The TI is already in the requested state"},
+ },
+)
+async def ti_update_state(
+ task_instance_id: UUID,
+ ti_patch_payload: Annotated[schemas.TIStateUpdate, Body()],
+ session: Annotated[Session, Depends(get_session)],
+):
+ """
+ Update the state of a TaskInstance.
+
+ Not all state transitions are valid, and transitioning to some states required extra information to be
+ passed along. (Check our the schemas for details, the rendered docs might not reflect this accurately)
+ """
+ # We only use UUID above for validation purposes
+ ti_id_str = str(task_instance_id)
+
+ old = select(TI.state).where(TI.id == ti_id_str).with_for_update()
+ try:
+ (previous_state,) = session.execute(old).one()
+ except NoResultFound:
+ log.error("Task Instance %s not found", ti_id_str)
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found",
+ },
+ )
+
+ # We exclude_unset to avoid updating fields that are not set in the payload
+ data = ti_patch_payload.model_dump(exclude_unset=True)
+
+ query = update(TI).where(TI.id == ti_id_str).values(data)
+
+ if isinstance(ti_patch_payload, schemas.TIEnterRunningPayload):
+ if previous_state != State.QUEUED:
+ log.warning(
+ "Can not start Task Instance ('%s') in invalid state: %s",
+ ti_id_str,
+ previous_state,
+ )
+
+ # TODO: Pass a RFC 9457 compliant error message in "detail" field
+ # https://datatracker.ietf.org/doc/html/rfc9457
+ # to provide more information about the error
+ # FastAPI will automatically convert this to a JSON response
+ # This might be added in FastAPI in https://github.com/fastapi/fastapi/issues/10370
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail={
+ "reason": "invalid_state",
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": previous_state,
+ },
+ )
+ log.info("Task with %s state started on %s ", previous_state, ti_patch_payload.hostname)
+ # Ensure there is no end date set.
+ query = query.values(
+ end_date=None,
+ hostname=ti_patch_payload.hostname,
+ unixname=ti_patch_payload.unixname,
+ pid=ti_patch_payload.pid,
+ state=State.RUNNING,
+ )
+ elif isinstance(ti_patch_payload, schemas.TITerminalStatePayload):
+ # If state is skipped, we need to set the end_date to the start_date if it is not set
+ end_date = ti_patch_payload.end_date
+ if ti_patch_payload.state == State.SKIPPED and not ti_patch_payload.end_date:
+ query = query.values(end_date=TI.start_date, duration=0)
+ elif ti_patch_payload.state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ # We do not update any other fields for these states
+ # as we do not expect it to even reach the worker.
+ # The scheduler should handle these states.
+ pass
+ else:
+ if not end_date:
Review Comment:
Minor nit: Make this `elif not end_date:`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828560269
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
Review Comment:
Yeah `create_task_instance` doesn't commit the TI to the database unless the class is decorated with `@pytest.mark.need_serialized_dag`.
https://github.com/apache/airflow/blob/5ccfe5c4c7093984e56c79e3ac02f5677ff39af4/tests_common/pytest_plugin.py#L811-L816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828492387
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,232 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils import timezone
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+DEFAULT_START_DATE = timezone.parse("2024-10-31T11:00:00Z")
+DEFAULT_END_DATE = timezone.parse("2024-10-31T12:00:00Z")
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_running",
+ state=State.QUEUED,
+ session=session,
+ )
+
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ # Refresh the Task Instance from the database so that we can check the updated values
+ session.refresh(ti)
+ assert ti.state == State.RUNNING
+ assert ti.hostname == "random-hostname"
+ assert ti.unixname == "random-unixname"
+ assert ti.pid == 100
+ assert ti.start_date.isoformat() == "2024-10-31T12:00:00+00:00"
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+
+ response = client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ ("state", "end_date", "expected_state", "expected_end_date"),
+ [
+ # For SUCCESS & FAILED states, end_date is required
+ (State.SUCCESS, DEFAULT_END_DATE, State.SUCCESS, DEFAULT_END_DATE),
+ (State.FAILED, DEFAULT_END_DATE, State.FAILED, DEFAULT_END_DATE),
+ # When state is SKIPPED, end_date is optional because
+ # 1) a user can run a task
+ # and raise AirflowSkipException within the task without specifying the end_date
+ # 2) a task can be skipped by the scheduler without specifying the end_date
+ (State.SKIPPED, DEFAULT_END_DATE, State.SKIPPED, DEFAULT_END_DATE),
+ (State.SKIPPED, None, State.SKIPPED, None),
Review Comment:
Scheduler won't be hitting this API server though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1829083596
##########
airflow/api_fastapi/execution_api/routes/task_instance.py:
##########
@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from uuid import UUID
+
+from fastapi import Body, Depends, HTTPException, status
+from sqlalchemy import update
+from sqlalchemy.exc import NoResultFound, SQLAlchemyError
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import select
+from typing_extensions import Annotated
+
+from airflow.api_fastapi.common.db.common import get_session
+from airflow.api_fastapi.common.router import AirflowRouter
+from airflow.api_fastapi.execution_api import schemas
+from airflow.models.taskinstance import TaskInstance as TI
+from airflow.utils.state import State
+
+# TODO: Add dependency on JWT token
+ti_router = AirflowRouter(
+ prefix="/task_instance",
+ tags=["Task Instance"],
+)
+
+
+log = logging.getLogger(__name__)
+
+
+@ti_router.patch(
+ "/{task_instance_id}/state",
+ status_code=status.HTTP_204_NO_CONTENT,
+ # TODO: Add Operation ID to control the function name in the OpenAPI spec
+ # TODO: Do we need to use create_openapi_http_exception_doc here?
+ responses={
+ status.HTTP_404_NOT_FOUND: {"description": "Task Instance not found"},
+ status.HTTP_409_CONFLICT: {"description": "The TI is already in the requested state"},
+ },
+)
+async def ti_update_state(
+ task_instance_id: UUID,
+ ti_patch_payload: Annotated[schemas.TIStateUpdate, Body()],
+ session: Annotated[Session, Depends(get_session)],
+):
+ """
+ Update the state of a TaskInstance.
+
+ Not all state transitions are valid, and transitioning to some states required extra information to be
+ passed along. (Check our the schemas for details, the rendered docs might not reflect this accurately)
+ """
+ # We only use UUID above for validation purposes
+ ti_id_str = str(task_instance_id)
+
+ old = select(TI.state).where(TI.id == ti_id_str).with_for_update()
+ try:
+ (previous_state,) = session.execute(old).one()
+ except NoResultFound:
+ log.error("Task Instance %s not found", ti_id_str)
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found",
+ },
+ )
+
+ # We exclude_unset to avoid updating fields that are not set in the payload
+ data = ti_patch_payload.model_dump(exclude_unset=True)
+
+ query = update(TI).where(TI.id == ti_id_str).values(data)
+
+ if isinstance(ti_patch_payload, schemas.TIEnterRunningPayload):
+ if previous_state != State.QUEUED:
+ log.warning(
+ "Can not start Task Instance ('%s') in invalid state: %s",
+ ti_id_str,
+ previous_state,
+ )
+
+ # TODO: Pass a RFC 9457 compliant error message in "detail" field
+ # https://datatracker.ietf.org/doc/html/rfc9457
+ # to provide more information about the error
+ # FastAPI will automatically convert this to a JSON response
+ # This might be added in FastAPI in https://github.com/fastapi/fastapi/issues/10370
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail={
+ "reason": "invalid_state",
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": previous_state,
+ },
+ )
+ log.info("Task with %s state started on %s ", previous_state, ti_patch_payload.hostname)
+ # Ensure there is no end date set.
+ query = query.values(
+ end_date=None,
+ hostname=ti_patch_payload.hostname,
+ unixname=ti_patch_payload.unixname,
+ pid=ti_patch_payload.pid,
+ state=State.RUNNING,
+ )
+ elif isinstance(ti_patch_payload, schemas.TITerminalStatePayload):
+ query = TI.duration_expression_update(ti_patch_payload.end_date, query, session.bind)
+
+ # TODO: Replace this with FastAPI's Custom Exception handling:
+ # https://fastapi.tiangolo.com/tutorial/handling-errors/#install-custom-exception-handlers
+ try:
+ result = session.execute(query)
+ log.info("TI %s state updated: %s row(s) affected", ti_id_str, result.rowcount)
+ session.commit()
+ except SQLAlchemyError as e:
+ session.rollback()
+ log.error("Error updating Task Instance state: %s", e)
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Database error occurred"
Review Comment:
I think the rollback on `SQLAlchemyError` is already done by the `get_session` context manager. (This is inherent to all endpoints, all of them should rollback the transaction if something went wrong in the endpoint)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add endpoints for updating TI state & TI Heartbeat [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1827597478
##########
airflow/api_fastapi/execution_api/schemas.py:
##########
@@ -0,0 +1,121 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import Annotated, Literal, Union
+
+from pydantic import (
+ AfterValidator,
+ AwareDatetime,
+ BaseModel,
+ ConfigDict,
+ Discriminator,
+ Field,
+ Tag,
+ WithJsonSchema,
+)
+
+from airflow.utils import timezone
+from airflow.utils.state import State, TaskInstanceState as TIState
+
+UtcDateTime = Annotated[AwareDatetime, AfterValidator(lambda d: d.astimezone(timezone.utc))]
+"""UTCDateTime is a datetime with timezone information"""
Review Comment:
This feels like it should exist elsewhere already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1829255806
##########
airflow/api_fastapi/execution_api/routes/task_instance.py:
##########
@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from uuid import UUID
+
+from fastapi import Body, Depends, HTTPException, status
+from sqlalchemy import update
+from sqlalchemy.exc import NoResultFound, SQLAlchemyError
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import select
+from typing_extensions import Annotated
+
+from airflow.api_fastapi.common.db.common import get_session
+from airflow.api_fastapi.common.router import AirflowRouter
+from airflow.api_fastapi.execution_api import schemas
+from airflow.models.taskinstance import TaskInstance as TI
+from airflow.utils.state import State
+
+# TODO: Add dependency on JWT token
+ti_router = AirflowRouter(
+ prefix="/task_instance",
+ tags=["Task Instance"],
+)
+
+
+log = logging.getLogger(__name__)
+
+
+@ti_router.patch(
+ "/{task_instance_id}/state",
+ status_code=status.HTTP_204_NO_CONTENT,
+ # TODO: Add Operation ID to control the function name in the OpenAPI spec
+ # TODO: Do we need to use create_openapi_http_exception_doc here?
+ responses={
+ status.HTTP_404_NOT_FOUND: {"description": "Task Instance not found"},
+ status.HTTP_409_CONFLICT: {"description": "The TI is already in the requested state"},
+ },
+)
+async def ti_update_state(
+ task_instance_id: UUID,
+ ti_patch_payload: Annotated[schemas.TIStateUpdate, Body()],
+ session: Annotated[Session, Depends(get_session)],
+):
+ """
+ Update the state of a TaskInstance.
+
+ Not all state transitions are valid, and transitioning to some states required extra information to be
+ passed along. (Check our the schemas for details, the rendered docs might not reflect this accurately)
+ """
+ # We only use UUID above for validation purposes
+ ti_id_str = str(task_instance_id)
+
+ old = select(TI.state).where(TI.id == ti_id_str).with_for_update()
+ try:
+ (previous_state,) = session.execute(old).one()
+ except NoResultFound:
+ log.error("Task Instance %s not found", ti_id_str)
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found",
+ },
+ )
+
+ # We exclude_unset to avoid updating fields that are not set in the payload
+ data = ti_patch_payload.model_dump(exclude_unset=True)
+
+ query = update(TI).where(TI.id == ti_id_str).values(data)
+
+ if isinstance(ti_patch_payload, schemas.TIEnterRunningPayload):
+ if previous_state != State.QUEUED:
+ log.warning(
+ "Can not start Task Instance ('%s') in invalid state: %s",
+ ti_id_str,
+ previous_state,
+ )
+
+ # TODO: Pass a RFC 9457 compliant error message in "detail" field
+ # https://datatracker.ietf.org/doc/html/rfc9457
+ # to provide more information about the error
+ # FastAPI will automatically convert this to a JSON response
+ # This might be added in FastAPI in https://github.com/fastapi/fastapi/issues/10370
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail={
+ "reason": "invalid_state",
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": previous_state,
+ },
+ )
+ log.info("Task with %s state started on %s ", previous_state, ti_patch_payload.hostname)
+ # Ensure there is no end date set.
+ query = query.values(
+ end_date=None,
+ hostname=ti_patch_payload.hostname,
+ unixname=ti_patch_payload.unixname,
+ pid=ti_patch_payload.pid,
+ state=State.RUNNING,
+ )
+ elif isinstance(ti_patch_payload, schemas.TITerminalStatePayload):
+ query = TI.duration_expression_update(ti_patch_payload.end_date, query, session.bind)
+
+ # TODO: Replace this with FastAPI's Custom Exception handling:
+ # https://fastapi.tiangolo.com/tutorial/handling-errors/#install-custom-exception-handlers
+ try:
+ result = session.execute(query)
+ log.info("TI %s state updated: %s row(s) affected", ti_id_str, result.rowcount)
+ session.commit()
+ except SQLAlchemyError as e:
+ session.rollback()
+ log.error("Error updating Task Instance state: %s", e)
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Database error occurred"
Review Comment:
Removed session.commit & session.rollback but kept the error handling to raise HTTPException
##########
airflow/api_fastapi/execution_api/routes/task_instance.py:
##########
@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from uuid import UUID
+
+from fastapi import Body, Depends, HTTPException, status
+from sqlalchemy import update
+from sqlalchemy.exc import NoResultFound, SQLAlchemyError
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import select
+from typing_extensions import Annotated
+
+from airflow.api_fastapi.common.db.common import get_session
+from airflow.api_fastapi.common.router import AirflowRouter
+from airflow.api_fastapi.execution_api import schemas
+from airflow.models.taskinstance import TaskInstance as TI
+from airflow.utils.state import State
+
+# TODO: Add dependency on JWT token
+ti_router = AirflowRouter(
+ prefix="/task_instance",
+ tags=["Task Instance"],
+)
+
+
+log = logging.getLogger(__name__)
+
+
+@ti_router.patch(
+ "/{task_instance_id}/state",
+ status_code=status.HTTP_204_NO_CONTENT,
+ # TODO: Add Operation ID to control the function name in the OpenAPI spec
+ # TODO: Do we need to use create_openapi_http_exception_doc here?
+ responses={
+ status.HTTP_404_NOT_FOUND: {"description": "Task Instance not found"},
+ status.HTTP_409_CONFLICT: {"description": "The TI is already in the requested state"},
+ },
+)
+async def ti_update_state(
+ task_instance_id: UUID,
+ ti_patch_payload: Annotated[schemas.TIStateUpdate, Body()],
+ session: Annotated[Session, Depends(get_session)],
+):
+ """
+ Update the state of a TaskInstance.
+
+ Not all state transitions are valid, and transitioning to some states required extra information to be
+ passed along. (Check our the schemas for details, the rendered docs might not reflect this accurately)
+ """
+ # We only use UUID above for validation purposes
+ ti_id_str = str(task_instance_id)
+
+ old = select(TI.state).where(TI.id == ti_id_str).with_for_update()
+ try:
+ (previous_state,) = session.execute(old).one()
+ except NoResultFound:
+ log.error("Task Instance %s not found", ti_id_str)
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found",
+ },
+ )
+
+ # We exclude_unset to avoid updating fields that are not set in the payload
+ data = ti_patch_payload.model_dump(exclude_unset=True)
+
+ query = update(TI).where(TI.id == ti_id_str).values(data)
+
+ if isinstance(ti_patch_payload, schemas.TIEnterRunningPayload):
+ if previous_state != State.QUEUED:
+ log.warning(
+ "Can not start Task Instance ('%s') in invalid state: %s",
+ ti_id_str,
+ previous_state,
+ )
+
+ # TODO: Pass a RFC 9457 compliant error message in "detail" field
+ # https://datatracker.ietf.org/doc/html/rfc9457
+ # to provide more information about the error
+ # FastAPI will automatically convert this to a JSON response
+ # This might be added in FastAPI in https://github.com/fastapi/fastapi/issues/10370
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail={
+ "reason": "invalid_state",
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": previous_state,
+ },
+ )
+ log.info("Task with %s state started on %s ", previous_state, ti_patch_payload.hostname)
+ # Ensure there is no end date set.
+ query = query.values(
+ end_date=None,
+ hostname=ti_patch_payload.hostname,
+ unixname=ti_patch_payload.unixname,
+ pid=ti_patch_payload.pid,
+ state=State.RUNNING,
+ )
+ elif isinstance(ti_patch_payload, schemas.TITerminalStatePayload):
+ query = TI.duration_expression_update(ti_patch_payload.end_date, query, session.bind)
+
+ # TODO: Replace this with FastAPI's Custom Exception handling:
+ # https://fastapi.tiangolo.com/tutorial/handling-errors/#install-custom-exception-handlers
+ try:
+ result = session.execute(query)
+ log.info("TI %s state updated: %s row(s) affected", ti_id_str, result.rowcount)
+ session.commit()
+ except SQLAlchemyError as e:
+ session.rollback()
+ log.error("Error updating Task Instance state: %s", e)
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Database error occurred"
Review Comment:
Removed session.commit & session.rollback but kept the error handling to raise HTTPException for now (for this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1827965071
##########
airflow/api_fastapi/execution_api/schemas.py:
##########
@@ -0,0 +1,121 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import Annotated, Literal, Union
+
+from pydantic import (
+ AfterValidator,
+ AwareDatetime,
+ BaseModel,
+ ConfigDict,
+ Discriminator,
+ Field,
+ Tag,
+ WithJsonSchema,
+)
+
+from airflow.utils import timezone
+from airflow.utils.state import State, TaskInstanceState as TIState
+
+UtcDateTime = Annotated[AwareDatetime, AfterValidator(lambda d: d.astimezone(timezone.utc))]
+"""UTCDateTime is a datetime with timezone information"""
Review Comment:
moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1829105579
##########
airflow/api_fastapi/execution_api/routes/task_instance.py:
##########
@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from uuid import UUID
+
+from fastapi import Body, Depends, HTTPException, status
+from sqlalchemy import update
+from sqlalchemy.exc import NoResultFound, SQLAlchemyError
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import select
+from typing_extensions import Annotated
+
+from airflow.api_fastapi.common.db.common import get_session
+from airflow.api_fastapi.common.router import AirflowRouter
+from airflow.api_fastapi.execution_api import schemas
+from airflow.models.taskinstance import TaskInstance as TI
+from airflow.utils.state import State
+
+# TODO: Add dependency on JWT token
+ti_router = AirflowRouter(
+ prefix="/task_instance",
+ tags=["Task Instance"],
+)
+
+
+log = logging.getLogger(__name__)
+
+
+@ti_router.patch(
+ "/{task_instance_id}/state",
+ status_code=status.HTTP_204_NO_CONTENT,
+ # TODO: Add Operation ID to control the function name in the OpenAPI spec
+ # TODO: Do we need to use create_openapi_http_exception_doc here?
+ responses={
+ status.HTTP_404_NOT_FOUND: {"description": "Task Instance not found"},
+ status.HTTP_409_CONFLICT: {"description": "The TI is already in the requested state"},
+ },
+)
+async def ti_update_state(
+ task_instance_id: UUID,
+ ti_patch_payload: Annotated[schemas.TIStateUpdate, Body()],
+ session: Annotated[Session, Depends(get_session)],
+):
+ """
+ Update the state of a TaskInstance.
+
+ Not all state transitions are valid, and transitioning to some states required extra information to be
+ passed along. (Check our the schemas for details, the rendered docs might not reflect this accurately)
+ """
+ # We only use UUID above for validation purposes
+ ti_id_str = str(task_instance_id)
+
+ old = select(TI.state).where(TI.id == ti_id_str).with_for_update()
+ try:
+ (previous_state,) = session.execute(old).one()
+ except NoResultFound:
+ log.error("Task Instance %s not found", ti_id_str)
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found",
+ },
+ )
+
+ # We exclude_unset to avoid updating fields that are not set in the payload
+ data = ti_patch_payload.model_dump(exclude_unset=True)
+
+ query = update(TI).where(TI.id == ti_id_str).values(data)
+
+ if isinstance(ti_patch_payload, schemas.TIEnterRunningPayload):
+ if previous_state != State.QUEUED:
+ log.warning(
+ "Can not start Task Instance ('%s') in invalid state: %s",
+ ti_id_str,
+ previous_state,
+ )
+
+ # TODO: Pass a RFC 9457 compliant error message in "detail" field
+ # https://datatracker.ietf.org/doc/html/rfc9457
+ # to provide more information about the error
+ # FastAPI will automatically convert this to a JSON response
+ # This might be added in FastAPI in https://github.com/fastapi/fastapi/issues/10370
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail={
+ "reason": "invalid_state",
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": previous_state,
+ },
+ )
+ log.info("Task with %s state started on %s ", previous_state, ti_patch_payload.hostname)
+ # Ensure there is no end date set.
+ query = query.values(
+ end_date=None,
+ hostname=ti_patch_payload.hostname,
+ unixname=ti_patch_payload.unixname,
+ pid=ti_patch_payload.pid,
+ state=State.RUNNING,
+ )
+ elif isinstance(ti_patch_payload, schemas.TITerminalStatePayload):
+ query = TI.duration_expression_update(ti_patch_payload.end_date, query, session.bind)
+
+ # TODO: Replace this with FastAPI's Custom Exception handling:
+ # https://fastapi.tiangolo.com/tutorial/handling-errors/#install-custom-exception-handlers
+ try:
+ result = session.execute(query)
+ log.info("TI %s state updated: %s row(s) affected", ti_id_str, result.rowcount)
+ session.commit()
+ except SQLAlchemyError as e:
+ session.rollback()
+ log.error("Error updating Task Instance state: %s", e)
+ raise HTTPException(
+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Database error occurred"
Review Comment:
It should rollback on any error, right, so this would work?
Do we need the `session.commit()` too, or is that handled for us as well?
```suggestion
result = session.execute(query)
log.info("TI %s state updated: %s row(s) affected", ti_id_str, result.rowcount)
session.commit()
except SQLAlchemyError as e:
log.error("Error updating Task Instance state: %s", e)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Database error occurred"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828116232
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
+ [
+ (State.SUCCESS, "2024-10-31T12:00:00Z", State.SUCCESS),
+ (State.FAILED, "2024-10-31T12:05:00Z", State.FAILED),
+ (State.SKIPPED, "2024-10-31T12:05:00Z", State.SKIPPED),
+ (State.SKIPPED, None, State.SKIPPED),
+ (State.UPSTREAM_FAILED, None, State.UPSTREAM_FAILED),
+ (State.REMOVED, None, State.REMOVED),
+ ],
+ )
+ def test_ti_update_state_to_terminal(
+ self, client, session, create_task_instance, state, end_date, expected_state
+ ):
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_terminal",
+ state=State.RUNNING,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ payload = {"state": state}
+ if end_date:
+ payload["end_date"] = end_date
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json=payload,
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == expected_state
+
+ if expected_state == State.SKIPPED and not end_date:
+ assert (
+ session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) == ti.start_date
+ )
+ elif expected_state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ assert session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) is None
+
+ def test_ti_update_state_not_found(self, client, session):
+ """
+ Test that a 404 error is returned when the Task Instance does not exist.
+ """
+ task_instance_id = "0182e924-0f1e-77e6-ab50-e977118bc139"
+
+ # Assert that the Task Instance does not exist
+ assert session.scalar(select(TaskInstance.id).where(TaskInstance.id == task_instance_id)) is None
+ session.commit()
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828489845
##########
airflow/api_fastapi/execution_api/routes/task_instance.py:
##########
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import logging
+from uuid import UUID
+
+from fastapi import Body, Depends, HTTPException, status
+from sqlalchemy import update
+from sqlalchemy.exc import NoResultFound, SQLAlchemyError
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import select
+from typing_extensions import Annotated
+
+from airflow.api_fastapi.common.db.common import get_session
+from airflow.api_fastapi.common.router import AirflowRouter
+from airflow.api_fastapi.execution_api import schemas
+from airflow.models.taskinstance import TaskInstance as TI
+from airflow.utils.state import State
+
+# TODO: Add dependency on JWT token
+ti_router = AirflowRouter(
+ prefix="/task_instance",
+ tags=["Task Instance"],
+)
+
+
+log = logging.getLogger(__name__)
+
+
+@ti_router.patch(
+ "/{task_instance_id}/state",
+ status_code=status.HTTP_204_NO_CONTENT,
+ # TODO: Add Operation ID to control the function name in the OpenAPI spec
+ # TODO: Do we need to use create_openapi_http_exception_doc here?
+ responses={
+ status.HTTP_404_NOT_FOUND: {"description": "Task Instance not found"},
+ status.HTTP_409_CONFLICT: {"description": "The TI is already in the requested state"},
+ },
+)
+async def ti_update_state(
+ task_instance_id: UUID,
+ ti_patch_payload: Annotated[schemas.TIStateUpdate, Body()],
+ session: Annotated[Session, Depends(get_session)],
+):
+ """
+ Update the state of a TaskInstance.
+
+ Not all state transitions are valid, and transitioning to some states required extra information to be
+ passed along. (Check our the schemas for details, the rendered docs might not reflect this accurately)
+ """
+ # We only use UUID above for validation purposes
+ ti_id_str = str(task_instance_id)
+
+ old = select(TI.state).where(TI.id == ti_id_str).with_for_update()
+ try:
+ (previous_state,) = session.execute(old).one()
+ except NoResultFound:
+ log.error("Task Instance %s not found", ti_id_str)
+ raise HTTPException(
+ status_code=status.HTTP_404_NOT_FOUND,
+ detail={
+ "reason": "not_found",
+ "message": "Task Instance not found",
+ },
+ )
+
+ # We exclude_unset to avoid updating fields that are not set in the payload
+ data = ti_patch_payload.model_dump(exclude_unset=True)
+
+ query = update(TI).where(TI.id == ti_id_str).values(data)
+
+ if isinstance(ti_patch_payload, schemas.TIEnterRunningPayload):
+ if previous_state != State.QUEUED:
+ log.warning(
+ "Can not start Task Instance ('%s') in invalid state: %s",
+ ti_id_str,
+ previous_state,
+ )
+
+ # TODO: Pass a RFC 9457 compliant error message in "detail" field
+ # https://datatracker.ietf.org/doc/html/rfc9457
+ # to provide more information about the error
+ # FastAPI will automatically convert this to a JSON response
+ # This might be added in FastAPI in https://github.com/fastapi/fastapi/issues/10370
+ raise HTTPException(
+ status_code=status.HTTP_409_CONFLICT,
+ detail={
+ "reason": "invalid_state",
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": previous_state,
+ },
+ )
+ log.info("Task with %s state started on %s ", previous_state, ti_patch_payload.hostname)
+ # Ensure there is no end date set.
+ query = query.values(
+ end_date=None,
+ hostname=ti_patch_payload.hostname,
+ unixname=ti_patch_payload.unixname,
+ pid=ti_patch_payload.pid,
+ state=State.RUNNING,
+ )
+ elif isinstance(ti_patch_payload, schemas.TITerminalStatePayload):
+ # If state is skipped, we need to set the end_date to the start_date if it is not set
+ end_date = ti_patch_payload.end_date
+ if ti_patch_payload.state == State.SKIPPED and not ti_patch_payload.end_date:
+ query = query.values(end_date=TI.start_date, duration=0)
+ elif ti_patch_payload.state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ # We do not update any other fields for these states
+ # as we do not expect it to even reach the worker.
+ # The scheduler should handle these states.
+ pass
+ else:
+ if not end_date:
Review Comment:
Should this be handled as a model validation on the schema perhaps? That way all validation logic remains on the schema keeping the controller simpler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828562805
##########
tests/api_fastapi/execution_api/conftest.py:
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+from fastapi.testclient import TestClient
+
+from airflow.api_fastapi.app import create_app
+
+
+@pytest.fixture
+def client():
+ return TestClient(create_app(apps="execution"))
Review Comment:
Updated in https://github.com/apache/airflow/pull/43602/commits/fa4f2228a2a057d7252e2d1dda64e3575f4b8b94
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828125676
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828116594
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
+ [
+ (State.SUCCESS, "2024-10-31T12:00:00Z", State.SUCCESS),
+ (State.FAILED, "2024-10-31T12:05:00Z", State.FAILED),
+ (State.SKIPPED, "2024-10-31T12:05:00Z", State.SKIPPED),
+ (State.SKIPPED, None, State.SKIPPED),
+ (State.UPSTREAM_FAILED, None, State.UPSTREAM_FAILED),
+ (State.REMOVED, None, State.REMOVED),
+ ],
+ )
+ def test_ti_update_state_to_terminal(
+ self, client, session, create_task_instance, state, end_date, expected_state
+ ):
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_terminal",
+ state=State.RUNNING,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ payload = {"state": state}
+ if end_date:
+ payload["end_date"] = end_date
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json=payload,
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == expected_state
+
+ if expected_state == State.SKIPPED and not end_date:
+ assert (
+ session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) == ti.start_date
+ )
+ elif expected_state in [State.UPSTREAM_FAILED, State.REMOVED]:
+ assert session.scalar(select(TaskInstance.end_date).where(TaskInstance.id == ti.id)) is None
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1827994466
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
Review Comment:
Actually, rather than fiel local this should probably by in `tests/api_fastapi/execution_api/conftest.py` -- and if we can call it `client` too still that might be good?
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
Review Comment:
Actually, rather than file local this should probably by in `tests/api_fastapi/execution_api/conftest.py` -- and if we can call it `client` too still that might be good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828117211
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
+ [
+ (State.SUCCESS, "2024-10-31T12:00:00Z", State.SUCCESS),
+ (State.FAILED, "2024-10-31T12:05:00Z", State.FAILED),
+ (State.SKIPPED, "2024-10-31T12:05:00Z", State.SKIPPED),
+ (State.SKIPPED, None, State.SKIPPED),
+ (State.UPSTREAM_FAILED, None, State.UPSTREAM_FAILED),
+ (State.REMOVED, None, State.REMOVED),
+ ],
+ )
+ def test_ti_update_state_to_terminal(
+ self, client, session, create_task_instance, state, end_date, expected_state
+ ):
+ ti = create_task_instance(
+ task_id="test_ti_update_state_to_terminal",
+ state=State.RUNNING,
+ )
+ session.commit()
Review Comment:
Updated
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
+
+ def test_ti_update_state_conflict_if_not_queued(self, client, session, create_task_instance):
+ """
+ Test that a 409 error is returned when the Task Instance is not in a state where it can be marked as
+ running. In this case, the Task Instance is first in NONE state so it cannot be marked as running.
+ """
+ ti = create_task_instance(
+ task_id="test_ti_update_state_conflict_if_not_queued",
+ state=State.NONE,
+ )
+ session.commit()
+ test_client = client(apps="execution")
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 409
+ assert response.json() == {
+ "detail": {
+ "message": "TI was not in a state where it could be marked as running",
+ "previous_state": State.NONE,
+ "reason": "invalid_state",
+ }
+ }
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.NONE
+
+ @pytest.mark.parametrize(
+ "state, end_date, expected_state",
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828117544
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
@@ -0,0 +1,192 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+
+from airflow.models.taskinstance import TaskInstance
+from airflow.utils.state import State
+
+from tests_common.test_utils.db import clear_db_runs
+
+pytestmark = pytest.mark.db_test
+
+
+class TestTIUpdateState:
+ def setup_method(self):
+ clear_db_runs()
+
+ def teardown_method(self):
+ clear_db_runs()
+
+ def test_ti_update_state_to_running(self, client, session, create_task_instance):
+ """
+ Test that the Task Instance state is updated to running when the Task Instance is in a state where it can be
+ marked as running.
+ """
+ test_client = client(apps="execution")
+ ti = create_task_instance(task_id="test_ti_update_state_to_running", state=State.QUEUED)
+
+ session.commit()
+
+ response = test_client.patch(
+ f"/execution/task_instance/{ti.id}/state",
+ json={
+ "state": "running",
+ "hostname": "random-hostname",
+ "unixname": "random-unixname",
+ "pid": 100,
+ "start_date": "2024-10-31T12:00:00Z",
+ },
+ )
+
+ assert response.status_code == 204
+ assert response.text == ""
+
+ assert session.scalar(select(TaskInstance.state).where(TaskInstance.id == ti.id)) == State.RUNNING
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828491128
##########
tests/api_fastapi/execution_api/conftest.py:
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+from fastapi.testclient import TestClient
+
+from airflow.api_fastapi.app import create_app
+
+
+@pytest.fixture
+def client():
+ return TestClient(create_app(apps="execution"))
Review Comment:
`create_app` has caching right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #43602:
URL: https://github.com/apache/airflow/pull/43602#discussion_r1828493044
##########
tests/api_fastapi/execution_api/routes/test_task_instance.py:
##########
Review Comment:
Do we need the `session.commit()`s after the create_task_instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AIP-72: Add "update TI state" endpoint for Execution API [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #43602:
URL: https://github.com/apache/airflow/pull/43602#issuecomment-2456710118
I have a (maybe/probably) crazy idea that might make our tests nicer, and easier to parallelize in future:
If we use dependency overrides, we can make the session inside our routers use the same session https://fastapi.tiangolo.com/advanced/testing-dependencies/#use-the-appdependency_overrides-attribute
But for now: lets set up this dep override and change the `commit()` in tests to `flush()` (possibly putting the flush inside the dag_maker fixture itself?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org