You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2025/07/14 07:25:42 UTC
[PR] ⚠️ WIP: POC of task sdk integration tests [airflow]
amoghrajesh opened a new pull request, #53332:
URL: https://github.com/apache/airflow/pull/53332
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225433759
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
+
+
+def print_diagnostics(compose, compose_version, docker_version):
+ """Print diagnostic information when test fails."""
+ console.print("[red]=== DIAGNOSTIC INFORMATION ===[/]")
+ console.print(f"Docker version: {docker_version}")
+ console.print(f"Docker Compose version: {compose_version}")
+ console.print("\n[yellow]Container Status:[/]")
+ try:
+ containers = compose.compose.ps()
+ for container in containers:
+ console.print(f" {container.name}: {container.state}")
+ except Exception as e:
+ console.print(f" Error getting container status: {e}")
+
+ console.print("\n[yellow]Container Logs:[/]")
+ try:
+ logs = compose.compose.logs()
+ console.print(logs)
+ except Exception as e:
+ console.print(f" Error getting logs: {e}")
+
+
+def test_task_sdk_health(tmp_path_factory, monkeypatch):
+ """Test Task SDK health check using docker-compose environment."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-task-sdk-test")
+ console.print(f"[yellow]Tests are run in {tmp_dir}")
+
+ # Copy docker-compose.yaml to temp directory
+ docker_compose_file = TASK_SDK_TESTS_ROOT / "docker" / "docker-compose.yaml"
+ tmp_docker_compose_file = tmp_dir / "docker-compose.yaml"
+ copyfile(docker_compose_file, tmp_docker_compose_file)
+
+ # Set environment variables for the test
+ monkeypatch.setenv("AIRFLOW_IMAGE_NAME", os.environ.get("DOCKER_IMAGE", "apache/airflow:3.0.3"))
Review Comment:
Yep did that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2230287496
##########
dev/breeze/doc/05_test_commands.rst:
##########
@@ -317,6 +317,25 @@ through ``breeze testing docker-compose-tests`` command.
The docker-compose tests are in ``docker-tests/`` folder in the main repo.
+Running task-sdk integration tests
Review Comment:
Yeah sounds like a good idea. Let me do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh merged PR #53332:
URL: https://github.com/apache/airflow/pull/53332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3092401224
> Update on this: the tests work when I run them using breeze locally but fail in Ci for some reason, need to get to the bottom of it
cool nice amogh, am also nearly there with testcontainers running e2e tests . will open PR tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3092441796
@gopidesupavan nice nice. I will be able to get to this only on monday
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220489052
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
+
+
+def print_diagnostics(compose, compose_version, docker_version):
+ """Print diagnostic information when test fails."""
+ console.print("[red]=== DIAGNOSTIC INFORMATION ===[/]")
+ console.print(f"Docker version: {docker_version}")
+ console.print(f"Docker Compose version: {compose_version}")
+ console.print("\n[yellow]Container Status:[/]")
+ try:
+ containers = compose.compose.ps()
+ for container in containers:
+ console.print(f" {container.name}: {container.state}")
+ except Exception as e:
+ console.print(f" Error getting container status: {e}")
+
+ console.print("\n[yellow]Container Logs:[/]")
+ try:
+ logs = compose.compose.logs()
+ console.print(logs)
+ except Exception as e:
+ console.print(f" Error getting logs: {e}")
+
+
+def test_task_sdk_health(tmp_path_factory, monkeypatch):
+ """Test Task SDK health check using docker-compose environment."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-task-sdk-test")
+ console.print(f"[yellow]Tests are run in {tmp_dir}")
+
+ # Copy docker-compose.yaml to temp directory
+ docker_compose_file = TASK_SDK_TESTS_ROOT / "docker" / "docker-compose.yaml"
Review Comment:
Yeah and moving to constants :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220481903
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
Review Comment:
```suggestion
TASK_SDK_TESTS_ROOT = Path(__file__).parents[2]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2230330606
##########
dev/breeze/doc/05_test_commands.rst:
##########
@@ -317,6 +317,25 @@ through ``breeze testing docker-compose-tests`` command.
The docker-compose tests are in ``docker-tests/`` folder in the main repo.
+Running task-sdk integration tests
Review Comment:
Vibe code a lof of it and got a good document up: 941aa7ada3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2227673495
##########
task-sdk-tests/tests/task_sdk_tests/conftest.py:
##########
@@ -0,0 +1,88 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+
+import pytest
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, DOCKER_IMAGE
+
+
+@pytest.fixture
+def default_docker_image():
+ return DOCKER_IMAGE
+
+
+@pytest.fixture(scope="session")
+def install_task_sdk():
Review Comment:
Good idea, just integrated it to see how it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3102446204
Alright, to debug this better, I am going to log in into codespaces to try. It provides the closest environment to a runner as I understand it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220171585
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ environment:
+ &airflow-common-env
+ AIRFLOW__CORE__EXECUTOR: LocalExecutor
+ AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
+ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ AIRFLOW__CORE__FERNET_KEY: ''
+ AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
+ AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'
+ user: "${AIRFLOW_UID:-50000}:0"
+ depends_on:
+ &airflow-common-depends-on
+ postgres:
+ condition: service_healthy
+
+services:
+ postgres:
+ image: postgres:13
+ environment:
+ POSTGRES_USER: airflow
+ POSTGRES_PASSWORD: airflow
+ POSTGRES_DB: airflow
+ healthcheck:
+ test: ["CMD", "pg_isready", "-U", "airflow"]
+ interval: 10s
+ retries: 5
+ start_period: 5s
+ restart: always
+
+ airflow-init:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Creating missing opt dirs if missing:"
+ mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
+ echo "Airflow version:"
+ /entrypoint airflow version
+ echo "Running airflow config list to create default config file if missing."
+ /entrypoint airflow config list >/dev/null
+ echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID}:0"
+ chown -R "${AIRFLOW_UID}:0" /opt/airflow/
+ environment:
+ <<: *airflow-common-env
+ _AIRFLOW_DB_MIGRATE: 'true'
+ _AIRFLOW_WWW_USER_CREATE: 'true'
+ _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+ _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ _PIP_ADDITIONAL_REQUIREMENTS: ''
+ user: "0:0"
+ depends_on:
+ <<: *airflow-common-depends-on
+
+ airflow-apiserver:
+ <<: *airflow-common
+ command: api-server
+ ports:
+ - "8080:8080"
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version"]
+ interval: 30s
+ timeout: 10s
+ retries: 5
+ start_period: 30s
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-scheduler:
Review Comment:
Are you not needing a dag parser as component in the compose to make it "working"?
##########
task-sdk-tests/pyproject.toml:
##########
Review Comment:
Instead of making a new top-level-folder for integration tests - can you change the folder structure similar like in providers that the `task-sdk/tests` contains a subfolder for `unit` and another for `integration`? That would save another top-level folder and all tests unit/integration would be below the package tree?
##########
task-sdk-tests/tests/task_sdk_tests/constants.py:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+AIRFLOW_ROOT_PATH = Path(__file__).resolve().parents[3]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.10"
+DEFAULT_DOCKER_IMAGE = f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
Review Comment:
A lot of such constants in other cases are within breeze - should we leverage them or breeze is passing them or should re really define the python version defaults here separately?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220498150
##########
dev/breeze/doc/05_test_commands.rst:
##########
@@ -317,6 +317,24 @@ through ``breeze testing docker-compose-tests`` command.
The docker-compose tests are in ``docker-tests/`` folder in the main repo.
+Running task-sdk integration tests
+..................................
+
+You can use Breeze to run the task-sdk integration tests. Those tests are run using Production image
+and they are running test with the Quick-start docker compose we have. Some scenarios of task-sdk tests are run here.
+
+.. image:: ./images/output_testing_docker-compose-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_task-sdk-integration-tests.svg
+ :width: 100%
+ :alt: Breeze testing task-sdk-integration-tests
+
+You can also iterate over those tests with pytest command, but - unlike regular unit tests and
+Helm tests, they need to be run in local virtual environment. They also require to have
+``DOCKER_IMAGE`` environment variable set, pointing to the image to test if you do not run them
+through ``breeze testing task-sdk-integration-tests`` command.
Review Comment:
That's not entirely right. From what I see you can do:
```
breeze prod-image build
```
And then the local prod image `ghcr.io/apache/airflow/main/prod/python....`
(likely it shoudl be updated also above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3101489085
Thank you for your review folks, will start resolving your comments once I get a green run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed pull request #53332: 🚧 Task SDK Integration Testing Infrastructure
URL: https://github.com/apache/airflow/pull/53332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3092395363
Update on this: the tests work when I run them using breeze locally but fail in Ci for some reason, need to get to the bottom of it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2218927194
##########
task-sdk-tests/temp_healthcheck.txt:
##########
@@ -0,0 +1 @@
+ test: ["CMD-SHELL", "pgrep -f scheduler"]
Review Comment:
Nit: might've been a miss from my end, is this being used? And if so does it need it's own file?
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
Review Comment:
Nit: Missed comment cleanup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3116672280
@potiuk added a new document, thanks for the suggestion. What do you think now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3107734342
Alright, so the test that I was worried about finally passes now: https://github.com/apache/airflow/actions/runs/16469399106/job/46556857176.
Let me start addressing comments now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3095695849
Alright, it was passing because i had task sdk installed locally. Removed that and trying. The test runs on host machine not scheduler, my bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225432499
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
+
+
+def print_diagnostics(compose, compose_version, docker_version):
+ """Print diagnostic information when test fails."""
+ console.print("[red]=== DIAGNOSTIC INFORMATION ===[/]")
+ console.print(f"Docker version: {docker_version}")
+ console.print(f"Docker Compose version: {compose_version}")
+ console.print("\n[yellow]Container Status:[/]")
+ try:
+ containers = compose.compose.ps()
+ for container in containers:
+ console.print(f" {container.name}: {container.state}")
+ except Exception as e:
+ console.print(f" Error getting container status: {e}")
+
+ console.print("\n[yellow]Container Logs:[/]")
+ try:
+ logs = compose.compose.logs()
+ console.print(logs)
+ except Exception as e:
+ console.print(f" Error getting logs: {e}")
+
+
+def test_task_sdk_health(tmp_path_factory, monkeypatch):
+ """Test Task SDK health check using docker-compose environment."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-task-sdk-test")
+ console.print(f"[yellow]Tests are run in {tmp_dir}")
+
+ # Copy docker-compose.yaml to temp directory
+ docker_compose_file = TASK_SDK_TESTS_ROOT / "docker" / "docker-compose.yaml"
Review Comment:
Addressed in 1215209d22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225422877
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
Review Comment:
Fair ask, made the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3098424130
Besides comments above... looking forward to merge, because I mostly can replicate the efforts here to use them for a Edge Worker test as well that I wanted to implement since a long long time... and with compose this looks "leaner" then making another K8s based test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3102498484
Ran it on codespaces and I have some hope from the last commit 🤞🏽
```
Docker compose started for task SDK test
Installing apache-airflow-task-sdk==1.0.3...
Installing from: /workspaces/airflow/task-sdk
Using Python executable: /workspaces/airflow/.venv/bin/python
Running command: uv pip install --python /workspaces/airflow/.venv/bin/python /workspaces/airflow/task-sdk
Using Python 3.12.1 environment at: /workspaces/airflow/.venv
Resolved 123 packages in 2.25s
Built apache-airflow-task-sdk @ file:///workspaces/airflow/task-sdk
Prepared 1 package in 532ms
Uninstalled 1 package in 3ms
░░░░░░░░░░░░░░░░░░░░ [0/1] Installing wheels... warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
If the cache and target directories are on different filesystems, hardlinking may not be supported.
If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 1 package in 1ms
~ apache-airflow-task-sdk==1.1.0 (from file:///workspaces/airflow/task-sdk)
Task SDK installed successfully with UV!
Verifying installation...
✅ Import successful
Making health check request...
[2025-07-22T12:25:31.808+0000] {_client.py:1025} INFO - HTTP Request: GET http://localhost:8080/execution/health/ping "HTTP/1.1 200 OK"
======================== Health Check Response =========================
Status Code: 200
Response Headers:
date: Tue, 22 Jul 2025 12:25:30 GMT
server: uvicorn
content-type: application/json
airflow-api-version: 2025-08-10
vary: Accept-Encoding
content-encoding: gzip
transfer-encoding: chunked
Response Body:
{'ok': ['airflow.api_fastapi.auth.tokens.JWTValidator'], 'failing': {}}
========================================================================
Docker compose instance deleted
PASSED
```
Will clean up the code once I do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220487070
##########
task-sdk-tests/tests/task_sdk_tests/constants.py:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+AIRFLOW_ROOT_PATH = Path(__file__).resolve().parents[3]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.10"
+DEFAULT_DOCKER_IMAGE = f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
Review Comment:
This is a separate distribution and one that is "used" from breeze not the other way round, so that's not really wise to those constrants from there. Sometimes DRY < less coupling and circular dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2222106798
##########
task-sdk-tests/tests/task_sdk_tests/constants.py:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+AIRFLOW_ROOT_PATH = Path(__file__).resolve().parents[3]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.10"
+DEFAULT_DOCKER_IMAGE = f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
Review Comment:
> But was just a suggestion to reduce the number of lines to change if we bump the minor Python version next year...
I think what matters here is that you should be able to run the tests in local venv without breeze. Breeze is mean to make a reproducible one-command run that you should be able to copy&paste from CI and reproduce locally in case things do not work. But I think what @amoghrajesh strives for is (similar like in other distributions of ours) it should also be possible (but with less reproducibility due to environment and such) `uv run pytest` in the `task-sdk-tests` and get it running. this has the advantage of all the debugging stuff, running things from the IDE easily etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225448075
##########
task-sdk-tests/pyproject.toml:
##########
Review Comment:
I thought of that initially, but it would be non essential because I want this to be a top level project and not under task-sdk essentially. Think similar to docker-tests. This will allow me to package and test in workflows.
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ environment:
+ &airflow-common-env
+ AIRFLOW__CORE__EXECUTOR: LocalExecutor
+ AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
+ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ AIRFLOW__CORE__FERNET_KEY: ''
+ AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
+ AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'
+ user: "${AIRFLOW_UID:-50000}:0"
+ depends_on:
+ &airflow-common-depends-on
+ postgres:
+ condition: service_healthy
+
+services:
+ postgres:
+ image: postgres:13
+ environment:
+ POSTGRES_USER: airflow
+ POSTGRES_PASSWORD: airflow
+ POSTGRES_DB: airflow
+ healthcheck:
+ test: ["CMD", "pg_isready", "-U", "airflow"]
+ interval: 10s
+ retries: 5
+ start_period: 5s
+ restart: always
+
+ airflow-init:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Creating missing opt dirs if missing:"
+ mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
+ echo "Airflow version:"
+ /entrypoint airflow version
+ echo "Running airflow config list to create default config file if missing."
+ /entrypoint airflow config list >/dev/null
+ echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID}:0"
+ chown -R "${AIRFLOW_UID}:0" /opt/airflow/
+ environment:
+ <<: *airflow-common-env
+ _AIRFLOW_DB_MIGRATE: 'true'
+ _AIRFLOW_WWW_USER_CREATE: 'true'
+ _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+ _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ _PIP_ADDITIONAL_REQUIREMENTS: ''
+ user: "0:0"
+ depends_on:
+ <<: *airflow-common-depends-on
+
+ airflow-apiserver:
+ <<: *airflow-common
+ command: api-server
+ ports:
+ - "8080:8080"
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version"]
+ interval: 30s
+ timeout: 10s
+ retries: 5
+ start_period: 30s
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-scheduler:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Installing Task SDK version: ${TASK_SDK_VERSION:-1.1.0}"
+ pip install --no-deps apache-airflow-task-sdk==${TASK_SDK_VERSION:-1.1.0}
Review Comment:
Actually this part needn't be done at all i guess now that I think of it, the tests aren't running here but in the host. But, we are installing from current sources.
##########
task-sdk-tests/tests/task_sdk_tests/constants.py:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+AIRFLOW_ROOT_PATH = Path(__file__).resolve().parents[3]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.10"
+DEFAULT_DOCKER_IMAGE = f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
Review Comment:
> I think what matters here is that you should be able to run the tests in local venv without breeze
Precisely. I intend to have a way so that folks can run it either via breeze as well as from an venv. It will allow for local debugging and running from IDE too which I tried.
Eg:
1. breeze testing task-sdk-integration-tests
2. source .venv/bin/activate & uv run python -m pytest task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py -s --color=yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2230766427
##########
task-sdk-tests/tests/task_sdk_tests/constants.py:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+AIRFLOW_ROOT_PATH = Path(__file__).resolve().parents[3]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.10"
+DEFAULT_DOCKER_IMAGE = f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
Review Comment:
Resolving as I provided explanation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3112567122
> source .venv/bin/activate & uv run python -m pytest task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py -s --color=yes
FYI when you use uv run, you do not need to activate the venv at all. The `uv run` does it for you and it also syncs the venv according to the local pyproject.toml in your current working environment.
So the **right* way of running this is:
```bash
cd task-sdk-tests/
uv run pytest
```
I also suggest to enable color by default in pyproject toml :). I can't see a reason why we would not want color.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220498150
##########
dev/breeze/doc/05_test_commands.rst:
##########
@@ -317,6 +317,24 @@ through ``breeze testing docker-compose-tests`` command.
The docker-compose tests are in ``docker-tests/`` folder in the main repo.
+Running task-sdk integration tests
+..................................
+
+You can use Breeze to run the task-sdk integration tests. Those tests are run using Production image
+and they are running test with the Quick-start docker compose we have. Some scenarios of task-sdk tests are run here.
+
+.. image:: ./images/output_testing_docker-compose-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_task-sdk-integration-tests.svg
+ :width: 100%
+ :alt: Breeze testing task-sdk-integration-tests
+
+You can also iterate over those tests with pytest command, but - unlike regular unit tests and
+Helm tests, they need to be run in local virtual environment. They also require to have
+``DOCKER_IMAGE`` environment variable set, pointing to the image to test if you do not run them
+through ``breeze testing task-sdk-integration-tests`` command.
Review Comment:
That's not entirely right. From what I see you can do:
```
breeze prod-image build
```
And then the local prod image `ghcr.io/apache/airflow/main/prod/python....` will be used by default
(likely it shoudl be updated also above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2221102865
##########
task-sdk-tests/tests/task_sdk_tests/constants.py:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+AIRFLOW_ROOT_PATH = Path(__file__).resolve().parents[3]
+
+DEFAULT_PYTHON_MAJOR_MINOR_VERSION = "3.10"
+DEFAULT_DOCKER_IMAGE = f"ghcr.io/apache/airflow/main/prod/python{DEFAULT_PYTHON_MAJOR_MINOR_VERSION}:latest"
+DOCKER_IMAGE = os.environ.get("DOCKER_IMAGE") or DEFAULT_DOCKER_IMAGE
Review Comment:
No, might be a mis-understanding. Did not want to add python coupling but removing here and adding as default parameter when calling and enforcing a parameter given when breeze runs the tests.
But was just a suggestion to reduce the number of lines to change if we bump the minor Python version next year...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220512718
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ environment:
+ &airflow-common-env
+ AIRFLOW__CORE__EXECUTOR: LocalExecutor
+ AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
+ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ AIRFLOW__CORE__FERNET_KEY: ''
+ AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
+ AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'
+ user: "${AIRFLOW_UID:-50000}:0"
+ depends_on:
+ &airflow-common-depends-on
+ postgres:
+ condition: service_healthy
+
+services:
+ postgres:
+ image: postgres:13
+ environment:
+ POSTGRES_USER: airflow
+ POSTGRES_PASSWORD: airflow
+ POSTGRES_DB: airflow
+ healthcheck:
+ test: ["CMD", "pg_isready", "-U", "airflow"]
+ interval: 10s
+ retries: 5
+ start_period: 5s
+ restart: always
+
+ airflow-init:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Creating missing opt dirs if missing:"
+ mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
+ echo "Airflow version:"
+ /entrypoint airflow version
+ echo "Running airflow config list to create default config file if missing."
+ /entrypoint airflow config list >/dev/null
+ echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID}:0"
+ chown -R "${AIRFLOW_UID}:0" /opt/airflow/
+ environment:
+ <<: *airflow-common-env
+ _AIRFLOW_DB_MIGRATE: 'true'
+ _AIRFLOW_WWW_USER_CREATE: 'true'
+ _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+ _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ _PIP_ADDITIONAL_REQUIREMENTS: ''
+ user: "0:0"
+ depends_on:
+ <<: *airflow-common-depends-on
+
+ airflow-apiserver:
+ <<: *airflow-common
+ command: api-server
+ ports:
+ - "8080:8080"
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version"]
+ interval: 30s
+ timeout: 10s
+ retries: 5
+ start_period: 30s
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-scheduler:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Installing Task SDK version: ${TASK_SDK_VERSION:-1.1.0}"
+ pip install --no-deps apache-airflow-task-sdk==${TASK_SDK_VERSION:-1.1.0}
Review Comment:
I think we should test here also against the "current sources" version - not necessarily released version. those tests should be running continuously - similarly like a lot of other tests we have by building packages from current sources and installing those packages from local folders rather than from PyPI. We should also have back-compat tests of course, but installing task-sdk from local sources should be happening as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225425808
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
+
+
+def print_diagnostics(compose, compose_version, docker_version):
+ """Print diagnostic information when test fails."""
+ console.print("[red]=== DIAGNOSTIC INFORMATION ===[/]")
+ console.print(f"Docker version: {docker_version}")
+ console.print(f"Docker Compose version: {compose_version}")
+ console.print("\n[yellow]Container Status:[/]")
+ try:
+ containers = compose.compose.ps()
+ for container in containers:
+ console.print(f" {container.name}: {container.state}")
+ except Exception as e:
+ console.print(f" Error getting container status: {e}")
+
+ console.print("\n[yellow]Container Logs:[/]")
+ try:
+ logs = compose.compose.logs()
+ console.print(logs)
+ except Exception as e:
+ console.print(f" Error getting logs: {e}")
+
+
+def test_task_sdk_health(tmp_path_factory, monkeypatch):
+ """Test Task SDK health check using docker-compose environment."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-task-sdk-test")
+ console.print(f"[yellow]Tests are run in {tmp_dir}")
+
+ # Copy docker-compose.yaml to temp directory
+ docker_compose_file = TASK_SDK_TESTS_ROOT / "docker" / "docker-compose.yaml"
Review Comment:
Cool that makes sense too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225431767
##########
task-sdk-tests/tests/task_sdk_tests/conftest.py:
##########
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from task_sdk_tests.constants import DOCKER_IMAGE
+
+
+@pytest.fixture
+def default_docker_image():
Review Comment:
Addressed in 1215209d22
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
Review Comment:
Addressed in 1215209d22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3112426362
This is how a sample run would look like:
```
(airflow) ➜ airflow git:(docker-compose-task-sdk-integration) ✗ breeze testing task-sdk-integration-tests
Good version of Docker: 28.2.2.
Good version of docker-compose: 2.36.2
Executable permissions on entrypoints are OK
Running task SDK integration tests with PROD image: ghcr.io/apache/airflow/main/prod/python3.10
Using Task SDK version: 1.1.0
warning: The package `pandas==2.1.4` does not have an extra named `postgres`
Built apache-airflow-task-sdk @ file:///Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk
Built apache-airflow-task-sdk-tests @ file:///Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk-tests
Built apache-airflow-core @ file:///Users/amoghdesai/Documents/OSS/repos/airflow/airflow-core
Built apache-airflow-devel-common @ file:///Users/amoghdesai/Documents/OSS/repos/airflow/devel-common
Built apache-airflow-providers-common-sql @ file:///Users/amoghdesai/Documents/OSS/repos/airflow/providers/common/sql
Built apache-airflow-providers-standard @ file:///Users/amoghdesai/Documents/OSS/repos/airflow/providers/standard
Built apache-airflow @ file:///Users/amoghdesai/Documents/OSS/repos/airflow
Uninstalled 8 packages in 101ms
Installed 8 packages in 34ms
Installing apache-airflow-task-sdk==1.1.0 via pytest_sessionstart...
Installing from: /Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk
Installing to current UV environment...
Current Python: /Users/amoghdesai/Documents/OSS/repos/airflow/.venv/bin/python3
Running command: uv pip install /Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk
Using Python 3.12.9 environment at: /Users/amoghdesai/Documents/OSS/repos/airflow/.venv
Resolved 120 packages in 278ms
Built apache-airflow-task-sdk @ file:///Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk
Prepared 1 package in 231ms
Uninstalled 1 package in 2ms
Installed 1 package in 4ms
~ apache-airflow-task-sdk==1.1.0 (from file:///Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk)
Task SDK installed successfully to UV environment via pytest_sessionstart!
Verifying task Task installation via pytest_sessionstart...
✅ Task SDK import successful via pytest_sessionstart
========================================================================== test session starts ==========================================================================
platform darwin -- Python 3.12.9, pytest-8.4.1, pluggy-1.6.0 -- /Users/amoghdesai/Documents/OSS/repos/airflow/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk-tests
configfile: pyproject.toml
plugins: unordered-0.7.0, instafail-0.5.0, anyio-4.9.0, timeouts-1.2.1, xdist-3.8.0, time-machine-2.16.0, custom-exit-code-0.3.0, icdiff-0.9, rerunfailures-15.1, cov-6.2.1, kgb-7.2, mock-3.14.1, asyncio-0.25.0, requests-mock-1.12.1
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 1 item
tests/task_sdk_tests/test_task_sdk_health.py::test_task_sdk_health Tests are run in /private/var/folders/ww/wv3dswrd0hj2jymcq20l70nc0000gn/T/pytest-of-amoghdesai/pytest-134/airflow-task-sdk-test0
WARN[0000] The "AIRFLOW_UID" variable is not set. Defaulting to a blank string.
WARN[0000] The "AIRFLOW_UID" variable is not set. Defaulting to a blank string.
[+] Running 4/5
[+] Running 4/5low-task-sdk-test0_default Created 0.1s
[+] Running 4/5low-task-sdk-test0_default Created 0.1s
[+] Running 5/5low-task-sdk-test0_default Created 0.1s
✔ Network airflow-task-sdk-test0_default Created 0.1s
✔ Container airflow-task-sdk-test0-postgres-1 Healthy 29.1s
✔ Container airflow-task-sdk-test0-airflow-init-1 Exited 29.1s
✔ Container airflow-task-sdk-test0-airflow-apiserver-1 Healthy 44.0s
✔ Container airflow-task-sdk-test0-airflow-scheduler-1 Healthy 60.5s
Docker compose started for task SDK test
✅ Task SDK client imported successfully!
Making health check request...
[2025-07-24T13:17:48.861+0530] {_client.py:1026} INFO - HTTP Request: GET http://localhost:8080/execution/health/ping "HTTP/1.1 200 OK"
======================== Health Check Response =========================
Status Code: 200
Response: {'ok': ['airflow.api_fastapi.auth.tokens.JWTValidator'], 'failing': {}}
========================================================================
Docker compose instance deleted
PASSED
===================================================================== 1 passed in 66.43s (0:01:06) ======================================================================
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225400192
##########
task-sdk-tests/temp_healthcheck.txt:
##########
@@ -0,0 +1 @@
+ test: ["CMD-SHELL", "pgrep -f scheduler"]
Review Comment:
It isn't. Got added by mistake. handling it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3117282684
And ... Merged!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2230767547
##########
dev/breeze/doc/05_test_commands.rst:
##########
@@ -317,6 +317,24 @@ through ``breeze testing docker-compose-tests`` command.
The docker-compose tests are in ``docker-tests/`` folder in the main repo.
+Running task-sdk integration tests
+..................................
+
+You can use Breeze to run the task-sdk integration tests. Those tests are run using Production image
+and they are running test with the Quick-start docker compose we have. Some scenarios of task-sdk tests are run here.
+
+.. image:: ./images/output_testing_docker-compose-tests.svg
+ :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_testing_task-sdk-integration-tests.svg
+ :width: 100%
+ :alt: Breeze testing task-sdk-integration-tests
+
+You can also iterate over those tests with pytest command, but - unlike regular unit tests and
+Helm tests, they need to be run in local virtual environment. They also require to have
+``DOCKER_IMAGE`` environment variable set, pointing to the image to test if you do not run them
+through ``breeze testing task-sdk-integration-tests`` command.
Review Comment:
its been updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225431254
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
Review Comment:
Made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2227659844
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ environment:
+ &airflow-common-env
+ AIRFLOW__CORE__EXECUTOR: LocalExecutor
+ AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
+ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ AIRFLOW__CORE__FERNET_KEY: ''
+ AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
+ AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'
+ user: "${AIRFLOW_UID:-50000}:0"
+ depends_on:
+ &airflow-common-depends-on
+ postgres:
+ condition: service_healthy
+
+services:
+ postgres:
+ image: postgres:13
+ environment:
+ POSTGRES_USER: airflow
+ POSTGRES_PASSWORD: airflow
+ POSTGRES_DB: airflow
+ healthcheck:
+ test: ["CMD", "pg_isready", "-U", "airflow"]
+ interval: 10s
+ retries: 5
+ start_period: 5s
+ restart: always
+
+ airflow-init:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Creating missing opt dirs if missing:"
+ mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
+ echo "Airflow version:"
+ /entrypoint airflow version
+ echo "Running airflow config list to create default config file if missing."
+ /entrypoint airflow config list >/dev/null
+ echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID}:0"
+ chown -R "${AIRFLOW_UID}:0" /opt/airflow/
+ environment:
+ <<: *airflow-common-env
+ _AIRFLOW_DB_MIGRATE: 'true'
+ _AIRFLOW_WWW_USER_CREATE: 'true'
+ _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+ _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ _PIP_ADDITIONAL_REQUIREMENTS: ''
+ user: "0:0"
+ depends_on:
+ <<: *airflow-common-depends-on
+
+ airflow-apiserver:
+ <<: *airflow-common
+ command: api-server
+ ports:
+ - "8080:8080"
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version"]
+ interval: 30s
+ timeout: 10s
+ retries: 5
+ start_period: 30s
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-scheduler:
Review Comment:
Not as of now, testing the interfaces not the dags per-say.
If there is a need for future test cases, will add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2219748697
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
+
+
+def print_diagnostics(compose, compose_version, docker_version):
+ """Print diagnostic information when test fails."""
+ console.print("[red]=== DIAGNOSTIC INFORMATION ===[/]")
+ console.print(f"Docker version: {docker_version}")
+ console.print(f"Docker Compose version: {compose_version}")
+ console.print("\n[yellow]Container Status:[/]")
+ try:
+ containers = compose.compose.ps()
+ for container in containers:
+ console.print(f" {container.name}: {container.state}")
+ except Exception as e:
+ console.print(f" Error getting container status: {e}")
+
+ console.print("\n[yellow]Container Logs:[/]")
+ try:
+ logs = compose.compose.logs()
+ console.print(logs)
+ except Exception as e:
+ console.print(f" Error getting logs: {e}")
+
+
+def test_task_sdk_health(tmp_path_factory, monkeypatch):
+ """Test Task SDK health check using docker-compose environment."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-task-sdk-test")
+ console.print(f"[yellow]Tests are run in {tmp_dir}")
+
+ # Copy docker-compose.yaml to temp directory
+ docker_compose_file = TASK_SDK_TESTS_ROOT / "docker" / "docker-compose.yaml"
Review Comment:
Would making the `docker_compose_file` constant more consistent with other usage?
##########
task-sdk-tests/tests/task_sdk_tests/conftest.py:
##########
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from task_sdk_tests.constants import DOCKER_IMAGE
+
+
+@pytest.fixture
+def default_docker_image():
Review Comment:
It seems the `default_docker_image` fixture will never be used and the further test can just import from the `task_sdk_tests.constants`.
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
Review Comment:
Would placing the `TASK_SDK_TESTS_ROOT` variable in `task_sdk_tests.constants` more suitable ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2220510295
##########
task-sdk-tests/tests/task_sdk_tests/test_task_sdk_health.py:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+from pathlib import Path
+from shutil import copyfile
+
+from python_on_whales import DockerClient, docker
+from rich.console import Console
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, TASK_SDK_API_VERSION, TASK_SDK_HOST_PORT
+
+console = Console(width=400, color_system="standard")
+
+TASK_SDK_TESTS_ROOT = Path(__file__).parent.parent.parent
+
+
+def print_diagnostics(compose, compose_version, docker_version):
+ """Print diagnostic information when test fails."""
+ console.print("[red]=== DIAGNOSTIC INFORMATION ===[/]")
+ console.print(f"Docker version: {docker_version}")
+ console.print(f"Docker Compose version: {compose_version}")
+ console.print("\n[yellow]Container Status:[/]")
+ try:
+ containers = compose.compose.ps()
+ for container in containers:
+ console.print(f" {container.name}: {container.state}")
+ except Exception as e:
+ console.print(f" Error getting container status: {e}")
+
+ console.print("\n[yellow]Container Logs:[/]")
+ try:
+ logs = compose.compose.logs()
+ console.print(logs)
+ except Exception as e:
+ console.print(f" Error getting logs: {e}")
+
+
+def test_task_sdk_health(tmp_path_factory, monkeypatch):
+ """Test Task SDK health check using docker-compose environment."""
+ tmp_dir = tmp_path_factory.mktemp("airflow-task-sdk-test")
+ console.print(f"[yellow]Tests are run in {tmp_dir}")
+
+ # Copy docker-compose.yaml to temp directory
+ docker_compose_file = TASK_SDK_TESTS_ROOT / "docker" / "docker-compose.yaml"
+ tmp_docker_compose_file = tmp_dir / "docker-compose.yaml"
+ copyfile(docker_compose_file, tmp_docker_compose_file)
+
+ # Set environment variables for the test
+ monkeypatch.setenv("AIRFLOW_IMAGE_NAME", os.environ.get("DOCKER_IMAGE", "apache/airflow:3.0.3"))
Review Comment:
Should not we use DOCKER_IMAGE from constants here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3102632196
Closing and reopening to run full tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed pull request #53332: 🚧 Task SDK Integration Testing Infrastructure
URL: https://github.com/apache/airflow/pull/53332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ⚠️ WIP: POC of task sdk integration tests (DO NOT MERGE) [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh closed pull request #53332: ⚠️ WIP: POC of task sdk integration tests (DO NOT MERGE)
URL: https://github.com/apache/airflow/pull/53332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3092394803
👀 :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2223642741
##########
task-sdk-tests/tests/task_sdk_tests/conftest.py:
##########
@@ -0,0 +1,88 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+
+import pytest
+
+from task_sdk_tests.constants import AIRFLOW_ROOT_PATH, DOCKER_IMAGE
+
+
+@pytest.fixture
+def default_docker_image():
+ return DOCKER_IMAGE
+
+
+@pytest.fixture(scope="session")
+def install_task_sdk():
Review Comment:
i think its better to use pytest test `pytest_sessionstart` hook, this way we can make sure our setup fully ready to run tests. otherwise install_task_sdk should be called on the first test that it starts running, when we add more tests and if any order changes pytest test execution it may cause some errors.
https://docs.pytest.org/en/7.1.x/reference/reference.html#pytest.hookspec.pytest_sessionstart
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225401103
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
Review Comment:
Yeah, handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225401843
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
Review Comment:
Handled in 3d7c880b2fa3fdf620474f3dc701d75bc8032f2b
##########
task-sdk-tests/temp_healthcheck.txt:
##########
@@ -0,0 +1 @@
+ test: ["CMD-SHELL", "pgrep -f scheduler"]
Review Comment:
Handled in 3d7c880b2fa3fdf620474f3dc701d75bc8032f2b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2230766717
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ environment:
+ &airflow-common-env
+ AIRFLOW__CORE__EXECUTOR: LocalExecutor
+ AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
+ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ AIRFLOW__CORE__FERNET_KEY: ''
+ AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
+ AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'
+ user: "${AIRFLOW_UID:-50000}:0"
+ depends_on:
+ &airflow-common-depends-on
+ postgres:
+ condition: service_healthy
+
+services:
+ postgres:
+ image: postgres:13
+ environment:
+ POSTGRES_USER: airflow
+ POSTGRES_PASSWORD: airflow
+ POSTGRES_DB: airflow
+ healthcheck:
+ test: ["CMD", "pg_isready", "-U", "airflow"]
+ interval: 10s
+ retries: 5
+ start_period: 5s
+ restart: always
+
+ airflow-init:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Creating missing opt dirs if missing:"
+ mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
+ echo "Airflow version:"
+ /entrypoint airflow version
+ echo "Running airflow config list to create default config file if missing."
+ /entrypoint airflow config list >/dev/null
+ echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID}:0"
+ chown -R "${AIRFLOW_UID}:0" /opt/airflow/
+ environment:
+ <<: *airflow-common-env
+ _AIRFLOW_DB_MIGRATE: 'true'
+ _AIRFLOW_WWW_USER_CREATE: 'true'
+ _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+ _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ _PIP_ADDITIONAL_REQUIREMENTS: ''
+ user: "0:0"
+ depends_on:
+ <<: *airflow-common-depends-on
+
+ airflow-apiserver:
+ <<: *airflow-common
+ command: api-server
+ ports:
+ - "8080:8080"
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version"]
+ interval: 30s
+ timeout: 10s
+ retries: 5
+ start_period: 30s
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-scheduler:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Installing Task SDK version: ${TASK_SDK_VERSION:-1.1.0}"
+ pip install --no-deps apache-airflow-task-sdk==${TASK_SDK_VERSION:-1.1.0}
Review Comment:
Undid that part, resolving
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2230765892
##########
task-sdk-tests/docker/docker-compose.yaml:
##########
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+x-airflow-common:
+ &airflow-common
+ image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.0.3}
+ environment:
+ &airflow-common-env
+ AIRFLOW__CORE__EXECUTOR: LocalExecutor
+ AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
+ AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+ AIRFLOW__CORE__FERNET_KEY: ''
+ AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
+ AIRFLOW__CORE__EXECUTION_API_SERVER_URL: 'http://airflow-apiserver:8080/execution/'
+ user: "${AIRFLOW_UID:-50000}:0"
+ depends_on:
+ &airflow-common-depends-on
+ postgres:
+ condition: service_healthy
+
+services:
+ postgres:
+ image: postgres:13
+ environment:
+ POSTGRES_USER: airflow
+ POSTGRES_PASSWORD: airflow
+ POSTGRES_DB: airflow
+ healthcheck:
+ test: ["CMD", "pg_isready", "-U", "airflow"]
+ interval: 10s
+ retries: 5
+ start_period: 5s
+ restart: always
+
+ airflow-init:
+ <<: *airflow-common
+ entrypoint: /bin/bash
+ command:
+ - -c
+ - |
+ echo "Creating missing opt dirs if missing:"
+ mkdir -v -p /opt/airflow/{logs,dags,plugins,config}
+ echo "Airflow version:"
+ /entrypoint airflow version
+ echo "Running airflow config list to create default config file if missing."
+ /entrypoint airflow config list >/dev/null
+ echo "Change ownership of files in /opt/airflow to ${AIRFLOW_UID}:0"
+ chown -R "${AIRFLOW_UID}:0" /opt/airflow/
+ environment:
+ <<: *airflow-common-env
+ _AIRFLOW_DB_MIGRATE: 'true'
+ _AIRFLOW_WWW_USER_CREATE: 'true'
+ _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+ _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+ _PIP_ADDITIONAL_REQUIREMENTS: ''
+ user: "0:0"
+ depends_on:
+ <<: *airflow-common-depends-on
+
+ airflow-apiserver:
+ <<: *airflow-common
+ command: api-server
+ ports:
+ - "8080:8080"
+ healthcheck:
+ test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version"]
+ interval: 30s
+ timeout: 10s
+ retries: 5
+ start_period: 30s
+ restart: always
+ depends_on:
+ <<: *airflow-common-depends-on
+ airflow-init:
+ condition: service_completed_successfully
+
+ airflow-scheduler:
Review Comment:
Resolving it, will come back in a follow up as needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2225415012
##########
task-sdk-tests/tests/task_sdk_tests/conftest.py:
##########
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import pytest
+
+from task_sdk_tests.constants import DOCKER_IMAGE
+
+
+@pytest.fixture
+def default_docker_image():
Review Comment:
Good catch. Cleaned up the fixture and imported directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #53332:
URL: https://github.com/apache/airflow/pull/53332#issuecomment-3112734533
This is great @amoghrajesh , will take a look later today!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2228124384
##########
dev/breeze/doc/05_test_commands.rst:
##########
@@ -317,6 +317,25 @@ through ``breeze testing docker-compose-tests`` command.
The docker-compose tests are in ``docker-tests/`` folder in the main repo.
+Running task-sdk integration tests
Review Comment:
This is cool for reference, but we need a "howto" type of document. I think we also need one short docment describing the reasoning for this kind of testing and typical use patterns on how to run it locally - not only with `breeze testing`. As we know `breeze testing` is really to autoamate execution and provide CI reproducibility - but in many cases we want to let the contributors know how they can run the local testing - with command line in local venv and IDEs. And this is a new type of tests so it should be added to https://github.com/apache/airflow/blob/main/contributing-docs/09_testing.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] 🚧 Task SDK Integration Testing Infrastructure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #53332:
URL: https://github.com/apache/airflow/pull/53332#discussion_r2228127482
##########
dev/breeze/doc/05_test_commands.rst:
##########
@@ -317,6 +317,25 @@ through ``breeze testing docker-compose-tests`` command.
The docker-compose tests are in ``docker-tests/`` folder in the main repo.
+Running task-sdk integration tests
Review Comment:
BTW. Those images above could be part of the docs ... And I am thinking we should have similar pictures for other types of testing !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org