You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Andisha2004 (via GitHub)" <gi...@apache.org> on 2026/05/07 23:05:06 UTC
[PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Andisha2004 opened a new pull request, #66576:
URL: https://github.com/apache/airflow/pull/66576
## What this PR does
This PR adds Athena Spark calculation support to the Amazon provider.
Specifically, it:
- extends `AthenaHook` with Athena Spark calculation helpers
- adds `AthenaSparkOperator` for submitting and waiting on Athena Spark calculations
- adds `AthenaSparkSensor` for monitoring an existing calculation execution
- adds unit tests for the new hook/operator/sensor behavior
- adds provider documentation for Athena Spark usage
## Why this is needed
The Amazon provider already supports Athena query execution, but it does not currently expose first-class support for Athena Spark calculation APIs. This PR fills that gap by adding provider-native support for Athena Spark job submission and monitoring.
## Implementation notes
- `AthenaHook` wraps Athena Spark APIs including:
- `start_calculation_execution`
- `get_calculation_execution`
- `stop_calculation_execution`
- `AthenaSparkOperator` submits a calculation, polls until a terminal state, and returns execution metadata
- `AthenaSparkSensor` polls an existing calculation execution ID until completion or failure
- the implementation is intentionally scoped to provider-layer functionality only
## Tests
I added or updated unit tests for:
- `providers/amazon/tests/unit/amazon/aws/hooks/test_athena.py`
- `providers/amazon/tests/unit/amazon/aws/operators/test_athena_spark.py`
- `providers/amazon/tests/unit/amazon/aws/sensors/test_athena_spark.py`
## Backward compatibility
This PR adds new functionality and does not change the existing Athena query operator or sensor behavior.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: OpenAI Codex following the guidelines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3207771856
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
+ self.log.info("Starting Athena Spark calculation in session %s", self.session_id)
+
+ calculation_execution_id = self.hook.start_calculation(
+ session_id=self.session_id,
+ code_block=self.code_block,
+ description=self.description,
+ client_request_token=self.client_request_token,
+ )
+ self._calculation_execution_id = calculation_execution_id
+ initial_state = self.hook.check_calculation_status(calculation_execution_id)
+ self.log.info(
+ "Calculation submitted. CalculationExecutionId: %s, initial state: %s",
+ calculation_execution_id,
+ initial_state,
+ )
+
+ if initial_state and initial_state in AthenaHook.SPARK_TERMINAL_STATES:
+ return self._handle_terminal_state(calculation_execution_id, initial_state)
+
+ final_state = self._poll_until_terminal(calculation_execution_id)
+ return self._handle_terminal_state(calculation_execution_id, final_state)
+
+ def _poll_until_terminal(self, calculation_execution_id: str) -> str:
+ """Poll calculation status until a terminal state or timeout."""
+ for attempt in range(1, self.max_polling_attempts + 1):
+ if attempt > 1:
+ time.sleep(self.poll_interval)
+ state = self.hook.check_calculation_status(calculation_execution_id)
+
+ if state is None:
+ raise AirflowException(
+ f"Malformed or missing status for calculation {calculation_execution_id}. "
+ "Cannot continue polling."
+ )
+
+ self.log.info(
+ "CalculationExecutionId: %s, current state: %s (attempt %d/%d)",
+ calculation_execution_id,
+ state,
+ attempt,
+ self.max_polling_attempts,
+ )
+
+ if state in AthenaHook.SPARK_TERMINAL_STATES:
+ return state
+
+ raise AirflowException(
Review Comment:
We need to avoid introducing `AirflowException` in the provider layer. I would suggest using `RuntimeError`. Please check the rest of the diff and implement the same feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210784141
##########
providers/amazon/tests/unit/amazon/aws/hooks/test_athena.py:
##########
@@ -139,6 +172,78 @@ def test_run_query_hook_lineage(self, mock_conn, mock_send_lineage):
assert call_kw["sql"] == MOCK_DATA["query"]
assert call_kw["job_id"] == MOCK_DATA["query_execution_id"]
+ # new test cases
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_default_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_with_optional_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ calculation_configuration = {"CodeBlock": MOCK_DATA["code_block"]}
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ description=MOCK_DATA["description"],
+ calculation_configuration=calculation_configuration,
+ client_request_token=MOCK_DATA["client_request_token"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ "Description": MOCK_DATA["description"],
+ "CalculationConfiguration": calculation_configuration,
+ "ClientRequestToken": MOCK_DATA["client_request_token"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
Review Comment:
Since there is ambiguity regarding the shape of the response object received for `get_calculation_info`, shouldn't there also be a test validating that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3230204822
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
Review Comment:
The majority of the Amazon provider uses `max_attempts`, please use that throughout this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66576:
URL: https://github.com/apache/airflow/pull/66576#issuecomment-4548535139
@Andisha2004 — There are 1 unresolved review thread(s) on this PR from `o-nikolas`. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? When you believe the feedback is addressed, please mark the threads as resolved and ping the reviewer (`o-nikolas`) for a final look. Thanks!
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
---
Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "Andisha2004 (via GitHub)" <gi...@apache.org>.
Andisha2004 commented on PR #66576:
URL: https://github.com/apache/airflow/pull/66576#issuecomment-4401826897
I ran the relevant provider unit tests and Python lint/format checks locally. I used --no-verify for the local commit because the blacken-docs hook environment on this machine was resolving against Python 3.9 even after reinstalling prek, which appears to be a local tooling issue rather than a code issue in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66576:
URL: https://github.com/apache/airflow/pull/66576#issuecomment-4657820814
@Andisha2004 Converting to **draft** — this PR does not yet meet our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
- :x: **Pre-commit / static checks**. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
- :x: **Unit tests**. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/09_testing.rst).
- :x: **mypy (type checking)**. See [docs](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst).
See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is **not** a rejection — just an invitation to bring the PR up to standard. No rush.
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210784141
##########
providers/amazon/tests/unit/amazon/aws/hooks/test_athena.py:
##########
@@ -139,6 +172,78 @@ def test_run_query_hook_lineage(self, mock_conn, mock_send_lineage):
assert call_kw["sql"] == MOCK_DATA["query"]
assert call_kw["job_id"] == MOCK_DATA["query_execution_id"]
+ # new test cases
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_default_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_with_optional_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ calculation_configuration = {"CodeBlock": MOCK_DATA["code_block"]}
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ description=MOCK_DATA["description"],
+ calculation_configuration=calculation_configuration,
+ client_request_token=MOCK_DATA["client_request_token"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ "Description": MOCK_DATA["description"],
+ "CalculationConfiguration": calculation_configuration,
+ "ClientRequestToken": MOCK_DATA["client_request_token"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
Review Comment:
Since there is ambiguity regarding the shape of the response object received for `get_calculation_info`, shouldn't there also be a test validating that? Also, since you there is an option to use caching, I would suggest that you add a test or set of tests covering that behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210714532
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
+ self.log.info("Starting Athena Spark calculation in session %s", self.session_id)
+
+ calculation_execution_id = self.hook.start_calculation(
+ session_id=self.session_id,
+ code_block=self.code_block,
+ description=self.description,
+ client_request_token=self.client_request_token,
+ )
+ self._calculation_execution_id = calculation_execution_id
+ initial_state = self.hook.check_calculation_status(calculation_execution_id)
+ self.log.info(
+ "Calculation submitted. CalculationExecutionId: %s, initial state: %s",
+ calculation_execution_id,
+ initial_state,
+ )
+
+ if initial_state and initial_state in AthenaHook.SPARK_TERMINAL_STATES:
+ return self._handle_terminal_state(calculation_execution_id, initial_state)
+
+ final_state = self._poll_until_terminal(calculation_execution_id)
+ return self._handle_terminal_state(calculation_execution_id, final_state)
+
+ def _poll_until_terminal(self, calculation_execution_id: str) -> str:
+ """Poll calculation status until a terminal state or timeout."""
+ for attempt in range(1, self.max_polling_attempts + 1):
+ if attempt > 1:
+ time.sleep(self.poll_interval)
+ state = self.hook.check_calculation_status(calculation_execution_id)
+
+ if state is None:
+ raise AirflowException(
+ f"Malformed or missing status for calculation {calculation_execution_id}. "
+ "Cannot continue polling."
+ )
+
+ self.log.info(
+ "CalculationExecutionId: %s, current state: %s (attempt %d/%d)",
+ calculation_execution_id,
+ state,
+ attempt,
+ self.max_polling_attempts,
+ )
+
+ if state in AthenaHook.SPARK_TERMINAL_STATES:
+ return state
+
+ raise AirflowException(
+ f"Polling timed out after {self.max_polling_attempts} attempts for calculation "
+ f"{calculation_execution_id}. Use execution_timeout or increase max_polling_attempts."
+ )
+
+ def _handle_terminal_state(self, calculation_execution_id: str, state: str) -> dict[str, Any]:
+ """Resolve terminal state: raise on failure/cancel, build and return metadata."""
+ reason = self.hook.get_calculation_state_change_reason(calculation_execution_id)
+ execution_info = self.hook.get_calculation_info(calculation_execution_id)
+ status = (
+ execution_info.get("Status")
+ or (execution_info.get("CalculationExecution") or {}).get("Status")
+ or {}
+ )
+ submission_time = status.get("SubmissionDateTime")
+ completion_time = status.get("CompletionDateTime")
+ workgroup = (
+ execution_info.get("WorkGroup")
+ or (execution_info.get("CalculationExecution") or {}).get("WorkGroup")
+ or (execution_info.get("CalculationExecution") or {}).get("Workgroup")
+ )
+ output_location = (
+ execution_info.get("OutputLocation")
+ or (execution_info.get("CalculationExecution") or {}).get("OutputLocation")
+ or (execution_info.get("ResultConfiguration") or {}).get("OutputLocation")
+ )
+
+ result = {
+ "calculation_execution_id": calculation_execution_id,
+ "state": state,
+ "state_change_reason": reason,
+ "submission_time": str(submission_time) if submission_time else None,
+ "completion_time": str(completion_time) if completion_time else None,
+ "session_id": self.session_id,
+ "workgroup": workgroup,
+ "output_location": output_location,
+ }
+
+ if state in AthenaHook.SPARK_FAILURE_STATES:
+ self.log.error(
+ "Calculation failed. CalculationExecutionId: %s, state: %s, reason: %s",
+ calculation_execution_id,
+ state,
+ reason,
+ )
+ raise AirflowException(
+ f"Athena Spark calculation ended in {state}. "
+ f"CalculationExecutionId: {calculation_execution_id}. "
+ f"Reason: {reason or 'No reason provided.'}"
+ )
+
+ if state != "COMPLETED":
+ raise AirflowException(
+ f"Unexpected terminal state: {state} for calculation {calculation_execution_id}. "
+ "Expected COMPLETED, FAILED, or CANCELED."
+ )
+
+ self.log.info(
+ "Calculation completed successfully. CalculationExecutionId: %s",
+ calculation_execution_id,
+ )
+ return result
+
+ def on_kill(self) -> None:
+ """Request cancellation of the calculation when the task is killed."""
+ if self._calculation_execution_id:
+ self.log.info("Received kill signal; stopping calculation %s", self._calculation_execution_id)
+ try:
+ self.hook.stop_calculation(self._calculation_execution_id)
+ except Exception as e:
+ self.log.warning(
+ "Failed to stop calculation %s: %s",
+ self._calculation_execution_id,
+ e,
Review Comment:
This would better as it preserves debugging information:
```
except Exception:
self.log.warning(
"Failed to stop calculation %s",
self._calculation_execution_id,
exc_info=True,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210811547
##########
providers/amazon/tests/unit/amazon/aws/sensors/test_athena_spark.py:
##########
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from unittest import mock
+
+import pytest
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.sensors.athena_spark import AthenaSparkSensor
+
+CALC_ID = "calc-exec-123"
+
+
+@pytest.fixture
+def sensor() -> AthenaSparkSensor:
+ return AthenaSparkSensor(task_id="test_athena_spark_sensor", calculation_execution_id=CALC_ID)
+
+
+class TestAthenaSparkSensor:
+ def test_init(self, sensor: AthenaSparkSensor):
+ assert sensor.calculation_execution_id == CALC_ID
+ assert sensor.aws_conn_id == "aws_default"
+
+ def test_template_fields(self):
+ assert AthenaSparkSensor.template_fields == ("calculation_execution_id",)
+
+ @mock.patch.object(AthenaHook, "check_calculation_status", return_value="COMPLETED")
+ def test_poke_completed_returns_true(self, mock_check: mock.Mock, sensor: AthenaSparkSensor):
+ result = sensor.poke({})
+ assert result is True
+ mock_check.assert_called_once_with(CALC_ID)
+
+ @mock.patch.object(AthenaHook, "check_calculation_status", return_value="FAILED")
+ def test_poke_failed_raises(self, mock_check: mock.Mock, sensor: AthenaSparkSensor):
+ with pytest.raises(AirflowException, match="failed with state: FAILED"):
+ sensor.poke({})
+ mock_check.assert_called_once_with(CALC_ID)
+
+ @mock.patch.object(AthenaHook, "check_calculation_status", return_value="RUNNING")
+ def test_poke_running_returns_false(self, mock_check: mock.Mock, sensor: AthenaSparkSensor):
+ result = sensor.poke({})
+ assert result is False
+ mock_check.assert_called_once_with(CALC_ID)
Review Comment:
These tests should be fine once you move response normalization down to the hook. There is one small gap that would still remain: there does not appear to be coverage for the `Unexpected terminal state` branch in `_handle_terminal_state()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66576:
URL: https://github.com/apache/airflow/pull/66576#issuecomment-4529575866
@Andisha2004 A few things need addressing before review — see our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
- :x: **CI fails**: `CI image checks / Static checks` (and possibly other checks — see the [Checks tab](https://github.com/apache/airflow/pull/66576/checks) for the full list).
No rush.
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
---
Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210722332
##########
providers/amazon/src/airflow/providers/amazon/aws/sensors/athena_spark.py:
##########
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.sensors.base import BaseSensorOperator
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+
+class AthenaSparkSensor(BaseSensorOperator):
+ """
+ Poll the status of an AWS Athena Spark calculation until it reaches a terminal state.
+
+ :param calculation_execution_id: The ID of the calculation to monitor. (templated)
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ """
+
+ template_fields: Sequence[str] = ("calculation_execution_id",)
+ ui_color = "#44e2b5"
+
+ def __init__(
+ self,
+ *,
+ calculation_execution_id: str,
+ aws_conn_id: str = "aws_default",
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.calculation_execution_id = calculation_execution_id
+ self.aws_conn_id = aws_conn_id
+
+ def poke(self, context: Context) -> bool:
+ """Check the current status of the Spark calculation."""
+ del context
Review Comment:
No need for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #66576: Amazon: Add Athena Spark operator and sensor support
URL: https://github.com/apache/airflow/pull/66576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3207553090
##########
airflow-core/newsfragments/66576.feature.rst:
##########
@@ -0,0 +1 @@
+Add Athena Spark operator and sensor support to the Amazon provider.
Review Comment:
Provider changes do not require newsfragments.
##########
providers/amazon/docs/operators/athena/athena_spark.rst:
##########
@@ -0,0 +1,73 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Athena Spark Operators
+======================
+
+Amazon Athena supports Apache Spark calculations through session-based APIs.
+This page documents the provider support for submitting and monitoring those
+calculations from Airflow.
+
+AthenaSparkOperator
+-------------------
+
+Use :class:`~airflow.providers.amazon.aws.operators.athena_spark.AthenaSparkOperator`
+to submit Spark code to an existing Athena session and wait until the calculation
+reaches a terminal state.
+
+.. code-block:: python
+
+ from airflow import DAG
+ from airflow.providers.amazon.aws.operators.athena_spark import AthenaSparkOperator
+ from datetime import datetime
+
+ with DAG(
+ dag_id="example_athena_spark_operator",
+ start_date=datetime(2024, 1, 1),
+ schedule=None,
+ catchup=False,
+ ) as dag:
+ AthenaSparkOperator(
+ task_id="run_spark_code",
+ session_id="my-athena-session-id",
+ code_block="print('hello from athena spark')",
+ poll_interval=10,
+ max_polling_attempts=60,
+ )
+
+AthenaSparkSensor
+-----------------
+
+Use :class:`~airflow.providers.amazon.aws.sensors.athena_spark.AthenaSparkSensor`
+to wait for an existing calculation execution ID.
+
+.. code-block:: python
+
+ from airflow import DAG
+ from airflow.providers.amazon.aws.sensors.athena_spark import AthenaSparkSensor
+ from datetime import datetime
+
+ with DAG(
+ dag_id="example_athena_spark_sensor",
+ start_date=datetime(2024, 1, 1),
+ schedule=None,
+ catchup=False,
+ ) as dag:
+ AthenaSparkSensor(
+ task_id="wait_for_spark_calculation",
+ calculation_execution_id="calc-exec-123",
+ )
Review Comment:
So there are a few issues with this file:
1. You are inlining DAG code using these new operators/sensors. Standard practice in the provider docs is usually to add tasks using the new operators/sensors to the example DAG(s) and reference them via `exampleinclude` blocks instead.
2. Where is the `Prerequisite Tasks` section?
3. The structure of the Operators/Sensors sections looks inconsistent with the existing Amazon provider docs. For example, if you look at `athena_sql.rst`, there is a top-level `Operators` section followed by use-case-oriented subsections such as `Execute a SQL query`, along with explanatory text describing when/how the operators should be used. I think the same structure should be followed here for both operators and sensors.
4. The page is also missing some contextual guidance for users. For example:
- whether an Athena Spark session must already exist
- which AWS connection/authentication is expected
- when to use `AthenaSparkOperator` vs `AthenaSparkSensor`
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/athena.py:
##########
@@ -344,3 +344,135 @@ def stop_query(self, query_execution_id: str) -> dict:
"""
self.log.info("Stopping Query with executionId - %s", query_execution_id)
return self.get_conn().stop_query_execution(QueryExecutionId=query_execution_id)
+
+ # --- Athena Spark (Calculations) API ---
Review Comment:
Remove this.
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
+ self.log.info("Starting Athena Spark calculation in session %s", self.session_id)
+
+ calculation_execution_id = self.hook.start_calculation(
+ session_id=self.session_id,
+ code_block=self.code_block,
+ description=self.description,
+ client_request_token=self.client_request_token,
+ )
+ self._calculation_execution_id = calculation_execution_id
+ initial_state = self.hook.check_calculation_status(calculation_execution_id)
+ self.log.info(
+ "Calculation submitted. CalculationExecutionId: %s, initial state: %s",
+ calculation_execution_id,
+ initial_state,
+ )
+
+ if initial_state and initial_state in AthenaHook.SPARK_TERMINAL_STATES:
+ return self._handle_terminal_state(calculation_execution_id, initial_state)
+
+ final_state = self._poll_until_terminal(calculation_execution_id)
+ return self._handle_terminal_state(calculation_execution_id, final_state)
+
+ def _poll_until_terminal(self, calculation_execution_id: str) -> str:
+ """Poll calculation status until a terminal state or timeout."""
+ for attempt in range(1, self.max_polling_attempts + 1):
+ if attempt > 1:
+ time.sleep(self.poll_interval)
+ state = self.hook.check_calculation_status(calculation_execution_id)
+
+ if state is None:
+ raise AirflowException(
+ f"Malformed or missing status for calculation {calculation_execution_id}. "
+ "Cannot continue polling."
+ )
+
+ self.log.info(
+ "CalculationExecutionId: %s, current state: %s (attempt %d/%d)",
+ calculation_execution_id,
+ state,
+ attempt,
+ self.max_polling_attempts,
+ )
+
+ if state in AthenaHook.SPARK_TERMINAL_STATES:
+ return state
+
+ raise AirflowException(
+ f"Polling timed out after {self.max_polling_attempts} attempts for calculation "
+ f"{calculation_execution_id}. Use execution_timeout or increase max_polling_attempts."
+ )
+
+ def _handle_terminal_state(self, calculation_execution_id: str, state: str) -> dict[str, Any]:
+ """Resolve terminal state: raise on failure/cancel, build and return metadata."""
+ reason = self.hook.get_calculation_state_change_reason(calculation_execution_id)
+ execution_info = self.hook.get_calculation_info(calculation_execution_id)
+ status = (
+ execution_info.get("Status")
+ or (execution_info.get("CalculationExecution") or {}).get("Status")
+ or {}
+ )
+ submission_time = status.get("SubmissionDateTime")
+ completion_time = status.get("CompletionDateTime")
+ workgroup = (
+ execution_info.get("WorkGroup")
+ or (execution_info.get("CalculationExecution") or {}).get("WorkGroup")
+ or (execution_info.get("CalculationExecution") or {}).get("Workgroup")
+ )
+ output_location = (
+ execution_info.get("OutputLocation")
+ or (execution_info.get("CalculationExecution") or {}).get("OutputLocation")
+ or (execution_info.get("ResultConfiguration") or {}).get("OutputLocation")
+ )
Review Comment:
It looks like you are compensating for inconsistent response structures coming from the hook. If you need to normalize, why not push it down to the hook itself?
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
+ self.log.info("Starting Athena Spark calculation in session %s", self.session_id)
+
+ calculation_execution_id = self.hook.start_calculation(
+ session_id=self.session_id,
+ code_block=self.code_block,
+ description=self.description,
+ client_request_token=self.client_request_token,
+ )
+ self._calculation_execution_id = calculation_execution_id
+ initial_state = self.hook.check_calculation_status(calculation_execution_id)
+ self.log.info(
+ "Calculation submitted. CalculationExecutionId: %s, initial state: %s",
+ calculation_execution_id,
+ initial_state,
+ )
+
+ if initial_state and initial_state in AthenaHook.SPARK_TERMINAL_STATES:
+ return self._handle_terminal_state(calculation_execution_id, initial_state)
+
+ final_state = self._poll_until_terminal(calculation_execution_id)
+ return self._handle_terminal_state(calculation_execution_id, final_state)
+
+ def _poll_until_terminal(self, calculation_execution_id: str) -> str:
+ """Poll calculation status until a terminal state or timeout."""
+ for attempt in range(1, self.max_polling_attempts + 1):
+ if attempt > 1:
+ time.sleep(self.poll_interval)
Review Comment:
Instead of checking for the iteration count, maybe it would be cleaner to move `time.sleep` to the end of the loop?
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/athena.py:
##########
@@ -344,3 +344,135 @@ def stop_query(self, query_execution_id: str) -> dict:
"""
self.log.info("Stopping Query with executionId - %s", query_execution_id)
return self.get_conn().stop_query_execution(QueryExecutionId=query_execution_id)
+
+ # --- Athena Spark (Calculations) API ---
+
+ SPARK_INTERMEDIATE_STATES = (
+ "CREATING",
+ "CREATED",
+ "QUEUED",
+ "RUNNING",
+ "CANCELING",
+ )
+ SPARK_FAILURE_STATES = (
+ "FAILED",
+ "CANCELED",
+ )
+ SPARK_SUCCESS_STATES = ("COMPLETED",)
+ SPARK_TERMINAL_STATES = (
+ "COMPLETED",
+ "FAILED",
+ "CANCELED",
+ )
Review Comment:
Better to do this to avoid drift:
`SPARK_TERMINAL_STATES = SPARK_SUCCESS_STATES + SPARK_FAILURE_STATES`
Also, I would move these constants above the hook constructor near the existing hook constants.
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
+ self.log.info("Starting Athena Spark calculation in session %s", self.session_id)
+
+ calculation_execution_id = self.hook.start_calculation(
+ session_id=self.session_id,
+ code_block=self.code_block,
+ description=self.description,
+ client_request_token=self.client_request_token,
+ )
+ self._calculation_execution_id = calculation_execution_id
+ initial_state = self.hook.check_calculation_status(calculation_execution_id)
+ self.log.info(
+ "Calculation submitted. CalculationExecutionId: %s, initial state: %s",
+ calculation_execution_id,
+ initial_state,
+ )
+
+ if initial_state and initial_state in AthenaHook.SPARK_TERMINAL_STATES:
+ return self._handle_terminal_state(calculation_execution_id, initial_state)
+
+ final_state = self._poll_until_terminal(calculation_execution_id)
+ return self._handle_terminal_state(calculation_execution_id, final_state)
+
+ def _poll_until_terminal(self, calculation_execution_id: str) -> str:
+ """Poll calculation status until a terminal state or timeout."""
+ for attempt in range(1, self.max_polling_attempts + 1):
+ if attempt > 1:
+ time.sleep(self.poll_interval)
+ state = self.hook.check_calculation_status(calculation_execution_id)
+
+ if state is None:
+ raise AirflowException(
+ f"Malformed or missing status for calculation {calculation_execution_id}. "
+ "Cannot continue polling."
+ )
+
+ self.log.info(
+ "CalculationExecutionId: %s, current state: %s (attempt %d/%d)",
+ calculation_execution_id,
+ state,
+ attempt,
+ self.max_polling_attempts,
+ )
+
+ if state in AthenaHook.SPARK_TERMINAL_STATES:
+ return state
+
+ raise AirflowException(
Review Comment:
We need to avoid introducing `AirflowException` in the provider layer. I would suggest using `RuntimeError`.
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/athena.py:
##########
@@ -344,3 +344,135 @@ def stop_query(self, query_execution_id: str) -> dict:
"""
self.log.info("Stopping Query with executionId - %s", query_execution_id)
return self.get_conn().stop_query_execution(QueryExecutionId=query_execution_id)
+
+ # --- Athena Spark (Calculations) API ---
+
+ SPARK_INTERMEDIATE_STATES = (
+ "CREATING",
+ "CREATED",
+ "QUEUED",
+ "RUNNING",
+ "CANCELING",
+ )
+ SPARK_FAILURE_STATES = (
+ "FAILED",
+ "CANCELED",
+ )
+ SPARK_SUCCESS_STATES = ("COMPLETED",)
+ SPARK_TERMINAL_STATES = (
+ "COMPLETED",
+ "FAILED",
+ "CANCELED",
+ )
+
+ def start_calculation(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ calculation_configuration: dict[str, Any] | None = None,
+ client_request_token: str | None = None,
+ ) -> str:
+ """
+ Start an Athena Spark calculation execution.
+
+ .. seealso::
+ - :external+boto3:py:meth:`Athena.Client.start_calculation_execution`
+
+ :param session_id: The Athena session ID.
+ :param code_block: Spark code to execute (typically notebook-like code).
+ :param description: Optional description of the calculation.
+ :param calculation_configuration: Contains configuration information for the calculation.
+ :param client_request_token: Optional idempotency token.
+ :return: CalculationExecutionId
+ """
+ params: dict[str, Any] = {
+ "SessionId": session_id,
+ "CodeBlock": code_block,
+ }
+ if description:
+ params["Description"] = description
+
+ if calculation_configuration:
+ params["CalculationConfiguration"] = calculation_configuration
+
+ if client_request_token:
+ params["ClientRequestToken"] = client_request_token
+
+ if self.log_query:
+ self.log.info("Starting CalculationExecution with params:\n%s", query_params_to_string(params))
+ response = self.get_conn().start_calculation_execution(**params)
+ calc_execution_id = response["CalculationExecutionId"]
+ self.log.info("Calculation execution id: %s", calc_execution_id)
+ return calc_execution_id
+
+ def get_calculation_info(self, calculation_execution_id: str, use_cache: bool = False) -> dict[str, Any]:
+ """
+ Get information about a single execution of a calculation.
+
+ .. seealso::
+ - :external+boto3:py:meth:`Athena.Client.get_calculation_execution`
+
+ :param calculation_execution_id: CalculationExecutionId returned by start_calculation
+ :param use_cache: If True, use execution information cache
+ """
+ cache_key = f"calc:{calculation_execution_id}"
+ if use_cache and cache_key in self.__query_results:
+ return self.__query_results[cache_key]
+
+ response = self.get_conn().get_calculation_execution(CalculationExecutionId=calculation_execution_id)
+
+ if use_cache:
+ self.__query_results[cache_key] = response
+ return response
+
+ def check_calculation_status(self, calculation_execution_id: str, use_cache: bool = False) -> str | None:
+ """
+ Fetch the state of a submitted calculation execution.
+
+ .. seealso::
+ - :external+boto3:py:meth:`Athena.Client.get_calculation_execution`
+
+ :param calculation_execution_id: CalculationExecutionId returned by start_calculation
+ :return: One of valid calculation states, or *None* if the response is malformed.
+ """
+ response = self.get_calculation_info(
+ calculation_execution_id=calculation_execution_id, use_cache=use_cache
+ )
+
+ try:
+ return response["Status"]["State"]
+ except KeyError:
+ self.log.error("Could not parse status for calculation %s", calculation_execution_id)
+ return None
+
+ def get_calculation_state_change_reason(
+ self, calculation_execution_id: str, use_cache: bool = False
+ ) -> str | None:
+ """
+ Fetch the reason for a calculation state change (e.g. error message).
+
+ :param calculation_execution_id: CalculationExecutionId returned by start_calculation
+ :param use_cache: If True, use execution information cache
+ :return: State change reason string, or None.
+ """
+ response = self.get_calculation_info(
+ calculation_execution_id=calculation_execution_id, use_cache=use_cache
+ )
+ try:
+ return response["CalculationExecution"]["Status"].get("StateChangeReason")
Review Comment:
Why is it ["CalculationExecution"]["Status"]["StateChangeReason"] here but response["Status"]["State"] above?
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/athena.py:
##########
@@ -344,3 +344,135 @@ def stop_query(self, query_execution_id: str) -> dict:
"""
self.log.info("Stopping Query with executionId - %s", query_execution_id)
return self.get_conn().stop_query_execution(QueryExecutionId=query_execution_id)
+
+ # --- Athena Spark (Calculations) API ---
+
+ SPARK_INTERMEDIATE_STATES = (
+ "CREATING",
+ "CREATED",
+ "QUEUED",
+ "RUNNING",
+ "CANCELING",
+ )
+ SPARK_FAILURE_STATES = (
+ "FAILED",
+ "CANCELED",
+ )
+ SPARK_SUCCESS_STATES = ("COMPLETED",)
+ SPARK_TERMINAL_STATES = (
+ "COMPLETED",
+ "FAILED",
+ "CANCELED",
+ )
+
+ def start_calculation(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ calculation_configuration: dict[str, Any] | None = None,
+ client_request_token: str | None = None,
+ ) -> str:
+ """
+ Start an Athena Spark calculation execution.
+
+ .. seealso::
+ - :external+boto3:py:meth:`Athena.Client.start_calculation_execution`
+
+ :param session_id: The Athena session ID.
+ :param code_block: Spark code to execute (typically notebook-like code).
+ :param description: Optional description of the calculation.
+ :param calculation_configuration: Contains configuration information for the calculation.
+ :param client_request_token: Optional idempotency token.
+ :return: CalculationExecutionId
Review Comment:
Shouldn't this be:
`:return: str`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210680122
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
+ self.log.info("Starting Athena Spark calculation in session %s", self.session_id)
+
+ calculation_execution_id = self.hook.start_calculation(
+ session_id=self.session_id,
+ code_block=self.code_block,
+ description=self.description,
+ client_request_token=self.client_request_token,
+ )
+ self._calculation_execution_id = calculation_execution_id
+ initial_state = self.hook.check_calculation_status(calculation_execution_id)
+ self.log.info(
+ "Calculation submitted. CalculationExecutionId: %s, initial state: %s",
+ calculation_execution_id,
+ initial_state,
+ )
+
+ if initial_state and initial_state in AthenaHook.SPARK_TERMINAL_STATES:
+ return self._handle_terminal_state(calculation_execution_id, initial_state)
Review Comment:
Why is this needed when you have lines 128-129 to poll for and then handle the terminal state? I would remove it unless you there is a strong justification that I cannot see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210650041
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
Review Comment:
Why is `del context` needed here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210784141
##########
providers/amazon/tests/unit/amazon/aws/hooks/test_athena.py:
##########
@@ -139,6 +172,78 @@ def test_run_query_hook_lineage(self, mock_conn, mock_send_lineage):
assert call_kw["sql"] == MOCK_DATA["query"]
assert call_kw["job_id"] == MOCK_DATA["query_execution_id"]
+ # new test cases
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_default_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_with_optional_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ calculation_configuration = {"CodeBlock": MOCK_DATA["code_block"]}
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ description=MOCK_DATA["description"],
+ calculation_configuration=calculation_configuration,
+ client_request_token=MOCK_DATA["client_request_token"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ "Description": MOCK_DATA["description"],
+ "CalculationConfiguration": calculation_configuration,
+ "ClientRequestToken": MOCK_DATA["client_request_token"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
Review Comment:
Since there is ambiguity regarding the shape of the response object received for `get_calculation_info`, shouldn't there also be a test validating that? Also, since there is an option to use caching, I would suggest that you add a test or set of tests covering that behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210708157
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/athena_spark.py:
##########
@@ -0,0 +1,229 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+AthenaSparkOperator for running Apache Spark calculations in Amazon Athena.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
+from airflow.providers.common.compat.sdk import AirflowException
+
+if TYPE_CHECKING:
+ from airflow.sdk import Context
+
+
+class AthenaSparkOperator(AwsBaseOperator[AthenaHook]):
+ """
+ Run an Apache Spark calculation in an Amazon Athena session.
+
+ Submits a calculation (e.g. PySpark code) via the Athena API, polls until
+ the calculation reaches a terminal state (COMPLETED, FAILED, or CANCELED),
+ and returns execution metadata.
+
+ .. seealso::
+ - :class:`airflow.providers.amazon.aws.hooks.athena.AthenaHook`
+ - `Athena for Apache Spark
+ <https://docs.aws.amazon.com/athena/latest/ug/notebooks-spark-api-list.html>`__
+
+ :param session_id: The Athena session ID in which to run the calculation. (templated)
+ :param code_block: The calculation code (e.g. PySpark) to execute. (templated)
+ :param description: Optional description of the calculation.
+ :param client_request_token: Optional idempotency token for the submission.
+ :param poll_interval: Seconds to wait between status checks. Default 30.
+ :param max_polling_attempts: Maximum number of polling attempts before timing out.
+ To limit total task time, use execution_timeout on the task as well.
+ :param log_query: Whether to log submission details. Default True.
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ :param region_name: AWS region. If not set, default boto3 behaviour is used.
+ :param verify: Whether to verify SSL certificates.
+ :param botocore_config: Optional botocore configuration dict.
+ """
+
+ aws_hook_class = AthenaHook
+ ui_color = "#44b5e2"
+ template_fields: Sequence[str] = aws_template_fields("session_id", "code_block", "description")
+ template_ext: Sequence[str] = (".py",)
+ template_fields_renderers = {"code_block": "python"}
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ code_block: str,
+ description: str | None = None,
+ client_request_token: str | None = None,
+ poll_interval: int = 30,
+ max_polling_attempts: int = 120,
+ log_query: bool = True,
+ aws_conn_id: str | None = "aws_default",
+ region_name: str | None = None,
+ verify: bool | str | None = None,
+ botocore_config: dict | None = None,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ aws_conn_id=aws_conn_id,
+ region_name=region_name,
+ verify=verify,
+ botocore_config=botocore_config,
+ **kwargs,
+ )
+ self.session_id = session_id
+ self.code_block = code_block
+ self.description = description
+ self.client_request_token = client_request_token
+ self.poll_interval = poll_interval
+ self.max_polling_attempts = max_polling_attempts
+ self.log_query = log_query
+ self._calculation_execution_id: str | None = None
+
+ @property
+ def _hook_parameters(self) -> dict[str, Any]:
+ return {**super()._hook_parameters, "log_query": self.log_query}
+
+ def execute(self, context: Context) -> dict[str, Any]:
+ """Submit the Spark calculation, poll until terminal state, then return metadata."""
+ del context
+ self.log.info("Starting Athena Spark calculation in session %s", self.session_id)
+
+ calculation_execution_id = self.hook.start_calculation(
+ session_id=self.session_id,
+ code_block=self.code_block,
+ description=self.description,
+ client_request_token=self.client_request_token,
+ )
+ self._calculation_execution_id = calculation_execution_id
+ initial_state = self.hook.check_calculation_status(calculation_execution_id)
+ self.log.info(
+ "Calculation submitted. CalculationExecutionId: %s, initial state: %s",
+ calculation_execution_id,
+ initial_state,
+ )
+
+ if initial_state and initial_state in AthenaHook.SPARK_TERMINAL_STATES:
+ return self._handle_terminal_state(calculation_execution_id, initial_state)
+
+ final_state = self._poll_until_terminal(calculation_execution_id)
+ return self._handle_terminal_state(calculation_execution_id, final_state)
+
+ def _poll_until_terminal(self, calculation_execution_id: str) -> str:
+ """Poll calculation status until a terminal state or timeout."""
+ for attempt in range(1, self.max_polling_attempts + 1):
+ if attempt > 1:
+ time.sleep(self.poll_interval)
+ state = self.hook.check_calculation_status(calculation_execution_id)
+
+ if state is None:
+ raise AirflowException(
+ f"Malformed or missing status for calculation {calculation_execution_id}. "
+ "Cannot continue polling."
+ )
+
+ self.log.info(
+ "CalculationExecutionId: %s, current state: %s (attempt %d/%d)",
+ calculation_execution_id,
+ state,
+ attempt,
+ self.max_polling_attempts,
+ )
+
+ if state in AthenaHook.SPARK_TERMINAL_STATES:
+ return state
+
+ raise AirflowException(
+ f"Polling timed out after {self.max_polling_attempts} attempts for calculation "
+ f"{calculation_execution_id}. Use execution_timeout or increase max_polling_attempts."
+ )
+
+ def _handle_terminal_state(self, calculation_execution_id: str, state: str) -> dict[str, Any]:
+ """Resolve terminal state: raise on failure/cancel, build and return metadata."""
+ reason = self.hook.get_calculation_state_change_reason(calculation_execution_id)
+ execution_info = self.hook.get_calculation_info(calculation_execution_id)
+ status = (
+ execution_info.get("Status")
+ or (execution_info.get("CalculationExecution") or {}).get("Status")
+ or {}
+ )
+ submission_time = status.get("SubmissionDateTime")
+ completion_time = status.get("CompletionDateTime")
+ workgroup = (
+ execution_info.get("WorkGroup")
+ or (execution_info.get("CalculationExecution") or {}).get("WorkGroup")
+ or (execution_info.get("CalculationExecution") or {}).get("Workgroup")
+ )
+ output_location = (
+ execution_info.get("OutputLocation")
+ or (execution_info.get("CalculationExecution") or {}).get("OutputLocation")
+ or (execution_info.get("ResultConfiguration") or {}).get("OutputLocation")
+ )
+
+ result = {
+ "calculation_execution_id": calculation_execution_id,
+ "state": state,
+ "state_change_reason": reason,
+ "submission_time": str(submission_time) if submission_time else None,
+ "completion_time": str(completion_time) if completion_time else None,
+ "session_id": self.session_id,
+ "workgroup": workgroup,
+ "output_location": output_location,
+ }
+
+ if state in AthenaHook.SPARK_FAILURE_STATES:
+ self.log.error(
+ "Calculation failed. CalculationExecutionId: %s, state: %s, reason: %s",
+ calculation_execution_id,
+ state,
+ reason,
+ )
+ raise AirflowException(
+ f"Athena Spark calculation ended in {state}. "
+ f"CalculationExecutionId: {calculation_execution_id}. "
+ f"Reason: {reason or 'No reason provided.'}"
+ )
+
+ if state != "COMPLETED":
Review Comment:
Why are you using a string here for state checks when you have a constant called `SPARK_SUCCESS_STATES` ? It would be better to do:
`if state not in AthenaHook.SPARK_SUCCESS_STATES:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210757436
##########
providers/amazon/tests/unit/amazon/aws/hooks/test_athena.py:
##########
@@ -139,6 +172,78 @@ def test_run_query_hook_lineage(self, mock_conn, mock_send_lineage):
assert call_kw["sql"] == MOCK_DATA["query"]
assert call_kw["job_id"] == MOCK_DATA["query_execution_id"]
+ # new test cases
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_default_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_start_calculation_with_optional_params(self, mock_conn):
+ mock_conn.return_value.start_calculation_execution.return_value = MOCK_CALCULATION_EXECUTION
+
+ calculation_configuration = {"CodeBlock": MOCK_DATA["code_block"]}
+
+ result = self.athena.start_calculation(
+ session_id=MOCK_DATA["session_id"],
+ code_block=MOCK_DATA["code_block"],
+ description=MOCK_DATA["description"],
+ calculation_configuration=calculation_configuration,
+ client_request_token=MOCK_DATA["client_request_token"],
+ )
+
+ expected_call_params = {
+ "SessionId": MOCK_DATA["session_id"],
+ "CodeBlock": MOCK_DATA["code_block"],
+ "Description": MOCK_DATA["description"],
+ "CalculationConfiguration": calculation_configuration,
+ "ClientRequestToken": MOCK_DATA["client_request_token"],
+ }
+ mock_conn.return_value.start_calculation_execution.assert_called_with(**expected_call_params)
+ assert result == MOCK_DATA["calculation_execution_id"]
+
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_hook_get_calculation_info(self, mock_conn):
+ mock_conn.return_value.get_calculation_execution.return_value = MOCK_SUCCEEDED_CALC_EXECUTION
+
+ result = self.athena.get_calculation_info(
+ calculation_execution_id=MOCK_DATA["calculation_execution_id"]
+ )
+
+ mock_conn.return_value.get_calculation_execution.assert_called_once_with(
+ CalculationExecutionId=MOCK_DATA["calculation_execution_id"]
+ )
+ assert result == MOCK_SUCCEEDED_CALC_EXECUTION
+
+ @mock.patch.object(AthenaHook, "get_conn")
+ def test_check_calculation_status_normal(self, mock_conn):
+ mock_conn.return_value.get_calculation_execution.return_value = MOCK_RUNNING_CALC_EXECUTION
+
+ state = self.athena.check_calculation_status(
+ calculation_execution_id=MOCK_DATA["calculation_execution_id"]
+ )
+
+ assert state == "RUNNING"
Review Comment:
What about when the API returns a `None` or empty state? Have you covered that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #66576:
URL: https://github.com/apache/airflow/pull/66576#issuecomment-4401810778
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210720605
##########
providers/amazon/src/airflow/providers/amazon/aws/sensors/athena_spark.py:
##########
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.athena import AthenaHook
+from airflow.sensors.base import BaseSensorOperator
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+
+class AthenaSparkSensor(BaseSensorOperator):
+ """
+ Poll the status of an AWS Athena Spark calculation until it reaches a terminal state.
+
+ :param calculation_execution_id: The ID of the calculation to monitor. (templated)
+ :param aws_conn_id: The Airflow connection used for AWS credentials.
+ """
+
+ template_fields: Sequence[str] = ("calculation_execution_id",)
+ ui_color = "#44e2b5"
+
+ def __init__(
+ self,
+ *,
+ calculation_execution_id: str,
+ aws_conn_id: str = "aws_default",
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.calculation_execution_id = calculation_execution_id
+ self.aws_conn_id = aws_conn_id
+
+ def poke(self, context: Context) -> bool:
+ """Check the current status of the Spark calculation."""
+ del context
+ hook = AthenaHook(aws_conn_id=self.aws_conn_id)
+ state = hook.check_calculation_status(self.calculation_execution_id)
+
+ self.log.info("Calculation %s state is: %s", self.calculation_execution_id, state)
+
+ if state in hook.SPARK_FAILURE_STATES:
+ raise AirflowException(f"Calculation {self.calculation_execution_id} failed with state: {state}")
+
+ return state == "COMPLETED"
Review Comment:
Use the constant `AthenaHook.SPARK_SUCCESS_STATES`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #66576:
URL: https://github.com/apache/airflow/pull/66576#discussion_r3210730696
##########
providers/amazon/tests/unit/amazon/aws/hooks/test_athena.py:
##########
@@ -14,11 +14,21 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+"""
+Unit tests for AthenaHook with boto3 client mocked.
+
+Test strategy:
+- Mock the boto3 client via AthenaHook.get_conn() so no real AWS calls are made.
+- Cover success, failure (exceptions), and bad/edge-case input for each hook method.
+- Use botocore.exceptions.ClientError for API failure scenarios.
+"""
Review Comment:
Why is this docstring here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66576:
URL: https://github.com/apache/airflow/pull/66576#issuecomment-4476519138
@Andisha2004 — There are 20 unresolved review thread(s) on this PR from @SameerMesiah97, @o-nikolas, and you have engaged with each one (post-review commits and/or in-thread replies). Could you confirm whether you believe the feedback is fully addressed and the PR is ready for maintainer review confirmation?
If yes, reply here (a short "yes / ready" is fine) and an Apache Airflow maintainer will pick the PR up from the review queue on the next sweep.
If you are still working on a thread, please reply with what is outstanding so the threads stay unresolved on purpose.
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Amazon: Add Athena Spark operator and sensor support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #66576:
URL: https://github.com/apache/airflow/pull/66576#issuecomment-4746445796
Closing as a stale triaged draft — no update in ~9 days since the triage note. No worries: reopen this or open a fresh PR whenever you're ready to continue. Thanks for the contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org