You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "omar-massfih (via GitHub)" <gi...@apache.org> on 2026/10/12 15:12:20 UTC
[PR] fix(azure): handle enhanced Power BI refresh execution status [airflow]
omar-massfih opened a new pull request, #74600:
URL: https://github.com/apache/airflow/pull/74600
## What
Handle enhanced Power BI refresh execution statuses in the existing hook and trigger result path. Standard refreshes continue using history polling.
## Why
History `Unknown` is normalized to `In Progress`. Without reading enhanced execution details, polling cannot use a terminal `extendedStatus` or expose engine and attempt-level error details. The regression responses are synthetic; no live history/execution response pair is claimed.
Closes #74597
## How
For the matching `ViaEnhancedApi` history entry, fetch execution details using the submitted refresh ID and the existing authenticated adapter. Prefer meaningful `extendedStatus`, fall back to general `status`, recognize completion and terminal failures, and retain bounded polling for active or unknown states. Include engine messages and service exceptions in failure output without treating messages as a terminal signal.
Add hook and trigger regression coverage for status precedence, failure details, ongoing polling, timeout, cancellation, and standard-refresh behavior. Submission, deferral, and the operator result interface are reused; no dependency changes.
Validation: all 63 Power BI hook, trigger, and operator tests pass in Breeze. The failure regression fails against the original hook and passes with this change. Regular upstream prek checks and `git diff --check` pass. The broader manual type checks are still running; the full downstream CI matrix has not been run locally.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Codex (GPT-6)
Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(azure): detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "omar-massfih (via GitHub)" <gi...@apache.org>.
omar-massfih commented on code in PR #74600:
URL: https://github.com/apache/airflow/pull/74600#discussion_r4270011278
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -187,6 +195,39 @@ async def get_refresh_details_by_refresh_id(
refresh_details = refresh_histories[refresh_ids.index(refresh_id)]
+ if refresh_details.get(PowerBIDatasetRefreshFields.REFRESH_TYPE.value) == "ViaEnhancedApi":
+ try:
+ execution_details = await self.run(
+ url="myorg/groups/{group_id}/datasets/{dataset_id}/refreshes/{refresh_id}",
+ path_parameters={
+ "group_id": group_id,
+ "dataset_id": dataset_id,
+ "refresh_id": refresh_id,
+ },
+ )
+ except AirflowException as error:
+ raise PowerBIDatasetRefreshException(
+ "Failed to retrieve refresh execution details"
+ ) from error
+ status = execution_details.get("extendedStatus")
+ if status not in PowerBIDatasetRefreshStatus.TERMINAL_STATUSES | {"NotStarted", "InProgress"}:
Review Comment:
Added `ACTIVE_EXTENDED_STATUSES` and normalized `NotStarted`, `InProgress`, and `Unknown` to `In Progress`. Extended-status precedence is preserved. Timeout tests cover all three values; trigger event fields and XCom keys are unchanged.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -187,6 +195,39 @@ async def get_refresh_details_by_refresh_id(
refresh_details = refresh_histories[refresh_ids.index(refresh_id)]
+ if refresh_details.get(PowerBIDatasetRefreshFields.REFRESH_TYPE.value) == "ViaEnhancedApi":
+ try:
+ execution_details = await self.run(
+ url="myorg/groups/{group_id}/datasets/{dataset_id}/refreshes/{refresh_id}",
+ path_parameters={
+ "group_id": group_id,
+ "dataset_id": dataset_id,
+ "refresh_id": refresh_id,
+ },
+ )
+ except AirflowException as error:
+ raise PowerBIDatasetRefreshException(
+ "Failed to retrieve refresh execution details"
+ ) from error
+ status = execution_details.get("extendedStatus")
+ if status not in PowerBIDatasetRefreshStatus.TERMINAL_STATUSES | {"NotStarted", "InProgress"}:
+ status = execution_details.get("status")
+ errors = [message["message"] for message in execution_details.get("messages", [])]
Review Comment:
The parser now skips missing, null, and empty message text and excludes warnings. Untyped messages and top-level and attempt-level service exceptions are preserved. Messages never determine terminal status.
The active-to-completed regression includes a message without text and verifies cancellation is never awaited. All three active-status cases failed before the fix and now pass.
The supplied live pair returned HTTP 200 and `Completed` from both endpoints, validating normal completion only. The stuck-history condition remains unreproduced; failure and divergence coverage is synthetic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(azure): detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "omar-massfih (via GitHub)" <gi...@apache.org>.
omar-massfih commented on code in PR #74600:
URL: https://github.com/apache/airflow/pull/74600#discussion_r4270009474
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -42,9 +43,11 @@ class PowerBIDatasetRefreshStatus:
FAILED = "Failed"
COMPLETED = "Completed"
DISABLED = "Disabled"
+ TIMED_OUT = "TimedOut"
+ CANCELLED = "Cancelled"
- TERMINAL_STATUSES = {FAILED, COMPLETED}
- FAILURE_STATUSES = {FAILED, DISABLED}
+ FAILURE_STATUSES = {FAILED, DISABLED, TIMED_OUT, CANCELLED}
+ TERMINAL_STATUSES = FAILURE_STATUSES | {COMPLETED}
Review Comment:
Restored the original ordering: `TERMINAL_STATUSES` comes first, and `FAILURE_STATUSES` is derived by subtracting `Completed`.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -159,6 +164,9 @@ def raw_to_refresh_details(cls, refresh_details: dict) -> dict[str, str]:
),
PowerBIDatasetRefreshFields.ERROR.value: str(refresh_details.get("serviceExceptionJson")),
}
+ if refresh_details.get("refreshType") == "ViaEnhancedApi":
Review Comment:
Added `PowerBIHook.VIA_ENHANCED_API`. Normalization now always returns `refresh_type`, including `"None"` when absent, and execution results retain the history refresh type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla merged PR #74600:
URL: https://github.com/apache/airflow/pull/74600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on PR #74600:
URL: https://github.com/apache/airflow/pull/74600#issuecomment-6402267859
> @dabla Hi, I'm not able to merge the PR.
Let's await a full successful build first, I just rebased the PR to make sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #74600:
URL: https://github.com/apache/airflow/pull/74600#discussion_r4307174502
##########
providers/microsoft/azure/tests/unit/microsoft/azure/hooks/test_powerbi.py:
##########
@@ -56,6 +61,184 @@
class TestPowerBIHook:
+ @pytest.mark.asyncio
+ @mock.patch.object(KiotaRequestAdapterHook, "run", autospec=True)
+ async def test_completed_enhanced_refresh_observation(self, mock_run, powerbi_hook):
+ fixture = json.loads(
Review Comment:
**[nit]** Load the fixture with the shared fixture loader.
The other tests that read from `resources/` (for example `hooks/test_msgraph.py`) use `load_json_from_resources` from `tests_common.test_utils.file_loading`. That helper also opens the file as UTF-8. `Path.read_text()` with no `encoding` argument uses the locale's encoding instead ([ruff PLW1514](https://docs.astral.sh/ruff/rules/unspecified-encoding/)):
```python
from os.path import dirname
from tests_common.test_utils.file_loading import load_json_from_resources
fixture = load_json_from_resources(dirname(__file__), "..", "resources", "powerbi_completed_refresh.json")
```
With this change, the `json` and `Path` imports can also go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #74600:
URL: https://github.com/apache/airflow/pull/74600#discussion_r4315612111
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -187,6 +196,47 @@ async def get_refresh_details_by_refresh_id(
refresh_details = refresh_histories[refresh_ids.index(refresh_id)]
+ if refresh_details.get(PowerBIDatasetRefreshFields.REFRESH_TYPE.value) == self.VIA_ENHANCED_API:
Review Comment:
**[warning]** After the rebase on the durable operator (#74675), the retry decision and the trigger read the status of an enhanced refresh from different sources.
The line that needs to change, `PowerBIDatasetRefreshOperator.get_job_status` in `operators/powerbi.py`, is not part of this diff, so I'm commenting here, where the execution-details lookup starts.
With `durable=True`, a retry starts with `get_job_status`, and the `ResumableJobMixin` uses the status it returns to choose between three things: reconnect to an active refresh, skip a refresh that already succeeded, or submit a new one when the old one failed. `get_job_status` still scans `get_refresh_history` itself, so for an enhanced refresh it never sees the execution details this PR adds. The trigger does see them, through `get_refresh_details_by_refresh_id`.
Take the case this PR is built around: history stays `Unknown` while the execution details say `Failed`. A probe at this head (operator and hook against a mocked `run`) gives:
```
resume decision sees: 'In Progress' -> is_job_active=True
trigger sees: 'Failed'
```
So every durable retry reconnects to the failed refresh, the trigger reports `Failed` straight away, and the task fails again without ever submitting a new refresh. The mixin's "failed job → resubmit fresh" path is never reached, and all retries are used up on the same failed refresh. Before this PR the trigger also read only the history, so the same case waited until the timeout. The PR improves the trigger side but leaves the retry decision on the old source.
Reading the status from the same lookup as the trigger fixes it, and keeps the "not in history" case for refreshes Power BI has dropped:
```python
def get_job_status(self, external_id: JsonValue, context: Context) -> str:
refresh_histories = self.hook.run_sync(
self.hook.get_refresh_history(dataset_id=self.dataset_id, group_id=self.group_id)
)
if not any(
details[PowerBIDatasetRefreshFields.REQUEST_ID.value] == external_id for details in refresh_histories
):
return _REFRESH_NOT_IN_HISTORY
refresh_details = self.hook.run_sync(
self.hook.get_refresh_details_by_refresh_id(
dataset_id=self.dataset_id, group_id=self.group_id, refresh_id=str(external_id)
)
)
return refresh_details[PowerBIDatasetRefreshFields.STATUS.value]
```
The history is then fetched twice, but only once per retry, not on every poll. A failed execution-details request still raises, so the retry fails instead of submitting a second refresh while the first one may still be running. A durable test next to the existing `TaskStateStore` ones in `operators/test_powerbi.py` (stored ID, enhanced history `Unknown`, execution `Failed`, expect a new submit) would cover it.
This only matters if history really does stay `Unknown` after the refresh has ended, which the fixture README says wasn't reproduced. But that is the premise of the whole PR, so the retry decision should follow it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #74600:
URL: https://github.com/apache/airflow/pull/74600#issuecomment-6403926483
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #74600:
URL: https://github.com/apache/airflow/pull/74600#discussion_r4297198444
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -187,6 +196,47 @@ async def get_refresh_details_by_refresh_id(
refresh_details = refresh_histories[refresh_ids.index(refresh_id)]
+ if refresh_details.get(PowerBIDatasetRefreshFields.REFRESH_TYPE.value) == self.VIA_ENHANCED_API:
+ try:
+ execution_details = await self.run(
+ url="myorg/groups/{group_id}/datasets/{dataset_id}/refreshes/{refresh_id}",
+ path_parameters={
+ "group_id": group_id,
+ "dataset_id": dataset_id,
+ "refresh_id": refresh_id,
+ },
+ )
+ except AirflowException as error:
+ raise PowerBIDatasetRefreshException(
+ "Failed to retrieve refresh execution details"
+ ) from error
+ status = execution_details.get("extendedStatus")
+ if status not in (
+ PowerBIDatasetRefreshStatus.TERMINAL_STATUSES
+ | PowerBIDatasetRefreshStatus.ACTIVE_EXTENDED_STATUSES
+ ):
+ status = execution_details.get("status")
+ errors = [
+ message["message"]
+ for message in execution_details.get("messages", [])
Review Comment:
**[nit]** A `null` `messages` or `refreshAttempts` still ends the same way as the missing `message` key did.
`execution_details.get("messages", [])` falls back to `[]` only when the key is absent. When the response has `"messages": null`, `.get` returns `None`, and iterating over it raises `TypeError`. As with the earlier `KeyError`, that is not a `PowerBIDatasetRefreshException`, so it goes to `PowerBITrigger.run`'s `except Exception`, which cancels the running refresh. The `refreshAttempts` read at line 228 has the same problem.
The API reference doesn't say these arrays can be `null`, but guarding against it costs one `or` each:
```python
for message in execution_details.get("messages") or []
...
for attempt in execution_details.get("refreshAttempts") or []
```
A `"messages": None` case in `test_enhanced_refresh_optional_message_text` would cover it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(azure): handle enhanced Power BI refresh execution status [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #74600:
URL: https://github.com/apache/airflow/pull/74600#issuecomment-6127075181
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "omar-massfih (via GitHub)" <gi...@apache.org>.
omar-massfih commented on PR #74600:
URL: https://github.com/apache/airflow/pull/74600#issuecomment-6291259096
The two failures in the previous [Tests (AMD) run](https://github.com/apache/airflow/actions/runs/39399107440) on `d92cd599` were dependency setup failures:
- [Python 3.11 CI image build](https://github.com/apache/airflow/actions/runs/39399107440/job/118561502895)
- [Shared distributions tests](https://github.com/apache/airflow/actions/runs/39399107440/job/118561502951)
Both stopped because `uv` could not parse `uv.lock`: duplicate package `pyvespa==1.2.9`. Neither failure reached the Power BI tests.
The upstream repair in #75200 is already included in the current head, `54f87749`. With CI's pinned `uv==0.13.1`, `uv lock --check` passes, as do frozen dependency dry runs for shared serialization tests and the Python 3.11 Linux CI image. These validate dependency setup, not the full test suites.
The latest [Tests (AMD) run](https://github.com/apache/airflow/actions/runs/39421743463), CodeQL, and newsfragment workflows are marked `action_required`, with no jobs executed. Could a maintainer approve the latest workflow runs?
---
Drafted-by: Codex (GPT-6) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(azure): detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #74600:
URL: https://github.com/apache/airflow/pull/74600#discussion_r4264776845
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -42,9 +43,11 @@ class PowerBIDatasetRefreshStatus:
FAILED = "Failed"
COMPLETED = "Completed"
DISABLED = "Disabled"
+ TIMED_OUT = "TimedOut"
+ CANCELLED = "Cancelled"
- TERMINAL_STATUSES = {FAILED, COMPLETED}
- FAILURE_STATUSES = {FAILED, DISABLED}
+ FAILURE_STATUSES = {FAILED, DISABLED, TIMED_OUT, CANCELLED}
+ TERMINAL_STATUSES = FAILURE_STATUSES | {COMPLETED}
Review Comment:
**[nit]** Extending the status sets is right, but swapping their order makes the diff look like a bigger change than it is.
`TERMINAL_STATUSES` used to come before `FAILURE_STATUSES`. The diff now shows both lines removed and added again in the other order, so a reviewer has to compare them side by side to see what actually changed. I see the reason: `TERMINAL_STATUSES` is now built from `FAILURE_STATUSES`, so it has to come second. You can keep the original order and still define one set from the other by deriving in the opposite direction:
```python
TERMINAL_STATUSES = {FAILED, COMPLETED, DISABLED, TIMED_OUT, CANCELLED}
FAILURE_STATUSES = TERMINAL_STATUSES - {COMPLETED}
```
The diff then only touches the two set literals.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -159,6 +164,9 @@ def raw_to_refresh_details(cls, refresh_details: dict) -> dict[str, str]:
),
PowerBIDatasetRefreshFields.ERROR.value: str(refresh_details.get("serviceExceptionJson")),
}
+ if refresh_details.get("refreshType") == "ViaEnhancedApi":
Review Comment:
**[nit]** `"ViaEnhancedApi"` is spelled out three times, and `refresh_type` is only set for that one value.
`raw_to_refresh_details` writes the literal into the dict, and `get_refresh_details_by_refresh_id` compares against it again at line 198. This also changes what the public `get_refresh_history` returns, but only for enhanced entries: some dicts have a `refresh_type` key and others don't. Two suggestions:
- add a constant, e.g. `PowerBIDatasetRefreshType.VIA_ENHANCED_API` (or a class attribute next to the statuses), and use it at both places;
- always store `refresh_type` (as `str(refresh_details.get("refreshType"))`), so every entry from `get_refresh_history` has the same keys and the check at line 198 is a plain equality on a field that always exists.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -187,6 +195,39 @@ async def get_refresh_details_by_refresh_id(
refresh_details = refresh_histories[refresh_ids.index(refresh_id)]
+ if refresh_details.get(PowerBIDatasetRefreshFields.REFRESH_TYPE.value) == "ViaEnhancedApi":
+ try:
+ execution_details = await self.run(
+ url="myorg/groups/{group_id}/datasets/{dataset_id}/refreshes/{refresh_id}",
+ path_parameters={
+ "group_id": group_id,
+ "dataset_id": dataset_id,
+ "refresh_id": refresh_id,
+ },
+ )
+ except AirflowException as error:
+ raise PowerBIDatasetRefreshException(
+ "Failed to retrieve refresh execution details"
+ ) from error
+ status = execution_details.get("extendedStatus")
+ if status not in PowerBIDatasetRefreshStatus.TERMINAL_STATUSES | {"NotStarted", "InProgress"}:
+ status = execution_details.get("status")
+ errors = [message["message"] for message in execution_details.get("messages", [])]
Review Comment:
**[warning]** If a message has no `message` key, `message["message"]` raises `KeyError`, and the trigger then cancels a refresh that is still running.
In the [execution-details API](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/get-refresh-execution-details-in-group#enginemessage), the `EngineMessage` fields `code`, `message` and `type` are not marked as required. The PR also says messages are present while a refresh is still running or retrying (`test_enhanced_refresh_messages_do_not_end_polling`). A `KeyError` here is not a `PowerBIDatasetRefreshException`, so `fetch_refresh_status_and_error`'s tenacity retry does not catch it. It goes straight to `PowerBITrigger.run`'s `except Exception`, which calls `cancel_dataset_refresh` on the live refresh. So one malformed message cancels the user's refresh. Before this PR, nothing parsed this payload.
Reading it defensively avoids that, and is a good point to drop warnings from the error text too. The API documents `type` as `Error` or `Warning`, and right now a failed refresh's error message lists warnings next to the real cause:
```python
errors = [
message["message"]
for message in execution_details.get("messages", [])
if message.get("message") and message.get("type") != "Warning"
]
```
A test with a message that has no `message` key (and one with `type: "Warning"`) would cover both.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/powerbi.py:
##########
@@ -187,6 +195,39 @@ async def get_refresh_details_by_refresh_id(
refresh_details = refresh_histories[refresh_ids.index(refresh_id)]
+ if refresh_details.get(PowerBIDatasetRefreshFields.REFRESH_TYPE.value) == "ViaEnhancedApi":
+ try:
+ execution_details = await self.run(
+ url="myorg/groups/{group_id}/datasets/{dataset_id}/refreshes/{refresh_id}",
+ path_parameters={
+ "group_id": group_id,
+ "dataset_id": dataset_id,
+ "refresh_id": refresh_id,
+ },
+ )
+ except AirflowException as error:
+ raise PowerBIDatasetRefreshException(
+ "Failed to retrieve refresh execution details"
+ ) from error
+ status = execution_details.get("extendedStatus")
+ if status not in PowerBIDatasetRefreshStatus.TERMINAL_STATUSES | {"NotStarted", "InProgress"}:
Review Comment:
**[nit]** Enhanced refreshes report "running" as `InProgress` / `NotStarted`, standard ones as `In Progress`.
`raw_to_refresh_details` maps only `Unknown` to `PowerBIDatasetRefreshStatus.IN_PROGRESS`. The two active values this line keeps from `extendedStatus` pass through unchanged. They reach the trigger's log line and the timeout event's `dataset_refresh_status`, and the operator pushes that value to XCom as `<task_id>.powerbi_dataset_refresh_status`. So a downstream task reading that XCom after a timeout sees `InProgress` or `NotStarted` for an enhanced refresh and `In Progress` for a standard one. `test_enhanced_refresh_unknown_times_out` asserts `In Progress` only because its fixture uses `Unknown`.
Mapping `NotStarted` and `InProgress` to `IN_PROGRESS` here, the same way history `Unknown` is mapped, gives callers a single value for "still running". The bare string set `{"NotStarted", "InProgress"}` could then become a named class attribute (e.g. `ACTIVE_EXTENDED_STATUSES`) next to `TERMINAL_STATUSES`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Detect when enhanced Power BI refreshes finish or fail [airflow]
Posted by "omar-massfih (via GitHub)" <gi...@apache.org>.
omar-massfih commented on PR #74600:
URL: https://github.com/apache/airflow/pull/74600#issuecomment-6314442712
@dabla Hi, I'm not able to merge the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org