You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Berenger-Wooclap (via GitHub)" <gi...@apache.org> on 2023/10/04 12:08:40 UTC
[I] Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED [airflow]
Berenger-Wooclap opened a new issue, #34754:
URL: https://github.com/apache/airflow/issues/34754
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
We are using Apache Airflow version 2.5.3, and we have a DAG in which multiple tasks are executed sequentially, with the output of the first task serving as input for the subsequent tasks. All these tasks are set to TriggerRule.NONE_FAILED, meaning they should execute even if one of them fails.
However, we've encountered an issue where if the second task, which is dynamic, is 'skipped' (i.e., it does not run), the following task does not receive the XCom data from the first task. We have implemented a temporary workaround by preventing the second task from being 'skipped,' but we are looking for a permanent solution where the third task always receives XCom data from the first task, even if the second task is 'skipped.'
### What you think should happen instead
I don't really know what is happening but i think the xcom is not propagate for any reason.
### How to reproduce
Here is the part of the dag that create the tasks :
` tables_to_export = PythonOperator(
task_id="show_configuration",
python_callable=get_tables_to_export_from_params,
on_failure_callback=slack_fail_alert,
provide_context=True,
)
mongodb_exporter = WooclapKubernetesPodOperator.partial(
config=config,
image_pull_policy="Always",
service_account_name=service_account_name,
default_image_name=default_image_name,
on_failure_callback=slack_fail_alert,
name="mongodb-exporter-pod",
task_id="mongodb-exporter-task",
pool="mongo-db-pool-limit",
).expand(expand_args=tables_to_export.output)
mongo_dump_to_pg = WooclapKubernetesPodOperator.partial(
config=config,
image_pull_policy="Always",
service_account_name=service_account_name,
default_image_name=default_image_name,
on_failure_callback=slack_fail_alert,
name="mongo-dump-to-pg-pod",
task_id="mongo-dump-to-pg-task",
pool="woostats-db-pool-limit",
).expand(expand_args=tables_to_export.output)`
### Operating System
Airflow 2.5.3 deploy on kubernetes
### Versions of Apache Airflow Providers
Airflow 2.5.3
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
The xcom is done with postgres db if i'm not wrong
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #34754: Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED
URL: https://github.com/apache/airflow/issues/34754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34754:
URL: https://github.com/apache/airflow/issues/34754#issuecomment-1802944199
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34754:
URL: https://github.com/apache/airflow/issues/34754#issuecomment-1746745203
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #34754:
URL: https://github.com/apache/airflow/issues/34754#issuecomment-1788203158
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Skipping an Airflow task make the dynamic task skipped too even with TriggerRule.NONE_FAILED [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #34754:
URL: https://github.com/apache/airflow/issues/34754#issuecomment-1765238654
I've unable to reproduce on current main, maybe it already fixed (I can't find related issue/PR)
However this one worked as expected
```python
from airflow.decorators import task
from airflow.models.baseoperator import BaseOperator
from airflow.models.dag import DAG
from airflow.utils.timezone import datetime
from airflow.exceptions import AirflowSkipException
@task
def make_list():
return [4, 42, 2]
class Double(BaseOperator):
def __init__(self, x: int, **kwargs):
super().__init__(**kwargs)
self.x = x
def execute(self, context):
if self.x == 42:
raise AirflowSkipException("42")
return self.x * 2
class Result(BaseOperator):
def __init__(self, result: int, **kwargs):
super().__init__(**kwargs)
self.result = result
def execute(self, context):
print(self.result)
with DAG(
"issue_34754",
start_date=datetime(2023, 10, 1),
schedule_interval="@daily",
catchup=False,
tags=["issue", "34754"]
) as dag:
double = Double.partial(
task_id="double-int"
).expand(x=make_list())
Result.partial(
task_id="print-result",
trigger_rule="none_failed"
).expand(result=double.output)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org