You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mahesh989 (via GitHub)" <gi...@apache.org> on 2025/01/22 03:53:47 UTC
[I] Add Validation on Task Executor Field to Prevent Incorrect Configurations [airflow]
mahesh989 opened a new issue, #45898:
URL: https://github.com/apache/airflow/issues/45898
### Apache Airflow version
2.10.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
## What Happened?
In Apache Airflow, tasks were allowed to be assigned to executors without validating whether the specified executor was properly configured or available. This lack of validation led to the following issues:
1. **Unexpected DAG Failures**: When a task was assigned an invalid or unconfigured executor, the DAG execution would fail either silently or with ambiguous error messages.
2. **Inaccurate Error Feedback**: The root cause of the failure—an unconfigured executor—was not explicitly identified, making it difficult to debug and resolve the issue.
3. **Workflow Disruptions**: Invalid executor assignments disrupted dependent tasks, causing pipeline inefficiencies and delays.
4. **Reduced Reliability**: Teams using hybrid executor setups faced challenges ensuring smooth task execution, as unvalidated configurations would propagate unnoticed until runtime.
These issues made it difficult to maintain reliable workflows and increased the risk of errors in production pipelines.
### What you think should happen instead?
## What You Think Should Happen Instead?
Airflow should validate the executor configuration for each task during DAG parsing or at runtime. This validation should ensure that:
1. **Executor Validity Check**: Tasks are only assigned to executors that are properly configured and available within the system.
2. **Clear Error Feedback**: If an invalid or unconfigured executor is specified, a descriptive error message should be raised, identifying the misconfiguration.
3. **Prevention of Workflow Disruptions**: DAG execution should be halted if an invalid executor assignment is detected, preventing downstream failures.
4. **Improved User Experience**: Users should be guided to correct the configuration before the DAG is executed, ensuring a smoother and more reliable pipeline execution.
These measures would enhance workflow reliability, reduce debugging time, and provide a more intuitive experience for Airflow users.
### How to reproduce
## How to Reproduce
1. **Setup Environment**:
- Deploy Apache Airflow version 2.10.0 or later.
- Configure Airflow to use CeleryExecutor or another hybrid executor setup.
2. **Create a DAG**:
- Define a DAG with multiple tasks.
- Assign one or more tasks to an executor that is not configured in the `airflow.cfg` file or is invalid.
3. **Trigger the DAG**:
- Trigger the DAG execution from the Airflow UI or CLI.
4. **Observe Behavior**:
- Check the Airflow UI and logs for errors or inconsistent behavior.
- Note whether tasks assigned to invalid executors proceed without validation or fail during execution without clear error messages.
### Operating System
macOS 14.6.1
### Versions of Apache Airflow Providers
- **Amazon** (`apache-airflow-providers-amazon`): `3.4.0`
- **Google** (`apache-airflow-providers-google`): `10.1.0`
- **Postgres** (`apache-airflow-providers-postgres`): `6.0.0`
- **MySQL** (`apache-airflow-providers-mysql`): `2.2.0`
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Validation on Task Executor Field to Prevent Incorrect Configurations [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #45898:
URL: https://github.com/apache/airflow/issues/45898#issuecomment-2613815128
Your pain is that dag author request executor that doesn't exist. This is why cluster admin should have Cluster Policy set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Validation on Task Executor Field to Prevent Incorrect Configurations [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #45898: Add Validation on Task Executor Field to Prevent Incorrect Configurations
URL: https://github.com/apache/airflow/issues/45898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Validation on Task Executor Field to Prevent Incorrect Configurations [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45898:
URL: https://github.com/apache/airflow/issues/45898#issuecomment-2614535887
Reported you to GitHub. I see that you reported 10 AI spam issues recently to a number of other repositories. If you are a human stop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Validation on Task Executor Field to Prevent Incorrect Configurations [airflow]
Posted by "mahesh989 (via GitHub)" <gi...@apache.org>.
mahesh989 commented on issue #45898:
URL: https://github.com/apache/airflow/issues/45898#issuecomment-2606505886
Hi Jarek,
I would like to apologize that you are not happy with my response. First
of all I would like to let you know that I am a genuine user and love to
code.
As you partly agree that some of the solutions seem legitimate and some are
not AI generated. However this is not true. I am not a big fan of AI when
guidelines clearly mentioned that I am not allowed to use. However, I must
accept the fact that I have made a template by doing some research in
Google. Lets say how to make a proper documentation.
If you agree we can have further discussions and love to work on that
project further because it's really interesting.
Onwards i will take care not to use any writing templates and rather use my
own ideas for writing reports.
Regards,
Maheshwor TIWARI
On Wed, Jan 22, 2025, 18:29 Jarek Potiuk ***@***.***> wrote:
> This looks like AI generated, generic report - that has absolutely no
> value. Can you please explain how you generated that isssue? I am very
> interested to hear - you seem to be a legitimate user, not a bot, and some
> of your issues seem valid - we've been flooded as of yesterday with such
> almost good looking, but otherwise completely useless issues. We are
> generally blocking such users from interacting wih our repos, because it
> takes awfully lot of maintainer's time to verify and filter out such issues.
>
> So:
>
> - explain how it was generated please
> - stop doing it
> - confirm you are a human and explain why you are doing it.
>
> Otherwise we will block and report your account.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/45898#issuecomment-2606483277>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHS6PINQLITRVIMF5TIHIAT2L5CFFAVCNFSM6AAAAABVT7AODGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBWGQ4DGMRXG4>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add Validation on Task Executor Field to Prevent Incorrect Configurations [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45898:
URL: https://github.com/apache/airflow/issues/45898#issuecomment-2606483277
This looks like AI generated, generic report - that has absolutely no value. Can you please explain how you generated that isssue? I am very interested to hear - you seem to be a legitimate user, not a bot, and some of your issues seem valid - we've been flooded as of yesterday with such almost good looking, but otherwise completely useless issues. We are generally blocking such users from interacting wih our repos, because it takes awfully lot of maintainer's time to verify and filter out such issues.
So:
* explain how it was generated please
* stop doing it
* confirm you are a human and explain why you are doing it.
Otherwise we will block and report your account.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org