You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/30 14:45:29 UTC
[GitHub] [airflow] Jorricks opened a new issue, #26068: Incompatibility with old Airflow 2.2.3
Jorricks opened a new issue, #26068:
URL: https://github.com/apache/airflow/issues/26068
### Apache Airflow version
2.3.4
### What happened
Some tasks do not have a `queue` value defined and equal `None`.
When you load the GRID view for these DAGs you get a 502 error with the following error appearing in the logs:
```
https://airflow-webserver-hdp-beta-b.is.adyen.com/api/v1/dags/shopper_connect_report/tasks validation error: None is not of type 'string'
Failed validating 'type' in schema['properties']['tasks']['items']['properties']['queue']:
{'readOnly': True, 'type': 'string'}
On instance['tasks'][4]['queue']:
None
```
### What you think should happen instead
The API should return a successful 200 response with the data.
The schema should allow for the `queue` to be nullable. Currently it's not as you can see here:
```
retries:
type: number
readOnly: true
queue:
type: string
readOnly: true
...
execution_timeout:
$ref: '#/components/schemas/TimeDelta'
nullable: true
```
We should add `nullable: true` to this field. Currently this crashes the webserver with this error.
### How to reproduce
Create some TaskInstances.
Then make the queue field empty in a Database.
You can check this in the TaskInstance view:
Then open the GRID view.
### Operating System
CentOS Linux 7 (Core)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-apache-druid==2.0.2
# via apache-airflow
apache-airflow-providers-apache-hdfs==2.1.1
# via apache-airflow
apache-airflow-providers-apache-hive==2.0.2
# via apache-airflow
apache-airflow-providers-apache-spark==2.0.1
# via apache-airflow
apache-airflow-providers-celery==2.1.0
# via
# -r adyen_airflow/requirements/requirements.txt
# apache-airflow
apache-airflow-providers-ftp==2.0.1
# via apache-airflow
apache-airflow-providers-http==2.0.1
# via apache-airflow
apache-airflow-providers-imap==2.0.1
# via apache-airflow
apache-airflow-providers-pagerduty==2.0.1
# via apache-airflow
apache-airflow-providers-postgres==2.3.0
# via apache-airflow
apache-airflow-providers-sqlite==2.0.1
# via apache-airflow
```
### Deployment
Other
### Deployment details
Deployed on centos boxes without virtualisation
### Anything else
This problem occurs only on specific DAGs. It remains unknown to me why we have tasks that do not have a Queue specified.
My expectation is that new tasks were added, then these tasks are set to success by setting the DAGRun to a success.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #26068: Missing Queue values breaks API
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #26068:
URL: https://github.com/apache/airflow/issues/26068#issuecomment-1239794978
completed https://github.com/apache/airflow/pull/26076 https://github.com/apache/airflow/pull/26077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jorricks commented on issue #26068: Missing Queue values breaks API
Posted by GitBox <gi...@apache.org>.
Jorricks commented on issue #26068:
URL: https://github.com/apache/airflow/issues/26068#issuecomment-1239878391
> > Given its removed from the 2.3.5. Milestone, cab u assume we are not planning on doing a 2.3.5 release?
>
> milestones are set on the PRs so no need for one on the issue.
> i don't think there will be 2.3.5
Thang for the swift reply!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #26068: Missing Queue values breaks API
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #26068:
URL: https://github.com/apache/airflow/issues/26068#issuecomment-1239831981
> Given its removed from the 2.3.5. Milestone, cab u assume we are not planning on doing a 2.3.5 release?
milestones are set on the PRs so no need for one on the issue.
i don't think there will be 2.3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jorricks commented on issue #26068: Incompatibility with old Airflow 2.2.3
Posted by GitBox <gi...@apache.org>.
Jorricks commented on issue #26068:
URL: https://github.com/apache/airflow/issues/26068#issuecomment-1231831467
I think it might fail on more than just this field.
It seems when new TaskInstances are spawned, there are a lot of fields that don't adhere to this schema:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jorricks commented on issue #26068: Incompatibility with old Airflow 2.2.3
Posted by GitBox <gi...@apache.org>.
Jorricks commented on issue #26068:
URL: https://github.com/apache/airflow/issues/26068#issuecomment-1231806851
From what I can see, this is only present for TaskInstance created by older Airflow versions as long as you don't use the `ti_mutation_hook`.
```
def _get_task_creator(
self, created_counts: Dict[str, int], ti_mutation_hook: Callable, hook_is_noop: bool
) -> Callable:
"""
Get the task creator function.
This function also updates the created_counts dictionary with the number of tasks created.
:param created_counts: Dictionary of task_type -> count of created TIs
:param ti_mutation_hook: task_instance_mutation_hook function
:param hook_is_noop: Whether the task_instance_mutation_hook is a noop
"""
if hook_is_noop:
def create_ti_mapping(task: "Operator", indexes: Tuple[int, ...]) -> Generator:
created_counts[task.task_type] += 1
for map_index in indexes:
yield TI.insert_mapping(self.run_id, task, map_index=map_index)
creator = create_ti_mapping
else:
def create_ti(task: "Operator", indexes: Tuple[int, ...]) -> Generator:
for map_index in indexes:
ti = TI(task, run_id=self.run_id, map_index=map_index)
ti_mutation_hook(ti)
created_counts[ti.operator] += 1
yield ti
creator = create_ti
return creator
```
The `TI.insert_mapping` will set the `queue` for you. The other one however, will not set the `queue` for you, so I can imagine if you just spawned a couple TaskInstances, you will get the same error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #26068: Missing Queue values breaks API
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #26068: Missing Queue values breaks API
URL: https://github.com/apache/airflow/issues/26068
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jorricks commented on issue #26068: Missing Queue values breaks API
Posted by GitBox <gi...@apache.org>.
Jorricks commented on issue #26068:
URL: https://github.com/apache/airflow/issues/26068#issuecomment-1239825229
> completed https://github.com/apache/airflow/pull/26076 https://github.com/apache/airflow/pull/26077
Given its removed from the 2.3.5. Milestone, cab u assume we are not planning on doing a 2.3.5 release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jorricks commented on issue #26068: Incompatibility with old Airflow 2.2.3
Posted by GitBox <gi...@apache.org>.
Jorricks commented on issue #26068:
URL: https://github.com/apache/airflow/issues/26068#issuecomment-1231826808
@potiuk I'll happily submit a PR but a lot has changed since the last time I made changes to the API.
I was wondering two things:
- Do we compile `v1.yaml`?
- It seems that besides the change in `v1.yaml`, the change `queue = fields.String(dump_only=True)` to `queue = fields.String(dump_only=True, allow_none=True)` inside the `TaskSchema` would be all right :)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org