You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/16 15:09:08 UTC
[GitHub] [airflow] patryk126p opened a new issue, #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
patryk126p opened a new issue, #23727:
URL: https://github.com/apache/airflow/issues/23727
### Apache Airflow version
2.3.0 (latest released)
### What happened
After migrating from Airflow 2.2.4 to 2.3.0 scheduler fell into crash loop throwing:
```
--- Logging error ---
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 736, in _execute
self._run_scheduler_loop()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 826, in _run_scheduler_loop
self.executor.heartbeat()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/base_executor.py", line 171, in heartbeat
self.sync()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py", line 613, in sync
self.kube_scheduler.run_next(task)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py", line 300, in run_next
self.log.info('Kubernetes job is %s', str(next_job).replace("\n", " "))
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 214, in __repr__
return self.to_str()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 210, in to_str
return pprint.pformat(self.to_dict())
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 196, in to_dict
result[attr] = value.to_dict()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1070, in to_dict
result[attr] = list(map(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1071, in <lambda>
lambda x: x.to_dict() if hasattr(x, "to_dict") else x,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 672, in to_dict
value = getattr(self, attr)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 464, in startup_probe
return self._startup_probe
AttributeError: 'V1Container' object has no attribute '_startup_probe'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit
msg = self.format(record)
File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format
return fmt.format(record)
File "/usr/local/lib/python3.9/logging/__init__.py", line 663, in format
record.message = record.getMessage()
File "/usr/local/lib/python3.9/logging/__init__.py", line 367, in getMessage
msg = msg % self.args
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 214, in __repr__
return self.to_str()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 210, in to_str
return pprint.pformat(self.to_dict())
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 196, in to_dict
result[attr] = value.to_dict()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1070, in to_dict
result[attr] = list(map(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1071, in <lambda>
lambda x: x.to_dict() if hasattr(x, "to_dict") else x,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 672, in to_dict
value = getattr(self, attr)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 464, in startup_probe
return self._startup_probe
AttributeError: 'V1Container' object has no attribute '_startup_probe'
Call stack:
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/__main__.py", line 38, in main
args.func(args)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/scheduler_command.py", line 75, in scheduler
_run_scheduler_job(args=args)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/cli/commands/scheduler_command.py", line 46, in _run_scheduler_job
job.run()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/base_job.py", line 244, in run
self._execute()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 757, in _execute
self.executor.end()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py", line 809, in end
self._flush_task_queue()
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py", line 767, in _flush_task_queue
self.log.warning('Executor shutting down, will NOT run task=%s', task)
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.
```
kubernetes python library version was exactly as specified in constraints file: https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0/constraints-3.9.txt
### What you think should happen instead
Scheduler should work
### How to reproduce
1. Run Airflow 2.2.4 using official Helm Chart
2. Run some dags to have some records in DB
3. Migrate to 2.3.0 (replace 2.2.4 image with 2.3.0 one)
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
irrelevant
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
KubernetesExecutor
PostgreSQL (RDS) as Airflow DB
Python 3.9
Docker images build from `apache/airflow:2.3.0-python3.9` (some additional libraries installed)
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1129830423
> We have no startup_probes defined and we are using the default pod template file that comes with the Helm chart - we only overwrite some options (resources limits and requests) via executor_config to match task needs
So this is likely some serialization issue as I suspected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] humbledude commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
humbledude commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1158430409
@dstandish I patched with commits on #24478 and it seems work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1145102329
draft pr here: https://github.com/apache/airflow/pull/24117/
resolves the scheduler crash and fails the task, but i want to see if we can make it so that executor config gets reloaded after clearing the failed task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1156670025
This issue is resolved by #24117 because it is about the scheduler crash, not the webserver. I was under the impression that the webserver issue may have been handled / documented elsewhere. If that's not the case can you please create an issue and tag me?
> I don't think pod reconcile function is the cause of this, since it only merges pod_template with pod_override,
I think what you miss is that the executor_config, if provided, is merged in pod reconciliation also. and executor_config is the root cause here --- it gets pickled with old k8s library and when read with new library bad things happen.
> symptom not cause
It's important to not crash the scheduler. That's fixed by #24117 and that's why it's merged. I have another PR https://github.com/apache/airflow/pull/24356 that works to make executor config pickling more tolerant of k8s library upgrades, but it's less trivial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1146319419
alright i think #24117 is a fix for this and it's ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134315657
Yeah - it does look as the deserialisation issue as I suspected. Just a thought - we do have the new
`airflow dags reserialize` command in Airflow.
I wonder @patryk126p if that is something that you could run to see if it helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1129832232
I will wait for @dstandish comment - maybe this is a "known" issue already, but for me it looks like one that should be solved in 2.3.1 as it's quite likely caused by the k8s library version migration between 2.2 and 2.3 in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1131196192
@dstandish we are not using kubernetes pod operators nor specifying `full_pod_spec`. When it comes to `executor_config` we use something like this:
```
{
"pod_override": V1Pod(
spec=V1PodSpec(
containers=[
V1Container(
name="base",
resources=V1ResourceRequirements(
limits={"cpu": "<X>", "memory": "<X>"},
requests={"cpu": "<X>", "memory": "<X>"},
),
)
]
),
# metadata=V1ObjectMeta(
# annotations={"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"}
# ),
)
}
```
Metadata/annotations are added only in case of very long running and critical tasks, to be sure that kubernetes won't evict pods in the middle of processing.
Sample dag may be a little tricky to provide as in almost all dags we are using custom operators (mostly based directly on BaseOperator), I could provide simplified version using PythonOperators but I'm not sure if that would be of much help. In majority of dags the flow is simple: collect data from API, dump it to S3, process (if needed), load to stage tables, load to final tables.
In general we are not doing anything related to startup_probe and the only direct link between our dag/task definitions and kubernetes is the `executor_config` that I've provided
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1142088809
Any news @patryk126p :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1129745411
There are few options where it could get wrong:
* you might have a startup_probe defined in one of your pod template files (can you please take a look and let us know). @patryk126p ?
* there are some left-over serialized objects in airflow DB that are left-over from 2.2.4 using the old version of kubernetes library - in which case we might want to catch such error and fix or remove the serialized data and reserialize it
@dstandish - if the second hypothesiss is confirmed, it might actually mean that we have to implement a fix for 2.3.1 as it might prevent the migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1143556459
not yet, sorry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KulykDmytro commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
KulykDmytro commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1343251798
in 2.5.0 #28227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1130207868
taking a look at this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1132480598
It's 1.21.9 on EKS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
tirkarthi commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134295021
A related Stackoverflow question : https://stackoverflow.com/questions/67929230/airflow-2-0-1-pod-template-override-not-working-as-expected-for-kubernetesexecu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hterik commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
hterik commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1156012556
I'm also a bit confused how #24117 closes this, it appears to catch errors in pod reconcile, which i don't see as part of neither the first stack trace from OP and as already mentioned by @humbledude above probably not the one in webserver.
I don't think pod reconcile function is the cause of this, since it only merges pod_template with pod_override, both which are functions of current state, not historical state? Failing the task is still better than crashing the whole scheduler so from that point of view it's still good change but it only treats one of the symptoms, not the cause.
Should we file a new bug for the webserver crash?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1156681731
@hterik @humbledude see https://github.com/apache/airflow/pull/24478 for webserver fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1152691754
Ah yeah. thanks for the info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1131233034
Here is the complete set of imports from all custom operators/sensors/hooks (somme of the imports are for type hints only):
```
import base64
import json
import logging
import math
import operator
import os
import pathlib
import re
import sys
import time
import typing
from collections import namedtuple
from contextlib import closing
from datetime import datetime, timedelta, timezone
from subprocess import PIPE, CompletedProcess, Popen, SubprocessError, run
from urllib.parse import urlencode
import backoff
import certifi
import google_auth_httplib2
import httplib2
import newrelic_telemetry_sdk
import pandas as pd
import psycopg2
import redis
import requests
import tableauserverclient
import yaml
from airflow.exceptions import AirflowException, AirflowSensorTimeout
from airflow.hooks.base import BaseHook
from airflow.models import DAG, BaseOperator, Connection, Variable
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from airflow.providers.microsoft.mssql.hooks.mssql import MsSqlHook
from airflow.providers.mysql.hooks.mysql import MySqlHook
from airflow.providers.postgres.hooks.postgres import PostgresHook
from airflow.providers.postgres.operators.postgres import PostgresOperator
from airflow.sensors.base import BaseSensorOperator
from airflow.sensors.external_task import ExternalTaskSensor
from airflow.utils import timezone
from airflow.utils.helpers import parse_template_string
from botocore.credentials import ReadOnlyCredentials
from google.api_core import retry
from google.oauth2 import service_account
from googleapiclient.discovery import Resource, build
from googleapiclient.http import set_user_agent
from jinja2 import Template
from tabulate import tabulate
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1132280487
Which Kubernetes version do you run (cluster version) @patryk126p ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish closed issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
URL: https://github.com/apache/airflow/issues/23727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1133297251
@patryk126p was hoping to see how you are importing the k8s objects used here:
```python
{
"pod_override": V1Pod(
spec=V1PodSpec(
containers=[
V1Container(
name="base",
resources=V1ResourceRequirements(
limits={"cpu": "<X>", "memory": "<X>"},
requests={"cpu": "<X>", "memory": "<X>"},
),
)
]
),
# metadata=V1ObjectMeta(
# annotations={"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"}
# ),
)
}
```
though i know it's a long shot that it leads us anywhere.
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1130362272
actually yeah does look like you must be putting something in executor config. please share the code related to this. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1127795078
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1221623855
> Is a separate fix needed for that?
Can you try 2.3.4 (It will be out in 2 days or so) and if it is still there please open a new issue with detailed description of the problem and how you got there. You can refer to this issue in the new one but piggybacking on existing, closed issue is not going to make it "active" I am afraid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1131197819
@patryk126p please show the imports for those classes
If you can establish repro steps too that would be really helpful too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1131319637
Are you sure that the kubernetes library has been upgraded everywhere in your system ? For me it looks like you have some image that has the old version of library still installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1155645370
this fix works: https://github.com/apache/airflow/pull/24117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] humbledude commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
humbledude commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1156001682
What about crashing webserver UI?
#24117 seems fix scheduler, but webserver still got crash on viewing old task instances.
My error is just like @hterik
(https://github.com/apache/airflow/issues/23727#issuecomment-1155288463)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1156859765
ok @hterik @humbledude webserver fix merged so you are welcome to try patching your envs with those changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1130272080
@patryk126p can you provide a sample dag? i attempted to repro using your steps (ran a simple dag in 2.2.4, upgraded to 2.3.0, ran it again) and there was no issue. perhaps you are doing something with executor config? or perhaps specifying `full_pod_spec` in kubernetes pod operator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134202626
@dstandish are you asking just for import statement? If yes then here it is:
```
from kubernetes.client.models import (
V1Container,
V1ObjectMeta,
V1Pod,
V1PodSpec,
V1ResourceRequirements,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1143674631
I looked into this yesterday. I think it's cus of pickled object executor_config on TI
I will work on it more today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1129783435
We have no startup_probes defined and we are using the default pod template file that comes with the Helm chart - we only overwrite some options (resources limits and requests) via executor_config to match task needs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1131418985
From inside container (same docker images were used for all components in helm chart):
```
$ pip freeze | grep -E "airflow|kubernetes"
apache-airflow==2.3.0
apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-celery==2.1.4
apache-airflow-providers-cncf-kubernetes==4.0.1
apache-airflow-providers-docker==2.6.0
apache-airflow-providers-elasticsearch==3.0.3
apache-airflow-providers-ftp==2.1.2
apache-airflow-providers-google==6.8.0
apache-airflow-providers-grpc==2.0.4
apache-airflow-providers-hashicorp==2.2.0
apache-airflow-providers-http==2.1.2
apache-airflow-providers-imap==2.2.3
apache-airflow-providers-microsoft-azure==3.8.0
apache-airflow-providers-mysql==2.2.3
apache-airflow-providers-odbc==2.0.4
apache-airflow-providers-postgres==4.1.0
apache-airflow-providers-redis==2.0.4
apache-airflow-providers-sendgrid==2.0.4
apache-airflow-providers-sftp==2.6.0
apache-airflow-providers-slack==4.2.3
apache-airflow-providers-sqlite==2.1.3
apache-airflow-providers-ssh==2.4.3
google-cloud-orchestration-airflow==1.3.1
kubernetes==23.3.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1152149987
> Yeah - it does look as the deserialisation issue as I suspected. Just a thought - we do have the new `airflow dags reserialize` command in Airflow.
>
> I wonder @patryk126p if that is something that you could run to see if it helps.
FYI @potiuk `reserialize` won't help, as this is a pickled column in TaskInstance and nothing to do with the serialized DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hterik commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
hterik commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1155288463
After upgrading from 2.2.4 to 2.3.2. I get same error in the **webserver** when trying to view any task result of runs that were produced before the upgrade.
Runs happening after the upgrade are still possible to view.
Eg
```
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/airflow/.local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/airflow/.local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/airflow/.local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/airflow/.local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/auth.py", line 43, in decorated
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/decorators.py", line 117, in view_func
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/decorators.py", line 80, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/views.py", line 2882, in graph
return self.render_template(
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/views.py", line 608, in render_template
return super().render_template(
File "/home/airflow/.local/lib/python3.9/site-packages/flask_appbuilder/baseviews.py", line 287, in render_template
return render_template(
File "/home/airflow/.local/lib/python3.9/site-packages/flask/templating.py", line 137, in render_template
return _render(
File "/home/airflow/.local/lib/python3.9/site-packages/flask/templating.py", line 120, in _render
rv = template.render(context)
File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/environment.py", line 1291, in render
self.environment.handle_exception()
File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/environment.py", line 925, in handle_exception
raise rewrite_traceback_stack(source=source)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/templates/airflow/graph.html", line 21, in top-level template code
{% from 'appbuilder/loading_dots.html' import loading_dots %}
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/templates/airflow/dag.html", line 36, in top-level template code
{% set execution_date_arg = request.args.get('execution_date') %}
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/templates/airflow/main.html", line 21, in top-level template code
{% from 'airflow/_messages.html' import show_message %}
File "/home/airflow/.local/lib/python3.9/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File "/home/airflow/.local/lib/python3.9/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 50, in top-level template code
{% block tail %}
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/www/templates/airflow/graph.html", line 137, in block 'tail'
let taskInstances = {{ task_instances|tojson }};
File "/home/airflow/.local/lib/python3.9/site-packages/flask/json/__init__.py", line 376, in tojson_filter
return Markup(htmlsafe_dumps(obj, **kwargs))
File "/home/airflow/.local/lib/python3.9/site-packages/flask/json/__init__.py", line 290, in htmlsafe_dumps
dumps(obj, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/flask/json/__init__.py", line 211, in dumps
rv = _json.dumps(obj, **kwargs)
File "/usr/local/lib/python3.9/json/__init__.py", line 234, in dumps
return cls(
File "/usr/local/lib/python3.9/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.9/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/json.py", line 84, in _default
return PodGenerator.serialize_pod(obj)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/kubernetes/pod_generator.py", line 404, in serialize_pod
return api_client.sanitize_for_serialization(pod)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 241, in sanitize_for_serialization
return {key: self.sanitize_for_serialization(val)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 241, in <dictcomp>
return {key: self.sanitize_for_serialization(val)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 237, in sanitize_for_serialization
obj_dict = {obj.attribute_map[attr]: getattr(obj, attr)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 239, in <dictcomp>
if getattr(obj, attr) is not None}
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 397, in ephemeral_containers
return self._ephemeral_containers
AttributeError: 'V1PodSpec' object has no attribute '_ephemeral_containers'
```
Using same kubernetes python client as listed in the official constraints file (kubernetes==23.6.0)
Cluster is a managed AKS, version 1.23.5
`airflow dags reserialize` did not help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joshzana commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
joshzana commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1207530417
@dstandish I'm seeing this on Airflow 2.3.3 with a vanilla web server docker image, which should have #24478 in it. We use the REST API for an integration with another system of ours, and when it calls the `get_task_instances_batch` API it's hitting a similar stack:
```
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/api_connexion/endpoints/task_instance_endpoint.py", line 412, in get_task_instances_batch
return task_instance_collection_schema.dump(
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 557, in dump
result = self._serialize(processed_obj, many=many)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 525, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 342, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 774, in _serialize
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 774, in <listcomp>
return [self.inner._serialize(each, attr, obj, **kwargs) for each in value]
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 643, in _serialize
return schema.dump(nested_obj, many=many)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 557, in dump
result = self._serialize(processed_obj, many=many)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 525, in _serialize
value = field_obj.serialize(attr_name, obj, accessor=self.get_attribute)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 342, in serialize
return self._serialize(value, attr, obj, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 893, in _serialize
return utils.ensure_text_type(value)
File "/home/airflow/.local/lib/python3.9/site-packages/marshmallow/utils.py", line 212, in ensure_text_type
return str(val)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 214, in __repr__
return self.to_str()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 210, in to_str
return pprint.pformat(self.to_dict())
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod.py", line 196, in to_dict
result[attr] = value.to_dict()
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1058, in to_dict
result[attr] = list(map(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_pod_spec.py", line 1059, in <lambda>
lambda x: x.to_dict() if hasattr(x, "to_dict") else x,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 660, in to_dict
value = getattr(self, attr)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/models/v1_container.py", line 458, in startup_probe
return self._startup_probe
AttributeError: 'V1Container' object has no attribute '_startup_probe'
```
Is a separate fix needed for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rodolphogarrido commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
rodolphogarrido commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1155621643
I'm having the exact same error and behavior as mentioned by hterik.
It's also not possible to clear the DAG run, only deleting the record (then if I run the day regarding the deleted record, it can be viewed since it will be using the new k8s object).
As stated by ashb and hterik running the `airflow dags reserialize` did not help. Curiously through some old runs are shown.
Looking forward for the fix 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1158498532
great, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patryk126p commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
patryk126p commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1135443024
Sounds like this may be it. Once we have a window when we can attempt moving to 2.3 again I will try this command. But in the meantime can such info be added to some migration docs or something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23727: Airflow 2.3 scheduler error: 'V1Container' object has no attribute '_startup_probe'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23727:
URL: https://github.com/apache/airflow/issues/23727#issuecomment-1134318907
This command is exactly foreseen for this kind of cases where the dags were serialized using older airflow versions.
I suspect what could have happened is that the old dags were serialized when there was an old kubernetes library installed, and it lacked the`_startup_probe`, and the new version of library `requires` the _startup_probe to be there.
In this case `airflow dags reserialize` should help as it cleans all the serialized dags and re-serializes them using the current version of the library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org