You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "karakanb (via GitHub)" <gi...@apache.org> on 2024/01/03 21:49:36 UTC
[I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
karakanb opened a new issue, #36571:
URL: https://github.com/apache/airflow/issues/36571
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.14.0
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-cncf-kubernetes==7.12.0
apache-airflow-providers-common-io==1.1.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-discord==3.5.0
apache-airflow-providers-docker==3.8.2
apache-airflow-providers-elasticsearch==5.3.0
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.13.0
apache-airflow-providers-grpc==3.4.0
apache-airflow-providers-hashicorp==3.6.0
apache-airflow-providers-http==4.8.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==8.4.0
apache-airflow-providers-mysql==5.5.0
apache-airflow-providers-odbc==4.2.0
apache-airflow-providers-openlineage==1.3.0
apache-airflow-providers-postgres==5.9.0
apache-airflow-providers-redis==3.5.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.8.0
apache-airflow-providers-slack==8.5.1
apache-airflow-providers-snowflake==5.2.1
apache-airflow-providers-sqlite==3.6.0
apache-airflow-providers-ssh==3.9.0
### Apache Airflow version
2.8.0
### Operating System
Debian 12 (bookworm)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
deployed on digitalocean k8s. I have a custom log format as `'%%(asctime)s | %%(levelname)s | %%(message)s'`
### What happened
I have recently upgraded the cncf-kubernetes provider from `7.4.2` to `7.12.0` and started seeing weird log lines being printed out in-between application logs:
```
2024-01-03T21:31:59.286+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.313+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.694053856Z
2024-01-03T21:32:00.694107912Z
2024-01-03T21:32:00.694384093Z
2024-01-03T21:32:00.694416033Z
2024-01-03T21:32:00.694514264Z
2024-01-03T21:32:00.694543915Z
2024-01-03T21:32:00.694553314Z
2024-01-03T21:32:00.694557782Z
2024-01-03T21:32:00.694629859Z
2024-01-03T21:32:00.694642682Z
2024-01-03T21:32:01.348+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
```
this also happens while the application is running:
```
2024-01-03T21:32:04.814+0000 | INFO | [base] application log line 1
2024-01-03T21:32:05.234+0000 | INFO | [base] application log line 2
2024-01-03T21:32:05.526911055Z
2024-01-03T21:32:05.529+0000 | INFO | [base] application log line 3
2024-01-03T21:32:05.529+0000 | INFO | [base] application log line 4
2024-01-03T21:32:05.654767914Z
2024-01-03T21:32:05.654830820Z
2024-01-03T21:32:05.654857369Z
```
As you can see the log lines have a different format than the actual airflow log format, and they seem to contain no messages. This causes confusion, and for a few cases where I parse the logs it broke my existing code.
I have checked through external log collectors as well as via tailing the pod logs and I can confirm that the logs do not come from the application/pod side, and being logged somewhere on the airflow side.
### What you think should happen instead
there should be no log lines printed that doesn't respect the logging format.
### How to reproduce
Create a pod with KubernetesPodOperator, it happens to me consistently.
### Anything else
Every time I run a pod it happens.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-2077427279
There's not much I can report unfortunately. I am running v2.9.0 and the issue is still there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-2041385244
please reopen this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #36571: KubernetesPodOperator prints empty log lines without respecting the log format
URL: https://github.com/apache/airflow/issues/36571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1930994862
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1959202638
I added `pinned` label to keep it open @karakanb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-2040804949
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1878821864
I can confirm that the issue is still there even after I disabled the context logger:
```cfg
[logging]
colored_console_log = False
enable_task_context_logger = False
log_format = %%(asctime)s | %%(levelname)s | %%(message)s
remote_logging = False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1879758569
@potiuk
according to the provided log:
```
2024-01-03T21:31:59.286+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.313+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.694053856Z
2024-01-03T21:32:00.694107912Z
2024-01-03T21:32:00.694384093Z
2024-01-03T21:32:00.694416033Z
2024-01-03T21:32:00.694514264Z
2024-01-03T21:32:00.694543915Z
2024-01-03T21:32:00.694553314Z
2024-01-03T21:32:00.694557782Z
2024-01-03T21:32:00.694629859Z
2024-01-03T21:32:00.694642682Z
2024-01-03T21:32:01.348+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
```
the empty log messages are related to this loop:
https://github.com/apache/airflow/blob/2c15dc9dd1b360bcb3b6d6c2e41e50ef0d97801e/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L355-L366
and most probably logged by a 3rd party lib (`tenacity.retry` or kuberntes `read_namespaced_pod` methods) 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1999680202
keep it open please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #36571: KubernetesPodOperator prints empty log lines without respecting the log format
URL: https://github.com/apache/airflow/issues/36571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1905060258
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1880597850
Here's the dockerfile I use for this deployment:
```dockerfile
FROM apache/airflow:2.8.0-python3.11
ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"
ENV SQLALCHEMY_SILENCE_UBER_WARNING=1
USER root
RUN apt-get update && apt-get install -y git gcc g++ unixodbc unixodbc-dev make nano curl
USER airflow
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
COPY . .
```
The `requirements.txt`:
```
apache-airflow-providers-amazon==8.14.0
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-cncf-kubernetes==7.13.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-discord==3.5.0
apache-airflow-providers-google==10.13.0
apache-airflow-providers-slack==8.5.1
apache-airflow-providers-snowflake==5.2.1
apache-airflow===2.8.0
deepdiff==6.7.1
isort==5.13.2
mypy==1.8.0
pendulum==2.1.2
pydantic==2.5.3
pytest-cov==4.1.0
pytest==7.4.3
pyyaml==6.0.1
types-PyYAML==6.0.1
requests-mock==1.11.0
ruff==0.1.9
tabulate==0.9.0
hvac==2.0.0
```
<details>
<summary>The full set of dependencies, in case it helps:</summary>
```
adal==1.2.7
adlfs==2023.10.0
aiobotocore==2.9.0
aiofiles==23.2.1
aiohttp==3.8.6
aioitertools==0.11.0
aiosignal==1.3.1
alembic==1.13.0
amqp==5.2.0
annotated-types==0.6.0
anyio==4.1.0
apache-airflow==2.8.0
apache-airflow-providers-amazon==8.14.0
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-cncf-kubernetes==7.13.0
apache-airflow-providers-common-io==1.1.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-discord==3.5.0
apache-airflow-providers-docker==3.8.2
apache-airflow-providers-elasticsearch==5.3.0
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.13.0
apache-airflow-providers-grpc==3.4.0
apache-airflow-providers-hashicorp==3.6.0
apache-airflow-providers-http==4.8.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-microsoft-azure==8.4.0
apache-airflow-providers-mysql==5.5.0
apache-airflow-providers-odbc==4.2.0
apache-airflow-providers-openlineage==1.3.0
apache-airflow-providers-postgres==5.9.0
apache-airflow-providers-redis==3.5.0
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sftp==4.8.0
apache-airflow-providers-slack==8.5.1
apache-airflow-providers-snowflake==5.2.1
apache-airflow-providers-sqlite==3.6.0
apache-airflow-providers-ssh==3.9.0
apispec==6.3.0
argcomplete==3.2.1
asgiref==3.7.2
asn1crypto==1.5.1
async-timeout==4.0.3
attrs==23.1.0
Authlib==1.2.1
azure-batch==14.0.0
azure-common==1.1.28
azure-core==1.29.6
azure-cosmos==4.5.1
azure-datalake-store==0.0.53
azure-identity==1.15.0
azure-keyvault-secrets==4.7.0
azure-kusto-data==4.3.0
azure-mgmt-containerinstance==10.1.0
azure-mgmt-containerregistry==10.3.0
azure-mgmt-core==1.4.0
azure-mgmt-cosmosdb==9.3.0
azure-mgmt-datafactory==4.0.0
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==23.0.1
azure-mgmt-storage==21.1.0
azure-nspkg==3.0.2
azure-servicebus==7.11.4
azure-storage-blob==12.19.0
azure-storage-file-datalake==12.14.0
azure-storage-file-share==12.15.0
azure-synapse-artifacts==0.18.0
azure-synapse-spark==0.7.0
Babel==2.14.0
backoff==2.2.1
bcrypt==4.1.2
beautifulsoup4==4.12.2
billiard==4.2.0
blinker==1.7.0
boto3==1.33.13
botocore==1.33.13
cachelib==0.9.0
cachetools==5.3.2
cattrs==23.2.3
celery==5.3.6
certifi==2023.11.17
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
clickclick==20.10.2
colorama==0.4.6
colorlog==4.8.0
ConfigUpdater==3.2
connexion==2.14.2
coverage==7.4.0
cron-descriptor==1.4.0
croniter==2.0.1
cryptography==41.0.7
db-dtypes==1.2.0
decorator==5.1.1
deepdiff==6.7.1
Deprecated==1.2.14
dill==0.3.1.1
distlib==0.3.8
dnspython==2.4.2
docker==7.0.0
docutils==0.20.1
elastic-transport==8.11.0
elasticsearch==8.11.1
email-validator==1.3.1
eventlet==0.33.3
filelock==3.13.1
Flask==2.2.5
Flask-AppBuilder==4.3.10
Flask-Babel==2.0.0
Flask-Caching==2.1.0
Flask-JWT-Extended==4.6.0
Flask-Limiter==3.5.0
Flask-Login==0.6.3
Flask-Session==0.5.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==1.2.1
flower==2.0.1
frozenlist==1.4.1
fsspec==2023.12.2
gcloud-aio-auth==4.2.3
gcloud-aio-bigquery==7.0.0
gcloud-aio-storage==9.0.0
gcsfs==2023.12.2.post1
gevent==23.9.1
google-ads==22.1.0
google-api-core==2.15.0
google-api-python-client==2.111.0
google-auth==2.25.2
google-auth-httplib2==0.2.0
google-auth-oauthlib==1.2.0
google-cloud-aiplatform==1.38.1
google-cloud-appengine-logging==1.4.0
google-cloud-audit-log==0.2.5
google-cloud-automl==2.12.0
google-cloud-batch==0.17.6
google-cloud-bigquery==3.14.1
google-cloud-bigquery-datatransfer==3.13.0
google-cloud-bigquery-storage==2.24.0
google-cloud-bigtable==2.22.0
google-cloud-build==3.22.0
google-cloud-compute==1.15.0
google-cloud-container==2.36.0
google-cloud-core==2.4.1
google-cloud-datacatalog==3.17.0
google-cloud-dataflow-client==0.8.6
google-cloud-dataform==0.5.5
google-cloud-dataplex==1.10.0
google-cloud-dataproc==5.8.0
google-cloud-dataproc-metastore==1.14.0
google-cloud-dlp==3.14.0
google-cloud-kms==2.20.0
google-cloud-language==2.12.0
google-cloud-logging==3.9.0
google-cloud-memcache==1.8.0
google-cloud-monitoring==2.18.0
google-cloud-orchestration-airflow==1.10.0
google-cloud-os-login==2.12.0
google-cloud-pubsub==2.19.0
google-cloud-redis==2.14.0
google-cloud-resource-manager==1.11.0
google-cloud-run==0.10.1
google-cloud-secret-manager==2.17.0
google-cloud-spanner==3.40.1
google-cloud-speech==2.23.0
google-cloud-storage==2.14.0
google-cloud-storage-transfer==1.10.0
google-cloud-tasks==2.15.0
google-cloud-texttospeech==2.15.0
google-cloud-translate==3.13.0
google-cloud-videointelligence==2.12.0
google-cloud-vision==3.5.0
google-cloud-workflows==1.13.0
google-crc32c==1.5.0
google-re2==1.1
google-resumable-media==2.7.0
googleapis-common-protos==1.62.0
graphviz==0.20.1
greenlet==3.0.2
grpc-google-iam-v1==0.13.0
grpcio==1.60.0
grpcio-gcp==0.2.2
grpcio-status==1.60.0
gunicorn==21.2.0
h11==0.14.0
httpcore==0.16.3
httplib2==0.22.0
httpx==0.23.3
humanize==4.9.0
hvac==2.0.0
idna==3.6
ijson==3.2.3
importlib-metadata==6.11.0
importlib-resources==6.1.1
inflection==0.5.1
iniconfig==2.0.0
isodate==0.6.1
isort==5.13.2
itsdangerous==2.1.2
Jinja2==3.1.2
jmespath==0.10.0
json-merge-patch==0.2
jsonpath-ng==1.6.0
jsonschema==4.20.0
jsonschema-specifications==2023.11.2
kombu==5.3.4
kubernetes==23.6.0
kubernetes-asyncio==24.2.3
lazy-object-proxy==1.10.0
ldap3==2.9.1
limits==3.7.0
linkify-it-py==2.0.2
lockfile==0.12.2
looker-sdk==23.20.0
lxml==4.9.3
Mako==1.3.0
Markdown==3.5.1
markdown-it-py==3.0.0
MarkupSafe==2.1.3
marshmallow==3.20.1
marshmallow-oneofschema==3.0.1
marshmallow-sqlalchemy==0.26.1
mdit-py-plugins==0.4.0
mdurl==0.1.2
msal==1.26.0
msal-extensions==1.1.0
msrest==0.7.1
msrestazure==0.6.4
multidict==6.0.4
mypy==1.8.0
mypy-extensions==1.0.0
mysql-connector-python==8.0.29
mysqlclient==2.2.1
numpy==1.24.4
oauthlib==3.2.2
openlineage-integration-common==1.6.2
openlineage-python==1.6.2
openlineage_sql==1.6.2
opentelemetry-api==1.22.0
opentelemetry-exporter-otlp==1.22.0
opentelemetry-exporter-otlp-proto-common==1.22.0
opentelemetry-exporter-otlp-proto-grpc==1.22.0
opentelemetry-exporter-otlp-proto-http==1.22.0
opentelemetry-proto==1.22.0
opentelemetry-sdk==1.22.0
opentelemetry-semantic-conventions==0.43b0
ordered-set==4.1.0
packaging==23.2
pandas==2.1.4
pandas-gbq==0.20.0
paramiko==3.3.1
pathspec==0.12.1
pendulum==2.1.2
platformdirs==3.11.0
pluggy==1.3.0
ply==3.11
portalocker==2.8.2
prison==0.2.1
prometheus-client==0.19.0
prompt-toolkit==3.0.43
proto-plus==1.23.0
protobuf==4.25.1
psutil==5.9.6
psycopg2-binary==2.9.9
pyarrow==11.0.0
pyarrow-hotfix==0.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pydantic==2.5.3
pydantic_core==2.14.6
pydata-google-auth==1.8.2
Pygments==2.17.2
PyJWT==2.8.0
PyNaCl==1.5.0
pyodbc==5.0.1
pyOpenSSL==23.3.0
pyparsing==3.1.1
pytest==7.4.3
pytest-cov==4.1.0
python-daemon==3.0.1
python-dateutil==2.8.2
python-dotenv==1.0.0
python-http-client==3.3.7
python-ldap==3.4.4
python-nvd3==0.15.0
python-slugify==8.0.1
pytz==2023.3.post1
pytzdata==2020.1
PyYAML==6.0.1
redis==4.6.0
redshift-connector==2.0.918
referencing==0.32.0
requests==2.31.0
requests-mock==1.11.0
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
rfc3339-validator==0.1.4
rfc3986==1.5.0
rich==13.7.0
rich-argparse==1.4.0
rpds-py==0.13.2
rsa==4.9
ruff==0.1.9
s3transfer==0.8.2
scramp==1.4.4
sendgrid==6.11.0
setproctitle==1.3.3
shapely==2.0.2
six==1.16.0
slack-sdk==3.26.1
sniffio==1.3.0
snowflake-connector-python==3.6.0
snowflake-sqlalchemy==1.5.1
sortedcontainers==2.4.0
soupsieve==2.5
SQLAlchemy==1.4.50
sqlalchemy-bigquery==1.9.0
SQLAlchemy-JSONField==1.0.2
sqlalchemy-redshift==0.8.14
sqlalchemy-spanner==1.6.2
SQLAlchemy-Utils==0.41.1
sqlparse==0.4.4
sshtunnel==0.4.0
starkbank-ecdsa==2.2.0
statsd==4.0.1
tabulate==0.9.0
tenacity==8.2.3
termcolor==2.4.0
text-unidecode==1.3
tomlkit==0.12.3
tornado==6.4
types-PyYAML==6.0.1
typing_extensions==4.9.0
tzdata==2023.3
uc-micro-py==1.0.2
unicodecsv==0.14.1
universal-pathlib==0.1.4
uritemplate==4.1.1
urllib3==2.0.6
vine==5.1.0
virtualenv==20.25.0
watchtower==3.0.1
wcwidth==0.2.12
websocket-client==1.7.0
Werkzeug==2.2.3
wrapt==1.16.0
WTForms==3.1.1
yarl==1.9.4
zipp==3.17.0
zope.event==5.0
zope.interface==6.1
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1958428350
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1879761764
Yeah. It's just strange changing provider version fixes it - but yeah it could be caused also by some changes to provider dependencies between those two versions:
```bash
[jarek:~/code/airflow] [apache-airflow] change-airflow-folder-detection-in-breeze+ 1 ± git diff providers-cncf-kubernetes/7.4.2..providers-cncf-kubernetes/7.12.0 -- airflow/providers/cncf/kubernetes/provider.yaml
```
```diff
diff --git a/airflow/providers/cncf/kubernetes/provider.yaml b/airflow/providers/cncf/kubernetes/provider.yaml
index 11a661e63e..f0b35ca392 100644
--- a/airflow/providers/cncf/kubernetes/provider.yaml
+++ b/airflow/providers/cncf/kubernetes/provider.yaml
@@ -22,7 +22,17 @@ description: |
`Kubernetes <https://kubernetes.io/>`__
suspended: false
+source-date-epoch: 1703288121
versions:
+ - 7.12.0
+ - 7.11.0
+ - 7.10.0
+ - 7.9.0
+ - 7.8.0
+ - 7.7.0
+ - 7.6.0
+ - 7.5.1
+ - 7.5.0
- 7.4.2
- 7.4.1
- 7.4.0
@@ -65,7 +75,8 @@ versions:
- 1.0.0
dependencies:
- - apache-airflow>=2.4.0
+ - aiofiles>=23.2.0
+ - apache-airflow>=2.6.0
- asgiref>=3.5.2
- cryptography>=2.0.0
# The Kubernetes API is known to introduce problems when upgraded to a MAJOR version. Airflow Core
@@ -100,7 +111,6 @@ integrations:
operators:
- integration-name: Kubernetes
python-modules:
- - airflow.providers.cncf.kubernetes.operators.kubernetes_pod
- airflow.providers.cncf.kubernetes.operators.pod
- airflow.providers.cncf.kubernetes.operators.spark_kubernetes
- airflow.providers.cncf.kubernetes.operators.resource
@@ -337,3 +347,6 @@ config:
type: string
example: ~
default: ""
+
+executors:
+ - airflow.providers.cncf.kubernetes.kubernetes_executor.KubernetesExecutor
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-2027836346
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #36571: KubernetesPodOperator prints empty log lines without respecting the log format
URL: https://github.com/apache/airflow/issues/36571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-2906523875
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1879619071
Exactly, the issue wasn't happening in 7.4.2, it has started happening in 7.12.0, although I have updated pretty much all the other providers as well as the airflow version at the same time. Let me test it by downgrading the cncf provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1879759250
@potiuk I also tried reproducing this as mentioned [here](https://github.com/apache/airflow/issues/36571#issuecomment-1876865865) and not able to repro with cncf 7.12.0, I also suspects it is being logged by some 3rd party lib.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1879756432
cc: @dstandish - likely #35416 was not THAT inconsequential
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1876865865
Agree log is very strange, @karakanb I am not able to replicate this with 2.8.0 & cncf-kubernetes==7.12.0, kindly do update us if you do not see this issue after disabling `TaskContextLogger`
`[2024-01-04, 09:15:41 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:42 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:43 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:44 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:45 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:46 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:47 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:48 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:49 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:50 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:51 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:52 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:53 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:54 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:55 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:56 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:57 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g
[2024-01-04, 09:15:58 UTC] {pod_manager.py:460} INFO - [base] hello world
[2024-01-04, 09:15:58 UTC] {pod_manager.py:488} WARNING - Pod passing-test-o9zwtu7g log read interrupted but container base still running
[2024-01-04, 09:15:59 UTC] {pod_manager.py:460} INFO - [base] hello world
[2024-01-04, 09:15:59 UTC] {pod.py:848} INFO - Deleting pod: passing-test-o9zwtu7g
[2024-01-04, 09:15:59 UTC] {taskinstance.py:1138} INFO - Marking task as SUCCESS. dag_id=k8pod_operator, task_id=passing-task, execution_date=20240104T091538, start_date=20240104T091541, end_date=20240104T091559
[2024-01-04, 09:15:59 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 0
[2024-01-04, 09:15:59 UTC] {taskinstance.py:3281} INFO - 0 downstream tasks scheduled from follow-on schedule check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1958914398
ah come on now, please stay open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1932916658
Please keep this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-2073690573
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-4565693546
Will be fixed by https://github.com/apache/airflow/pull/67652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-2995185956
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1984819568
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1876056360
This log is very strange; for example, for the pod starting wait, the check is running in a loop, and there are no other log statements:
https://github.com/apache/airflow/blob/2c15dc9dd1b360bcb3b6d6c2e41e50ef0d97801e/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L355-L366
These log messages come from something else, probably from the `TaskContextLogger` since you are using Airflow 2.8.0. Could you try to deactivate the feature and test the KPO?
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#enable-task-context-logger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1879536706
@karakanb, since you've upgraded the CNCF Kubernetes provider from version `7.4.2` to `7.12.0`, does that imply you weren't experiencing the issue with `7.4.2` and only observed it after the upgrade to `7.12.0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KubernetesPodOperator prints empty log lines without respecting the log format [airflow]
Posted by "karakanb (via GitHub)" <gi...@apache.org>.
karakanb commented on issue #36571:
URL: https://github.com/apache/airflow/issues/36571#issuecomment-1879638496
The issue disappeared when I went back to 7.4.2, which means it has nothing to do with Airflow 2.8.0 or my environment as far as I can tell. I have upgraded to v7.13.0 and the issue persists there as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org