You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/20 09:58:20 UTC
[GitHub] [airflow] joshuayeung opened a new issue, #23114: SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod
joshuayeung opened a new issue, #23114:
URL: https://github.com/apache/airflow/issues/23114
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==3.0.0
### Apache Airflow version
2.2.5 (latest released)
### Operating System
Debian GNU/Linux 10 (buster)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
When using `SparkKubernetesSensor` with `attach_log=True`, it cannot get the log correctly with the below error:
``` [2022-04-20, 08:42:04 UTC] {spark_kubernetes.py:95} WARNING - Could not read logs for pod spark-pi-0.4753748373914717-1-driver. It may have been disposed.
Make sure timeToLiveSeconds is set on your SparkApplication spec.
underlying exception: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '29ac5abb-452d-4411-a420-8d74155e187d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 20 Apr 2022 08:42:04 GMT', 'Content-Length': '259'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod spark-pi-0.4753748373914717-1-driver, choose one of: [istio-init istio-proxy spark-kubernetes-driver]","reason":"BadRequest","code":400}\n'
```
It is because no container is specified when calling kubernetes hook.get_pod_logs
https://github.com/apache/airflow/blob/501a3c3fbefbcc0d6071a00eb101110fc4733e08/airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py#L85
### What you think should happen instead
It should get the log of container `spark-kubernetes-driver`
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbenzikry commented on issue #23114: SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod
Posted by GitBox <gi...@apache.org>.
bbenzikry commented on issue #23114:
URL: https://github.com/apache/airflow/issues/23114#issuecomment-1103852329
@joshuayeung Yeah, it's derived from
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/74ea1c86c9a595e6c1c8a0c2c40a50bf89288ede/pkg/config/constants.go#L306
From what I remember while testing, there was some assurance of container ordering ( that is why the container name was omitted ) - even while using istio sidecars.
Can you tell me a bit about the setup / versions ( k8s, istio, operator, operator CRDs )?
Practically I can't be sure I'll have the time to submit a PR for this in the next few weeks - so feel free to open one to handle this.
Just hope the constant doesn't change in upcoming versions of the operator :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #23114: SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #23114: SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod
URL: https://github.com/apache/airflow/issues/23114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #23114: SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #23114:
URL: https://github.com/apache/airflow/issues/23114#issuecomment-1103737898
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joshuayeung commented on issue #23114: SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod
Posted by GitBox <gi...@apache.org>.
joshuayeung commented on issue #23114:
URL: https://github.com/apache/airflow/issues/23114#issuecomment-1103753611
@bbenzikry Do you know if the container name is always `spark-kubernetes-driver` in the driver pod?
If yes, can we just hardcode the container name or we can get the info from k8s hook response dynamically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hanna-liashchuk commented on issue #23114: SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod
Posted by GitBox <gi...@apache.org>.
hanna-liashchuk commented on issue #23114:
URL: https://github.com/apache/airflow/issues/23114#issuecomment-1264649173
The [PR](https://github.com/apache/airflow/pull/26560) is open, feel free to review :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org