You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/27 15:36:14 UTC
[GitHub] [airflow] mickaelgervais opened a new issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
mickaelgervais opened a new issue #10605:
URL: https://github.com/apache/airflow/issues/10605
** Use private docker repository with K8S operator and XCOM sidecar container **
An extra parameter to KubernetesPodOperator: docker_repository, this allows to specify the repository where the sidecar container is located
** My company force docker proxy usage for K8S **
I need to use my company docker repository, images that are not proxifed by the company docker repository are not allowed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] evanhlavaty commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
evanhlavaty commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-925215632
Is there a reason why this has not been implemented yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-682025919
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] fritzb commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
fritzb commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-731501546
Docker Inc started image pull rate limiting around October 30th 2020. As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we backport this fix to Airflow 1.10 branch?
Based on the severity of the failure, the bug priority should be raised higher.
` - image: alpine
imageID: ""
lastState: {}
name: airflow-xcom-sidecar
ready: false
restartCount: 0
started: false
state:
waiting:
message: 'rpc error: code = Unknown desc = Error response from daemon: toomanyrequests:
You have reached your pull rate limit. You may increase the limit by authenticating
and upgrading: https://www.docker.com/increase-rate-limit'
reason: ErrImagePull`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zoomzoomTnT commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
zoomzoomTnT commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-686642471
Hi @mickaelgervais
I have left a comment in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] evanhlavaty edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
evanhlavaty edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-941273076
@fritzb
> airflowLocalSettings: |
> from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZackingIt edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
ZackingIt edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-957069715
> > airflowLocalSettings: |
> > from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> > PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
>
> @fritzb
>
> I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
I was able to get my xcom to use a non-DockerHub repo also by creating a `airflow_local_settings.py` file in the root/config folder, due to this yaml:
https://github.com/apache/airflow/blob/73044af1c6908d1b075bd0127bc56c1db82b564b/chart/templates/_helpers.yaml#L416.
I then simply overwrote with the code:
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
You can verify the custom `airflow_local_settings.py` file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.
Also note that this works even as early as at least 2.1.0 for those of you who are having issues with 2.2.0 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzungdev commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
dzungdev commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-965896980
Hi @ZackingIt , could I know how do you overrote the code? Do you go to pod and change the airflow_local_settings.py file in root/config folder or you add it in values.yaml file as below:
`airflow
localSettings:
## the full content of the `airflow_local_settings.py` file (as a string)
stringOverride: |
# use a custom `xcom_sidecar` image for KubernetesPodOperator()
from airflow.kubernetes.pod_generator import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "private_artifactory/alpine"`
`from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zoomzoomTnT edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
zoomzoomTnT edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-686642471
Hi @mickaelgervais
I have left a comment in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.
Hi @potiuk
Is this feature going live in 1.10.13 or later versions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] fritzb edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
fritzb edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-731501546
Docker Inc started image pull rate limiting around October 30th 2020. As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we have the fix in Airflow 1.10 branch as well?
Based on the severity of the failure, the bug priority should be raised higher.
```
- image: alpine
imageID: ""
lastState: {}
name: airflow-xcom-sidecar
ready: false
restartCount: 0
started: false
state:
waiting:
message: 'rpc error: code = Unknown desc = Error response from daemon: toomanyrequests:
You have reached your pull rate limit. You may increase the limit by authenticating
and upgrading: https://www.docker.com/increase-rate-limit'
reason: ErrImagePull
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zoomzoomTnT commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
zoomzoomTnT commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-925279355
sure, I think it's covered in a different feature request, let me find out
and update the comments.
Can you let me know when are the workshops and how do I attend them?
Thanks,
Zhicong
On Wed, Sep 22, 2021 at 2:22 PM Jarek Potiuk ***@***.***>
wrote:
> Feel free to take it over and contribute! we even have Contributors
> workshops (3 of them till the end of the year). On Airflow Slack there is a
> #contributors-workshop dedicated channel for it! Anyone is welcome!
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/10605#issuecomment-925246273>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB52K2YS5FVVOH7QE55OSTTUDIUHXANCNFSM4QNED7HQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] fritzb commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
fritzb commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-925535664
Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:
values.yaml:
```
airflowLocalSettings: |
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZackingIt commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
ZackingIt commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-966571742
@dzungdev
I created an actual file, I did *not* overwrite a variable in the `values.yaml` i.e., part of my Docker build involves:
`cp /airflow-config/config/airflow_local_settings.py ${AIRFLOW_HOME}/config/airflow_local_settings.py`
When I say "due to this yaml" I mean that that yaml tells us that this is already a location,`${AIRFLOW_HOME}/config` is already one of the paths being read by default, as I did not need to explicitly set a PYTHONPATH environment variable to read that location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-683129444
would you like to add this feature @mickaelgervais ? I am happy to review it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexandrebunn commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
alexandrebunn commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-1014746127
We are facing the same issue, but in our case the error is related the absent of an option to define resources to the sidecar container. Our kubernetes cluster has this resource enforcement active and them even trying to set the resouces by airflow_local_settings.py, pod_template_file, it sounds like the code is ignoring these options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] evanhlavaty commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
evanhlavaty commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-941273076
> airflowLocalSettings: |
> from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzungdev edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
dzungdev edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-965896980
Hi @ZackingIt , could I know how do you overrote the code? Do you go to pod and change the airflow_local_settings.py file in root/config folder or you add it in values.yaml file as below:
airflow
localSettings:
## the full content of the `airflow_local_settings.py` file (as a string)
stringOverride: |
# use a custom `xcom_sidecar` image for KubernetesPodOperator()
from airflow.kubernetes.pod_generator import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "private_artifactory/alpine"`
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZackingIt edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
ZackingIt edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-957069715
> > airflowLocalSettings: |
> > from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> > PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
>
> @fritzb
>
> I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
I was able to get my xcom to use a non-DockerHub repo also by creating a `airflow_local_settings.py` file in the root/config folder, due to this yaml:
https://github.com/apache/airflow/blob/73044af1c6908d1b075bd0127bc56c1db82b564b/chart/templates/_helpers.yaml#L416.
I then simply overwrote with the code:
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
You can verify the custom `airflow_local_settings.py` file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.
Also note that this works even as early as at least 2.1.0 for those of you who are having issues with 2.2.0 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zoomzoomTnT edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
zoomzoomTnT edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-686642471
Hi @mickaelgervais
To understand your issue better, are you having trouble turning on xcom because 'alpine' give you 403 error? Because in your description, you are not able to use private registry, but in my case, I can pass the whole string to the image in my KubernetesPodOperator like below. I just have problem with the xcom-sidecar, because class PodDefaults hard coded image name to be 'alpine'.
`image="<private-registry>/<dir>/<dir>/<image>:<tag>"`
I have left some comments in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.
Hi @potiuk
Is this feature going live in 1.10.13 or later versions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] evanhlavaty edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
evanhlavaty edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-941273076
> airflowLocalSettings: |
> from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
@fritzb
I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZackingIt edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
ZackingIt edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-957069715
> > airflowLocalSettings: |
> > from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> > PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
>
> @fritzb
>
> I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
I was able to get my xcom to use a non-DockerHub repo also by creating a `airflow_local_settings.py` file in the root/config folder, due to this yaml:
https://github.com/apache/airflow/blob/73044af1c6908d1b075bd0127bc56c1db82b564b/chart/templates/_helpers.yaml#L416.
I then simply overwrote with the code:
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
You can verify the custom `airflow_local_settings.py` file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.
Also note that this works even as early as at least 2.1.0 for those of you who are having issues with 2.2.0 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lshw42 commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
lshw42 commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-968283182
@ZackingIt
Where exactly did you put the configuration? When I put the `PodDefaults.SIDECAR_CONTAINER.image` (as mentioned several times here and everywhere else) into the `values.yaml` it is just used by the helm chart. But I can't figure out where to put the `airflow_local_settings.py` to configure the worker pod image pulled by the `KubernetesExecutor`. I even can't build a Docker image with Airflow 2.1.4 which is configured as needed.
My current Dockerfile looks like the following:
```Dockerfile
FROM apache/airflow:2.1.4-python3.8@sha256:ae98e5e9eba596e24bce820e5f3a109274b82cd347bb9fdfca1e9d3f0dda4bb0
COPY --chown=airflow:root airflow_local_settings.py /opt/airflow/config/airflow_local_settings.py
USER airflow
```
But as soon as the `KubernetesExecutor` is pulling my image (because I configured it in the `values.yaml`) it runs into the Docker pull limit very easy, because I can't configure the image correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Shivarp1 edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
Shivarp1 edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-688014261
A work around would be to add this code block in the /home/airflow/airflow_local_settings.py
PodDefaults.SIDECAR_CONTAINER = k8s.V1Container(
name=PodDefaults.SIDECAR_CONTAINER_NAME,
command=['sh', '-c', PodDefaults.XCOM_CMD],
image='<local private repository image>',
volume_mounts=[PodDefaults.VOLUME_MOUNT],
resources=k8s.V1ResourceRequirements(
requests={
"cpu": "10m",
"memory":"32Mi"
},
limits={
"cpu": "0.25",
"memory":"32Mi"
}
),
)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mickaelgervais commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
mickaelgervais commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-683646867
Hi @potiuk ,
I've submitted a PR, do nopt hesitate to add comment or if I've implemented this badly.
The sidecar container declaration is low level in K8S operator so I've tried to not add complexity.
Regards
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-925244284
> Is there a reason why this has not been implemented yet?
It was waiting for someone like you to take that over and complete it possibly? Airflow has > 1700 contributors and you can become one if you want some features to be implemented faster :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-927082850
> Can you let me know when are the workshops and how do I attend them?
Just join the #contributors-workshop channel - we announce the workshops there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Shivarp1 commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
Shivarp1 commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-688014261
A work around would be add this code block in the /home/airflow/airflow_local_settings.py
PodDefaults.SIDECAR_CONTAINER = k8s.V1Container(
name=PodDefaults.SIDECAR_CONTAINER_NAME,
command=['sh', '-c', PodDefaults.XCOM_CMD],
image='<local private repository image>',
volume_mounts=[PodDefaults.VOLUME_MOUNT],
resources=k8s.V1ResourceRequirements(
requests={
"cpu": "10m",
"memory":"32Mi"
},
limits={
"cpu": "0.25",
"memory":"32Mi"
}
),
)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzungdev edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
dzungdev edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-965896980
Hi @ZackingIt , could I know how do you overrote the code? Do you go to pod and change the airflow_local_settings.py file in root/config folder or you add it in values.yaml file as below:
Your code
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
in values.yaml
```
airflow
localSettings:
## the full content of the `airflow_local_settings.py` file (as a string)
stringOverride: |
# use a custom `xcom_sidecar` image for KubernetesPodOperator()
from airflow.kubernetes.pod_generator import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "private_artifactory/alpine"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zoomzoomTnT edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
zoomzoomTnT edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-686642471
Hi @mickaelgervais
I have left some comments in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.
Hi @potiuk
Is this feature going live in 1.10.13 or later versions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] fritzb edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
fritzb edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-731501546
Docker Inc started image pull rate limiting around October 30th 2020. As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we backport this fix to Airflow 1.10 branch?
Based on the severity of the failure, the bug priority should be raised higher.
```
- image: alpine
imageID: ""
lastState: {}
name: airflow-xcom-sidecar
ready: false
restartCount: 0
started: false
state:
waiting:
message: 'rpc error: code = Unknown desc = Error response from daemon: toomanyrequests:
You have reached your pull rate limit. You may increase the limit by authenticating
and upgrading: https://www.docker.com/increase-rate-limit'
reason: ErrImagePull
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] fritzb edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
fritzb edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-731501546
Docker Inc started image pull rate limiting around October 30th 2020 (https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/). As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we have the fix in Airflow 1.10 branch as well?
Based on the severity of the failure, the bug priority should be raised higher.
```
- image: alpine
imageID: ""
lastState: {}
name: airflow-xcom-sidecar
ready: false
restartCount: 0
started: false
state:
waiting:
message: 'rpc error: code = Unknown desc = Error response from daemon: toomanyrequests:
You have reached your pull rate limit. You may increase the limit by authenticating
and upgrading: https://www.docker.com/increase-rate-limit'
reason: ErrImagePull
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-925246273
Feel free to take it over and contribute! we even have Contributors workshops (3 of them till the end of the year). On Airflow Slack there is a #contributors-workshop dedicated channel for it! Anyone is welcome!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZackingIt commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
ZackingIt commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-957069715
> > airflowLocalSettings: |
> > from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> > PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
>
> @fritzb
>
> I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
I was able to get my xcom to use a non-DockerHub repo also by creating a `airflow_local_settings.py` file in the root/config folder, due to this yaml:
https://github.com/apache/airflow/blob/73044af1c6908d1b075bd0127bc56c1db82b564b/chart/templates/_helpers.yaml#L416.
I then simply overwrote with the code:
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
You can verify the custom `airflow_local_settings.py` file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZackingIt commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
ZackingIt commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-957069715
> > airflowLocalSettings: |
> > from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> > PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
>
> @fritzb
>
> I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
I was able to get my xcom to use a non-DockerHub repo also by creating a `airflow_local_settings.py` file in the root/config folder, due to this yaml:
https://github.com/apache/airflow/blob/73044af1c6908d1b075bd0127bc56c1db82b564b/chart/templates/_helpers.yaml#L416.
I then simply overwrote with the code:
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
You can verify the custom `airflow_local_settings.py` file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ZackingIt commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
ZackingIt commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-957069715
> > airflowLocalSettings: |
> > from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> > PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
>
> @fritzb
>
> I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
I was able to get my xcom to use a non-DockerHub repo also by creating a `airflow_local_settings.py` file in the root/config folder, due to this yaml:
https://github.com/apache/airflow/blob/73044af1c6908d1b075bd0127bc56c1db82b564b/chart/templates/_helpers.yaml#L416.
I then simply overwrote with the code:
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
You can verify the custom `airflow_local_settings.py` file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check. That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] evanhlavaty edited a comment on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
evanhlavaty edited a comment on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-941273076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] evanhlavaty commented on issue #10605: Use private docker repository with K8S operator and XCOM sidecar container
Posted by GitBox <gi...@apache.org>.
evanhlavaty commented on issue #10605:
URL: https://github.com/apache/airflow/issues/10605#issuecomment-941273076
> airflowLocalSettings: |
> from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org