You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "repl-mike-roest (via GitHub)" <gi...@apache.org> on 2023/02/09 10:19:30 UTC
[GitHub] [airflow] repl-mike-roest opened a new issue, #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
repl-mike-roest opened a new issue, #21943:
URL: https://github.com/apache/airflow/issues/21943
### Official Helm Chart version
1.4.0 (latest released)
### Apache Airflow version
2.2.4 (latest released)
### Kubernetes Version
1.21.5 (EKS)
### Helm Chart configuration
We are using a external RDS DB server configured via secrets.
Also have specified
```
airflowVersion: 2.2.3
defaultAirflowTag: 2.2.3
```
Along with these flags As without them while deploying via codebuild the chart was never progressing as the create-user/run-db-migrations jobs were not running
```
createUserJob:
useHelmHooks: false
migrateDatabaseJob:
useHelmHooks: false
```
### Docker Image customisations
Happens both with a transition from airflow default image 2.2.3 -> 2.2.4 and with changing our custom image between versions or from a default airflow image to our custom image
### What happened
The following error was returned from the helm upgrade command
```
Error: UPGRADE FAILED: cannot patch "pre-production-create-user" with kind Job: Job.batch "pre-production-create-user" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"component":"create-user-job", "controller-uid":"52e67857-b3f0-414c-b176-3027c93e4a05", "job-name":"pre-production-create-user", "release":"pre-production", "tier":"airflow"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:"config", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.Empty
DirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(0xc0107dfdc0), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDi
skVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"create-user", Image:"434423891815.dkr.ecr.us-west-2.amazonaws.com/airflow-playground/airflow:b-23-IP2-51", Command:[]string(nil), Args:[]string{"bash", "-c", "airflow users create \"$@\"", "--", "-r", "Admin", "-u", "admin", "-e", "admin@example.com", "-f", "admin", "-l", "user", "-p", "DFPGxku#V#&h{C:)qiOmta3s"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{}, Env:[]core.EnvVar{core.EnvVar{Name:"AIRFLOW__CORE__FERNET_KEY", Value:"", ValueFrom:(*core.EnvVarSource)(0xc016c97560)}, core.EnvVar{Name:"AIRFLOW__CORE__SQL_ALCHEMY_CONN", Value:"", ValueFrom:(*core.EnvVarSource)(0xc016c9758
0)}, core.EnvVar{Name:"AIRFLOW_CONN_AIRFLOW_DB", Value:"", ValueFrom:(*core.EnvVarSource)(0xc016c975a0)}, core.EnvVar{Name:"AIRFLOW__WEBSERVER__SECRET_KEY", Value:"", ValueFrom:(*core.EnvVarSource)(0xc016c975c0)}, core.EnvVar{Name:"AIRFLOW__CELERY__CELERY_RESULT_BACKEND", Value:"", ValueFrom:(*core.EnvVarSource)(0xc016c97600)}, core.EnvVar{Name:"AIRFLOW__CELERY__RESULT_BACKEND", Value:"", ValueFrom:(*core.EnvVarSource)(0xc016c97620)}, core.EnvVar{Name:"AIRFLOW__CELERY__BROKER_URL", Value:"", ValueFrom:(*core.EnvVarSource)(0xc016c97640)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:"config", ReadOnly:true, MountPath:"/opt/airflow/airflow.cfg", SubPath:"airflow.cfg", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil),
Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc01a548358), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{}, ServiceAccountName:"pre-production-airflow-create-user-job", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc01771ca80), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(0xc01b38ff20), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration{}, HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfi
g:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable && cannot patch "pre-production-run-airflow-migrations" with kind Job: Job.batch "pre-production-run-airflow-migrations" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"component":"run-airflow-migrations", "controller-uid":"19d78edd-2df2-4f61-ba6b-01592d103327", "job-name":"pre-production-run-airflow-migrations", "release":"pre-production", "tier":"airflow"}, Annotations:map[string]string(nil), OwnerReferences:
[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:"config", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(nil), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*cor
e.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(0xc01a11e200), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"run-airflow-migrations", Image:"434423891815.dkr.ecr.us-west-2.amazonaws.com/airflow-playground/airflow:b-23-IP2-51", Command:[]string(nil), Args:[]string{"bash", "-c", "airflow db upgrade"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{}, Env:[]core.EnvVar{core.EnvVar{Name:"AIRFLOW__CORE__FERNET_KEY", Value:"",
ValueFrom:(*core.EnvVarSource)(0xc00cd021c0)}, core.EnvVar{Name:"AIRFLOW__CORE__SQL_ALCHEMY_CONN", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cd02200)}, core.EnvVar{Name:"AIRFLOW_CONN_AIRFLOW_DB", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cd02220)}, core.EnvVar{Name:"AIRFLOW__WEBSERVER__SECRET_KEY", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cd02260)}, core.EnvVar{Name:"AIRFLOW__CELERY__CELERY_RESULT_BACKEND", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cd02280)}, core.EnvVar{Name:"AIRFLOW__CELERY__RESULT_BACKEND", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cd022c0)}, core.EnvVar{Name:"AIRFLOW__CELERY__BROKER_URL", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cd022e0)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:"config", ReadOnly:true, MountPath:"/opt/airflow/airflow.cfg", SubPath:"airflow.cfg", MountPropagation:(*core.MountPropagationMode)(nil), SubPath
Expr:""}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc01bc01a08), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{}, ServiceAccountName:"pre-production-airflow-migrate-database-job", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc008e4f300), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(0xc0127b9ad0), SchedulerName:"default-scheduler", Tolerations:[
]core.Toleration{}, HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable
```
### What you expected to happen
The helm chart to successfully upgrade and change my running images to a new version
### How to reproduce
deploy helm chart with
```
airflowVersion: 2.2.3
defaultAirflowTag: 2.2.3
```
in your values.yaml
and the following command
helm upgrade --install --wait --timeout 900s pre-production apache-airflow/airflow --namespace airflow --version 1.4.0 \
-f values.yaml
Then run the same command after changing the image tags to 2.2.4
### Anything else
Seems to happen whenever we change the image tag (even within the same release) if we're using a custom image that contains our dags changing from one tag to the other gets the same error.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mconigliaro commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by GitBox <gi...@apache.org>.
mconigliaro commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1343331982
I think https://github.com/apache/airflow/pull/27148 might be the fix for this. It adds a `createUserJob.applyCustomEnv` option which we should probably set to `false`. We're just waiting on a new release of the helm chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477759549
Of course - aee https://github.com/apache/airflow/issues/21943#issuecomment-1475948017 and the usual Helm chart things. Helm chart is just a folder you can install. You have install instructions in https://github.com/apache/airflow/tree/main/chart (INSTALL) or you can host it yourself somewhere. This is where manual patching (or using latest sources) come into play, Just check it out, patch the changes (or use lates main) and use helm installation from there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477924337
No - not until we release it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] grjones commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by GitBox <gi...@apache.org>.
grjones commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1343031054
@repl-mike-roest Curious if you ever found a permanent resolution for this. We have this in terraform, so we can't easily remove the `useHelmHooks: false` flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477928315
cc: @jedcunningham @ephraimbuddy @pierrejeambrun -> FYI, might be useful to determine on when to release the new chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1423966074
I think https://github.com/apache/airflow/pull/29439 should handle it long term. Seems that it is a known issue with K8S @jay-olulana https://github.com/kubernetes/kubernetes/issues/89657 and it has been fixed in 1.23 by adding `ttlSecondsAfterFinished`.
There is no automated way for you to recover, but you can do it manually if I am right:
* have k8s 1.23+
* apply my PR to your chart
* nuke the chart - remove it. Since you have Terraform, that should be easy way and redeploying it should restore it.
* alternatively remove the affected job manually using kubectl or the like
* redeploy the chart with the fix
Once you redeploy the chart with the PR including the `ttlSecondsAfterFinished` - the finished job should get deleted automatically after ~5 minutes (you can also decrease the ttl before deploying it).
I would appreciate @jay-olulana if you could test some scenarios involved and confirm that my proposed fix works for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1471955698
> ttlSecondsAfterFinished: 300
Did you wait 6 minutes before updating the tag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elongl commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "elongl (via GitHub)" <gi...@apache.org>.
elongl commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477911107
Thanks again, that really helped.
Just wanted to confirm that using the chart of the `main` branch did in fact fix the issue for me and is working well now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elongl commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "elongl (via GitHub)" <gi...@apache.org>.
elongl commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477671576
@potiuk Thanks a lot for sharing the context on the Helm releases.
Yes, I'd definitely love to help, though I'm not yet sure on how can I test the RC locally.
Is there a guide on doing so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jay-olulana commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "jay-olulana (via GitHub)" <gi...@apache.org>.
jay-olulana commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1471890234
Hi @potiuk, Sorry that the reply is late. But I did apply your fix after upgrading to `Helm 1.8.0`, but the error persists. Although I think this is a `false positive`.
Below are the steps I took;
- As you know, I have airflow deployed via `helm` using terraform to an EKS Cluster.
- EKS Cluster has k8s `1.23`
- Helm Chart Version for Airflow is `1.8.0`
- I added the following to my `overrides.yaml`
```yaml
createUserJob:
useHelmHooks: false
applyCustomEnv: false
migrateDatabaseJob:
useHelmHooks: false
applyCustomEnv: false
ttlSecondsAfterFinished: 300
```
- Then I nuked the chart (destroyed the entire `airflow` namespace on EKS)
- Recreated airflow with an image tag. Say, `v1.1.0`
- Updated the image tag with a new version, `v1.1.1`
- Run `terraform apply` and this error pops up again;
**Good News** is that my airflow pods are recreated in EKS with the new tags and are healthy.
But terraform fails the apply.
Hope this provides some insights.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jay-olulana commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "jay-olulana (via GitHub)" <gi...@apache.org>.
jay-olulana commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1472078514
Yes, I did (more than even). But I will run more tests this weekend and let you know if it persists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jay-olulana commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "jay-olulana (via GitHub)" <gi...@apache.org>.
jay-olulana commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1423853492
> I think #27148 might be the fix for this. It adds `createUserJob.applyCustomEnv` and `migrateDatabaseJob.applyCustomEnv` options which we should set to `false`. We're just waiting on a new release of the helm chart.
The helm chart is released now, and this doesn't solve the problem. Updating the `image-tag` still fails the terraform apply with the `subject` error.
<img width="983" alt="image" src="https://user-images.githubusercontent.com/74787711/217766388-0cf2880f-ab79-4c3b-9ece-55946d0f4450.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1472142471
> Yes, I did (more than even). But I will run more tests this weekend and let you know if it persists.
Please - also you can check if the ttls is observed the job should disappear after 5 minutes, so if it is still there, maybe for some reason the version of K8S you run it has it disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elongl commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "elongl (via GitHub)" <gi...@apache.org>.
elongl commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477915356
Is there an alternative to including it into my version control?
Preferably I'd like to have a URL that returns the chart which seems like something that's supported by [`helm_release`](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release#example-usage---chart-url) but not by GitHub as far as I can tell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477661758
> @potiuk Is there any particular reason the chart is not released yet? I think it'd ease on deployment. Here are the solutions that I'm considering at the moment:
The image is released semi-regularly when release managers decide to relese it (I am not one for Helm Chart BTW). I think asking "reason for not releasing" is a wrongly asked question. It takes time and effort to publish the release. And it is done by volunteers when they see the time is good for it, and one issue affecting small group of users might not be enough to warrant it.
I think the right question you could ask is "what can I do to help with speeding up the release". Let me answer this question instead. I think if you confirm that the change fixes the problem by applying the changes locally and confirming it here, it might definitely increase the chances that release managers will make a decision about releasing the helm chart.
Also - as a follow up (after you confirrm it) it would immensely help if you help testing the release candidate. Subscribe to the devlist to get announcement about it and whenever we release an RC for chart, we ask people to test it and confirm that it works. I looked it up and I have not seen your help in
https://github.com/apache/airflow/issues?q=is%3Aissue+%22status+of+testing+Apache+Airflow+Helm+Chart%22+ so I think that is a great idea to get involved and help with it when we do release it.
Can we count on your help there @elongl to verify and confirm it and then later take part in testing when an RC is out? That would certainly help to speed up the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "<name>-create-user" with kind Job: Job.batch "<name>-create-user"
URL: https://github.com/apache/airflow/issues/21943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1475948017
Note that this fix has not yet been released and requires manual patching and recreating your deployment from scratch.
Is `ttlSecondsAfterFinished` part of your job definition @jay-olulana ? (https://github.com/apache/airflow/pull/29314/files)? Can you double check it? In order to use it, you will have to either use main version of the chart or apply the change manually to your chart and only after that deploy your chart (with the changeS) from the scratch. And your k8s version would have to support ttlSecondsAfterFinished. Only then any NEXT upgrade (assuming it is done later than the `ttlSecondsAfterFinished` seconds after the job finishes) should work. Simply K8S would automatically delete the completed job after this time if everything aligns.
So you need to make sure that your chart contains the changes, that your job gets the spec parameter and that k8s handles it.
If those conditions are not fulfilled you can always redeploy Airflow from the scratch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] grjones commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by GitBox <gi...@apache.org>.
grjones commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1343481171
@mconigliaro Thank you kindly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elongl commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "elongl (via GitHub)" <gi...@apache.org>.
elongl commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477642963
@potiuk Is there any particular reason the chart is not released yet? I think it'd ease on deployment.
Here are the solutions that I'm considering at the moment:
- Use `postrender` on `helm_release` and edit the `ttlSecondsAfterFinished `.
- Use `wait_for_jobs` on `helm_release`, and then `kubectl delete job` afterwards.
- Run `kubectl wait` followed by `kubectl delete job`.
- Clone and use the pre-release chart that has support for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477761229
And when RC is out you will also be able to install it following the RC instructions https://github.com/apache/airflow/blob/main/dev/README_RELEASE_HELM_CHART.md#verify-release-candidates-by-contributors
Those instructions are posted every time RC is out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1477926374
OK. Closing since it is confirmed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jay-olulana commented on issue #21943: Changing Image tags causes Error: UPGRADE FAILED: cannot patch "-create-user" with kind Job: Job.batch "-create-user"
Posted by "jay-olulana (via GitHub)" <gi...@apache.org>.
jay-olulana commented on issue #21943:
URL: https://github.com/apache/airflow/issues/21943#issuecomment-1475841200
Okay so I ran some tests;
I am going to provide you with more context;
This is a snapshot of our `airflow` namespace in our `test` env (using `k9s`):
The ⬆️ , was after killing the namespace and then reapplying;
A screenshot of the jobs at recreation of the namespace;
Then a screenshot of the jobs after `ttls`;
Updating the tags still fails terraform. So maybe the k8s version doesn't allow this deletion of the `create-user` job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org