You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "raphaelauv (via GitHub)" <gi...@apache.org> on 2023/10/27 10:52:53 UTC
[PR] [FIX] remove K8S 1.24 support [airflow]
raphaelauv opened a new pull request, #35214:
URL: https://github.com/apache/airflow/pull/35214
end of GKE 1.24 support in 4 days -> https://endoflife.date/google-kubernetes-engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1783568685
> > I think we should consider [Alibaba Cloud](https://www.alibabacloud.com/help/en/ack/product-overview/support-for-kubernetes-versions) as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"
>
> If we include Alibaba (we did not when we set the policy), IMHO we should change to "at least 3 providers out of (GCP/AWS/Azure/Alibaba)" which does not change the outcome. PR maybe @hussein-awala explaining it ?
Okay, let's keep them 2 of the top 3, but IMHO, we should wait for the next minor version to do that, removing the support in a patch version doesn't make sense. If you agree, I will update the doc to explain what are the major cloud providers and when we can remove the support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1782821510
> ( USA 😅😅 )
I think we should consider [Alibaba Cloud](https://www.alibabacloud.com/help/en/ack/product-overview/support-for-kubernetes-versions) as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FIX] remove K8S 1.24 support [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1782772040
`[FIX] remove K8S 1.24 support` what this PR fix? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1786100382
> @hussein-awala until rules update , we could already merge this ( since it follow current K8S policy )
yes, I'm ok with that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1782806303
Airflow policy is "until at least 2 major ( USA 😅😅 ) cloud provider support it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1785063471
Yeah, both Amazon and Google have changed how they handle k8s versions more recently, so I'd be open to revisit this, but it should probably be an on-list discussion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1785024167
> since cloud providers are way more up to date with K8S versions and also Apache airflow is not astronomer and we could have the same policy that we have with python and postgres
I do not think the rule we have is "Astronomer" rule - it's been proposed by @ashb quite long time ago https://lists.apache.org/thread/fr3tdy4nc8ywo3p0hpy9h04nyv1kldsy, and a few people agreed it makes sense back then without opposal (coincidentally they were all from Astronomer + me :D) .
But the discussion has (as I re-read it now) quite a good justification on why keep it until it is likely a number of people still use it. A lot of people keep they old clusters running for as long as they can and only forced migration will keep them to move. I believe k8s is a bit different than say "Postgres" - in the sense that it has many more internal components that might make things more difficult to migrate than say migrating the database. K8S is a conglomerate of about a hundred or so different - independently released components internally so there might be one small single thing that might prevent particular installation to migrate to a newer version without some serious pains.
But I agree, it would be quite a bit more consistent to have "EOL" as a cut-off time for "software" we release - because we do not run "service".
Maybe worth to restart the discussion on devlist and see what people think about it TODAY @raphaelauv ?. I'd support changing it to EOL only.
Just personally - I am (for now) generally fine with both approaches but I'd prefer following the "EOL" route. I see why it is good to keep it a little longer, but also I see how strictly following EOL from K8S might actually drive people to update things.
But this might be mandated in the future actually, so we might want to do it even now anticipating what will happen.
Putting my `security` hat on - we will likely in the future HAVE TO add the rule of not supporting EOL software because of security regulations. We cannot rely on Amazon or Google releasing fixes to K8S "to the public" and when we "put software on the market" (this is what making a release does) we have to make sure we do it without relying on software that does not have security fixes maintenance.
The landscape is changing a lot with Cyber Resilience Act and similar US legislations that all are going to be codified in 2024 and will likely have to be implemented 2/3 years from the moment they pass.
It will be legally and financially costly for companies NOT to upgrade the software with latest security fixes and use software that is known it will not receive security fixes. So basically what it might mean (this is what is the spirit of the regulations at least - the regulations and standards are not yet set in stone) that if someneone finds a fix in K8S that is EOL and we (Airflow) rely on it, we (Airflow/ASF) will have to (somehow) make sure that our users will get a fix.
Making it clear that we do not support EOL software, removes that obligation from us. It is of course pretty messy with smaller libraries and their dependencies etc. But there, vendoring-in and relasing security fixes is much easier for us even if we need to, rather than advising our customers on how to patch their k8s. So explicitly marking EOL software as not supported in latest version of Airflow is simply smart thing to do to prevent any responsibilities we might get.
I think we will see our industry shifting in the coming years from "costly to migrate so let's not think about it" to "costly to NOT migrate so let's actively manage it". But we will yet see what effect those regulations might have. But anticipating that - I think using EOL as cut-off day for us is a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1785092783
@hussein-awala until rules changed , we could already merge this ( since it follow current K8S policy )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1782777298
ok , I remove the prefix "fix"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1783309285
> I think we should consider [Alibaba Cloud](https://www.alibabacloud.com/help/en/ack/product-overview/support-for-kubernetes-versions) as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"
Is we include Alibaba (we did not when we set the policy), IMHO we should change to "at least 3 major providers" which does not change the outcome. PR maybe @hussein-awala explaining it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1783913253
> > > I think we should consider [Alibaba Cloud](https://www.alibabacloud.com/help/en/ack/product-overview/support-for-kubernetes-versions) as a major, no? Otherwise, we should replace "until at least 2 major" with "until at least 2 of (GCP, AWS, Azure)"
> >
> >
> > If we include Alibaba (we did not when we set the policy), IMHO we should change to "at least 3 providers out of (GCP/AWS/Azure/Alibaba)" which does not change the outcome. PR maybe @hussein-awala explaining it ?
>
> Okay, let's keep them 2 of the top 3, but IMHO, we should wait for the next minor version to do that, removing the support in a patch version doesn't make sense. If you agree, I will update the doc to explain what are the major cloud providers and when we can remove the support.
Absolutely Feel free and we can - I guess keep the discussion in PR and get few approvals for that.
Re - which version we drop support, we did not even plan to remove it in Patchlevel release. From our docs:
> We drop support for those EOL versions in main right after EOL date, and it is effectively removed when we release the first new MINOR (Or MAJOR if there is no new MINOR version)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1785119840
> Yeah, both Amazon and Google have changed how they handle k8s versions more recently, so I'd be open to revisit this, but it should probably be an on-list discussion?
Absolutely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1786099968
> Yeah, both Amazon and Google have changed how they handle k8s versions more recently, so I'd be open to revisit this, but it should probably be an on-list discussion?
+1 to discuss this on dev list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #35214:
URL: https://github.com/apache/airflow/pull/35214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] remove K8S 1.24 support [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #35214:
URL: https://github.com/apache/airflow/pull/35214#issuecomment-1784900472
we could just remove
```
Except for Kubernetes, a version stays supported by Airflow if two major cloud providers still provide support for it
```
since cloud providers are way more up to date with K8S versions and also Apache airflow is not astronomer and we could have the same policy that we have with python and postgres
wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org