[GitHub] [airflow] potiuk commented on issue #24456: Add Snowpark Operator that allows execution of Snowflake Snowpark code

[GitBox <gi...@apache.org>]

potiuk commented on issue #24456:
URL: https://github.com/apache/airflow/issues/24456#issuecomment-1217737310

   Just to explain the process @sfc-gh-madkins (@mik-laj already knows most of it) - but I think it would be good that you understand this and can arrange your process with your customers as well, so that there are no surprises and so that you can adapt to the rules we agreed in the community.
   
   The rule we have for Airflow providers is that they should be 2.2+ compatible till 11th of October:
   https://github.com/apache/airflow/blob/main/README.md#release-process-for-providers :
   
   > For example this means that by default we upgrade the minimum version of Airflow supported by providers to 2.3.0 in the first Provider's release after 11th of October 2022 (11th of October 2021 is the date when the first PATCHLEVEL of 2.2 (2.2.0) has been released.
   
   We are attemting to test the 2.2 compatibility basically with every PR - at laast to the level of whether the provider "imports" well on airflow 2.2: https://github.com/apache/airflow/runs/7872249712?check_suite_focus=true . The API (not yet formalized but soon will be when we split providers out to separate repos) between the Providers and Airflow are rather stable ones and we are very carefully reviewing any parts that might break potentially. But accidental breakages happen (which we fix if we find them).  Ideally, those errors are detected during manual testing of RC candidates and for that we kindly ask people employed by the stakeholders and those who are involved in preparing changes to test it when we release RC candidates.  There are at least 72 hours to test the releases and you are (including yourself @sfc-gh-madkins) heavily encouraged to run those tests when we release the provider. But involving your customer is also encouraged.
   
   We announce it at the devlist (roughly monthly) with some early announcements that we are about to prepare the new wave of providers and we create an issue like this one when RC candidates are ready, where you can coment and raise issues or confirm that things work as expected (the latter is very encouraged :): https://github.com/apache/airflow/issues/25640 when we prepare RC candidates. The last one is a good example where Databricks employees tested the Databricks provider and found out a slight Airlfow 2.2 compatibility issue (so we have RC4 candidate of databricks now with it fixed). You can see the discussion to see how it can play out.
   
   So - to summarise - the best way to be "sure" that a provider works on Airflow 2.2 is to help the community by performing the tests of RC candidates on 2.2 - and possibly even engage your customers to run such testing with the release candidates. And this is also a good answer to the customers of yours - if they ask if it works with earlier airflow version, the best answer is "yes, they are intended to be, but the best way to make sure of that is if you can help with testing the RC candidate" and inform them when the RC is out :).The RCs are installable from PyPI - same as any other versions - but they need to be installed deliberately, specifying the version to install.
   
   One important note. As mentioned above, the last 2.2-compatible release of providers will be at the beginning of October, so the November wave of providers will have 2.3+ compatibilty only. Then all providers will have their major version bumped.
   There is a way for Snowflake and (for any other stakeholders) to express their willingness to help with cherry-picking of some bugfixes and features to an earlier version of a provider from the previous line as described uhere. So if Snowflake would like to do it, it's possible, but it would have to take on the burden of preparing a branch made of cherry-picked bugfixes and features and testing it before we release it. See  https://github.com/apache/airflow/blob/main/README.md#release-process-for-providers for details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org