[GitHub] [airflow] potiuk commented on issue #14301: Helm Chart: Add Integration tests & expand unit tests coverage

[GitBox <gi...@apache.org>]

potiuk commented on issue #14301:
URL: https://github.com/apache/airflow/issues/14301#issuecomment-839503085


   Sure. We will need to make several configurations of tests for K8S. In short we need several different " modes" of running k8s - for now we only had one "KUBERNETES_MODE" : image . It was "legacy" and never really used beside some 1.5 years ago where we also had "git-sync" mode but I do not think we awnat to test that much granularity for integration tests so we can use this mode to become "deployment".
   
   We need several things here (and we can do it gradually and share it so that it's not only me who knows how to do it :)/
   
   1) we can change KUBERNETES_MODE  to determine type of deployment: "Calery", "Kubernetes", "CeleryKubernetes". The current "image" mode is actually nothing more than "Celery". Just look for KUBERNETES_MODE and kubernetes_mode everywhere (mostly scripts/ci, breeze , breeze-complete) and you should be able to replace the "image" with "Celery" for CI sp that we can commmit the change.
   
   2) In the `_kind.sh` in `kind::deploy_airflow_with_helm` we should deploy the right configuration of Aiflow based on the mode - basically overrid the values determining which K8S deployment we deply
   
   That will be a good start. At this stage we should be able to deploy Airflow via Helm Chart in different modes using Breeze - folowing https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes 
   
   You can first follow https://github.com/apache/airflow/blob/master/TESTING.rst#typical-testing-pattern-for-kubernetes-tests  to see how it is normally done.
   
   Once we have this, we should be able to run 
   
   * `./breeze kind-cluster start` (once to start the cluster)
   * `./breeze kind-cluster status` (once to start the cluster)
   * `./breeze kind-cluster deploy --kubernetes-mode CeleryKubernetes` 
   
   and you should be able to connect locally to Airflow running with CeleryKubernetes executor 
   
   Once we get there, we need to somehow decide and control which tests we run for which deployment. We can do it with custom pytest markers for example or separating them via sub-folders in kubernetes_tests.
   
   I think we can discuss this part when we get there.
   
   Does it sound reasonable? Happy to help/review/answer qustions on slack and get someone else to understand how it woks :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org