You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/09 12:58:59 UTC

[GitHub] [airflow] jensgrnb opened a new issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

jensgrnb opened a new issue #20168:
URL: https://github.com/apache/airflow/issues/20168


   ### Apache Airflow version
   
   2.2.1
   
   ### What happened
   
   Seems to only affect KubernetesExecutor jobs
   
   ```
   Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UndefinedColumn: column dag_run.data_interval_start does not exist
   ```
   
   ### What you expected to happen
   
   I expected that the pod would be started without problems. 
   
   ### How to reproduce
   
   Try running a Kubernetes job using the CeleryKubernetesExecutor. 
   
   ### Operating System
   
   PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
   
   ### Versions of Apache Airflow Providers
   
   airflow@2.2.1
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   google kubernetes engine deployment
   
   ### Anything else
   
   We upgraded from 2.1.2 to 2.2.1 and have not experienced any issues with the normal celeryexecutor jobs. 
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993691255


   > That's all true, but like what if apt worked the same way, and I specified some version of a package that had a dependency on an older version of linux. Would it just casually overwrite my OS?
   
   BTW. If you run `dist-upgrade` yes. The tools are doing what you tell them. This is the same story.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989924490


   Yeah. You have to keep your pod template and base image synced after the update. 
   
   I think this will be handled much better soon with #18439 - when DB migrations will be involved and not applied, your K8S job should fail with an appropriate message. But I think this also comes back to our earlier discussion @kaxil  @jedcunningham  @dimberman @ephraimbuddy  (with migration containers). The case which I am worried about is what happens if there are two different Airlow versions In KE and airflow when there is no "migration" between them?
   
   Should we flag that as an error? Should we check if the airflow version is right in the image? I think that would make sense to pass Airflow version when KE submits the job and we should at least check if the versions match, because there might be cases like this one that will be very difficult to debug otherwise.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993681366


   > That's all true, but like what if apt worked the same way, and I specified some version of a package that had a dependency on an older version of linux. Would it just casually overwrite my OS?
   
   1. First of all - this is how PIP works
   2. Secondly. This is also how APT works. If you have dependency that reguires newer version of another dependency it will upgrade it
   3. Thirdly. All package managers work like that.
   
   Airflow is not an OS. Airflow is an application that is installed on top of OS (debian Buster in this case). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993527948


   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   The big difference is that when you have your Dockerfile and resulting image, you  build your image in your CI/CD pipeline and you can run checks for it. You could for example verify airflow version, verify whether all imports are done correctly, verify if basic commands work. 
   
   All that LONG before you start hiting that in production. This is the best  common sense engineering practice. And this is the main reason why containers, isolation has been introduced for. To make a "frozen" environment that you can securely test and deploy in production. If you are using Helm/K8S/Docker but then you try to use it as if it was "local venv" - you are using the worst of both worlds - volatitlty,  lack of guarantees, lack of stability and no opportunity to test things before deploying - all those are coming with containers but you refuse to make use of it, at the same time terribly complicating your setup. 
   
   It's literallly like using a super-modern automated drilll to hit a nail. Not a good idea. Yes you can do it and it will do the job eventually. But it's quite expensive and dangerous thing to do. 
   
   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   I have no idea how the `extraPipPackages`  work under the hood (and no desire to learn). We deliberately refused to even  consider implementing it when asked by the users in the  [official Airflow Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) - anticipating this kind of problems . I think this is more for @thesuperzapper to debug. PIP has some switches (for example `--eager-upgrade` upgrade strategy, that might cause it, or maybe it is simply a result of using `--upgrade` in the pip install.  No idea. No desire to find out. Don't do it.
   
   If you on the other hand build your own image and follow the [nice documentation](https://airflow.apache.org/docs/docker-stack/build.html) we have with plenty of examples, where you directly control the `pip` commands used - I am quite sure you would not have the same problem. Even if you had - let me repeat it again - you would have a chance to find out and debug before it automatically migrated your database.
   
   And this is what I heartily recommend.
   
   BTW. Docker image testing is precisely what we do in our CI/CD for airflow. With every single PR in Airflow we verify that the image does what it is supposed to do. Heartily recommend. See here for an inspiration (code available https://github.com/apache/airflow/blob/main/docker_tests/test_prod_image.py):
   
   https://github.com/apache/airflow/runs/4519400249?check_suite_focus=true#step:6:310
   
   ```
   docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   [gw0] [  3%] PASSED docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   [gw1] [  6%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   docker_tests/test_prod_image.py::TestCommands::test_python_version 
   [gw1] [ 10%] PASSED docker_tests/test_prod_image.py::TestCommands::test_python_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   [gw0] [ 13%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   [gw1] [ 16%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   [gw0] [ 20%] PASSED docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   [gw1] [ 23%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   [gw0] [ 26%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   [gw1] [ 30%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   [gw0] [ 33%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   [gw1] [ 36%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   [gw0] [ 40%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   [gw1] [ 43%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   [gw0] [ 46%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   [gw0] [ 50%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   [gw1] [ 53%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   [gw0] [ 56%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   [gw1] [ 60%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   [gw0] [ 63%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   [gw1] [ 66%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   [gw0] [ 70%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   [gw1] [ 73%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   [gw0] [ 76%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   [gw1] [ 80%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   [gw0] [ 83%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   [gw1] [ 86%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   [gw0] [ 90%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   [gw1] [ 93%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw1] [ 96%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw0] [100%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thesuperzapper commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
thesuperzapper commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994100601


   @potiuk the pip install is very simple, it just uses `pip install --user {package_1} {package_2}`.
   
   [Here is the `pip install` code from the "user-community" chart.](https://github.com/airflow-helm/charts/blob/e1d49498426add959350ab8efacead8f96400759/charts/airflow/templates/_helpers/pods.tpl#L177-L180)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990769271


   We pull directly from dockerhub: https://hub.docker.com/layers/apache/airflow/2.2.1-python3.8/images/sha256-e0dd0f34848d0d87e6a372507ebcd844028e095ec29e95c4569dcb5a2042fc84?context=explore
   
   and run airflow cmds to set up the webserver, worker, etc. 
   
   We use this helm chart: https://github.com/airflow-helm/charts/tree/main/charts/airflow
   
   Is it even possible to run a different airflow version, than the one specified in the docker tag?
   
   From my grep and airflow version cmds in the worker pod it also seems like airflow itself is a bit divided on which version it's running..
   
   Funny thing is that we upgraded from 2.1.2, not 2.1.4... So 2.1.4 has never intentionally been deployed by us...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993527948


   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   The big difference is that when you have your Dockerfile and resulting image, you  build your image in your CI/CD pipeline and you can run checks for it. You could for example verify airflow version, verify whether all imports are done correctly, verify if basic commands work. 
   
   All that LONG before you start hiting that in production. This is the best  common sense engineering practice. And this is the main reason why containers, isolation has been introduced for. To make a "frozen" environment that you can securely test and deploy in production. If you are using Helm/K8S/Docker but then you try to use it as if it was "local venv" - you are using the worst of both worlds - volatitlty,  lack of guarantees, lack of stability and no opportunity to test things before deploying - all those are coming with containers but you refuse to make use of it, at the same time terribly complicating your setup. 
   
   It's literallly like using a super-modern automated drilll to hit a nail. Not a good idea. Yes you can do it and it will do the job eventually. But it's quite expensive and dangerous thing to do. 
   
   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   I have no idea how the `extraPipPackages`  work under the hood (and no desire to learn). We deliberately refused to even  consider implementing it when asked by the users in the  [official Airflow Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) - anticipating this kind of problems . I think this is more for @thesuperzapper to debug. PIP has some switches (for example `--eager-upgrade` upgrade strategy, that might cause it, or maybe it is simply a result of using `--upgrade` in the pip install.  No idea. No desire to find out. Don't do it.
   
   If you on the other hand build your own image and follow the [nice documentation](https://airflow.apache.org/docs/docker-stack/build.html) we have with plenty of examples, where you directly control the `pip` commands used  I am quite sure you would not have the same problem. Even if you had - let me repeat it again - you would have a chance to find out and debug before it automatically migrated your database.
   
   And this is what I heartily recommend.
   
   BTW. Docker image testing is precisely what we do in our CI/CD for airflow. With every single PR in Airflow we verify that the image does what it is supposed to do. Heartily recommend. See here for an inspiration (code available https://github.com/apache/airflow/blob/main/docker_tests/test_prod_image.py):
   
   https://github.com/apache/airflow/runs/4519400249?check_suite_focus=true#step:6:310
   
   ```
   docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   [gw0] [  3%] PASSED docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   [gw1] [  6%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   docker_tests/test_prod_image.py::TestCommands::test_python_version 
   [gw1] [ 10%] PASSED docker_tests/test_prod_image.py::TestCommands::test_python_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   [gw0] [ 13%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   [gw1] [ 16%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   [gw0] [ 20%] PASSED docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   [gw1] [ 23%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   [gw0] [ 26%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   [gw1] [ 30%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   [gw0] [ 33%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   [gw1] [ 36%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   [gw0] [ 40%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   [gw1] [ 43%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   [gw0] [ 46%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   [gw0] [ 50%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   [gw1] [ 53%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   [gw0] [ 56%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   [gw1] [ 60%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   [gw0] [ 63%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   [gw1] [ 66%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   [gw0] [ 70%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   [gw1] [ 73%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   [gw0] [ 76%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   [gw1] [ 80%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   [gw0] [ 83%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   [gw1] [ 86%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   [gw0] [ 90%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   [gw1] [ 93%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw1] [ 96%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw0] [100%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993527948


   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   The big difference is that when you have your Dockerfile and resulting image, you  build your image in your CI/CD pipeline and you can run checks for it. You could for example verify airflow version, verify whether all imports are done correctly, verify if basic commands work. 
   
   All that LONG before you start hiting that in production. This is the best  common sense engineering practice. And this is the main reasons why containers, isolation has been introduced for. To make a "frozen" environment that you can securely test and deploy in production. If you are using Helm/K8S/Docker but then you try to use it as if it was "local venv" - you are using the worst of both worlds - volatitlty,  lack of guarantees, lack of stability and no opportunity to test things before deploying - all those are coming with containers but you refuse to make use of it, at the same time terribly complicating your setup. 
   
   It's literallly like using a super-modern automated drilll to hit a nail. Not a good idea. Yes you can do it and it will do the job eventually. But it's quite expensive and dangerous thing to do. 
   
   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   I have no idea how the `extraPipPackages`  work under the hood (and no desire to learn). We deliberately refused to even  implement it when asked by the users in the  [official Airflow Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) - anticipating this kind of problems . I think this is more for @thesuperzapper to debug. PIP has some switches (for example `--eager-upgrade` upgrade strategy, that might cause it, or maybe it is simply a result of using `--upgrade` in the pip install.  No idea. No desire to find out. Don't do it.
   
   If you on the other hand build your own image and follow the [nice documentation](https://airflow.apache.org/docs/docker-stack/build.html) we have with plenty of examples, where you directly control the `pip` commands used  I am quite sure you would not have the same problem. Even if you had - let me repeat it again - you would have a chance to find out and debug before it automatically migrated your database.
   
   And this is what I heartily recommend.
   
   BTW. Docker image testing is precisely what we do in our CI/CD for airflow. With every single PR in Airflow we verify that the image does what it is supposed to do. Heartily recommend. See here for an inspiration (code available https://github.com/apache/airflow/blob/main/docker_tests/test_prod_image.py):
   
   https://github.com/apache/airflow/runs/4519400249?check_suite_focus=true#step:6:310
   
   ```
   docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   [gw0] [  3%] PASSED docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   [gw1] [  6%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   docker_tests/test_prod_image.py::TestCommands::test_python_version 
   [gw1] [ 10%] PASSED docker_tests/test_prod_image.py::TestCommands::test_python_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   [gw0] [ 13%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   [gw1] [ 16%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   [gw0] [ 20%] PASSED docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   [gw1] [ 23%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   [gw0] [ 26%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   [gw1] [ 30%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   [gw0] [ 33%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   [gw1] [ 36%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   [gw0] [ 40%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   [gw1] [ 43%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   [gw0] [ 46%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   [gw0] [ 50%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   [gw1] [ 53%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   [gw0] [ 56%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   [gw1] [ 60%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   [gw0] [ 63%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   [gw1] [ 66%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   [gw0] [ 70%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   [gw1] [ 73%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   [gw0] [ 76%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   [gw1] [ 80%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   [gw0] [ 83%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   [gw1] [ 86%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   [gw0] [ 90%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   [gw1] [ 93%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw1] [ 96%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw0] [100%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-991209264


   NEVER  use `extraPipPackages` or equivalent in production. 
   We warn against trying to do similar thing for similar case with images  https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements  - and we had a number of discussions and warned  @thesuperzapper that those kind of cases like yours will be happening. That's perfect example of why you should not do it.
   
    (@thesuperzapper - I hope you will be able to help diagnose the observed behaviour and help @jensgrnb). 
   
   Instead you should build your own image: https://airflow.apache.org/docs/docker-stack/build.html  and use that image for all your components. There ar plenty of examples, including how to upgrade provider, how to update packages etc. in our docs. 
   
   Then you can stay with Matthew's chart, or switch to the officially supported package by the Airflow Community - https://airflow.apache.org/docs/helm-chart/stable/index.html (where we would never, ever, advice our users to dyamically install packages in their image) - we don't really force you to use our chart. 
   
   Feel free to open an issue in https://github.com/airflow-helm/charts where it belongs and possibly get Matthew's help on fixing the problem, but this is not really airflow bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993420966






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993610995


   That's all true, but like what if apt worked the same way, and I specified some version of a package that had a dependency on an older version of linux. Would it just casually overwrite my OS?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990887257


   We're adding the following extra pip packages: 
   
   ```yaml
     extraPipPackages:
       - "apache-airflow[sendgrid,google]~=2.1.2"
       - "gcsfs~=2021.6.0"
       - "apache-airflow-providers-celery==2.0.0"
   ```
   
   From the pip logs that results in 
   
   ```
   ...
   Successfully installed aiohttp-3.8.1 aiosignal-1.2.0 amqp-2.6.1 apache-airflow-2.1.4 apache-airflow-providers-celery-2.0.0 async-timeout-4.0.1 celery-4.4.7 flower-0.9.7 frozenlist-1.2.0 fsspec-2021.6.1 gcsfs-2021.6.1 importlib-resources-1.5.0 kombu-4.6.11 multidict-5.2.0 prometheus-client-0.8.0 tenacity-6.2.0 vine-1.3.0 yarl-1.7.2
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989924490


   Yeah. You have to keep your pod template and base image synced after the update. 
   
   I think this will be handled much better soon with #18439 - when DB migrations will be involved and not applied, your K8S job should fail with an appropriate message. But I think this also comes back to our earlier discussion @kaxil  @jedcunningham  @dimberman  (with migration containers) -  what happens if there are two different Airlow versions In KE and airflow when there is no "migration" between them?
   
   Should we flag that as an error? Should we check if the airflow version is right in the image? I think that would make sense to pass Airflow version when KE submits the job and we should at least check if the versions match, because there might be cses like this one that will be very difficult to debug otherwise.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989876598


   Without doing anything to the pod template, it lists this label "airflow_version": 2.1.4 not 2.2.1 in the pod description. 
   
   On the last successful job we had running on 2.1.2 there was no discrepancy between the label and the docker tag. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989936977


   Oh oh, so because I don't touch the pod template, I'm actually running a different version of airflow in the cluster than what's specified in the docker tag?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990125357


   How do you get your images ? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] thesuperzapper commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
thesuperzapper commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993126518


   @potiuk to be clear, I put a warning after every `helm install` that tells users not to use the `extraPipPackages` in environments that are important. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994942798


   Just to explaining it to by-lookers. This is the first time ever I close (already closed) issue for discussion after sending the discussion 3 times to the place it belongs. Never happened to me before.
   
   @jensgrnb  Feel free to discuss this issue further in https://github.com/airflow-helm/charts, where the problem belongs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-991209264


   NEVER  use `extraPipPackages` or equivalent in production. 
   We warn against trying to do similar thing for similar case with images  https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements  - and we had a number of discussions and warned  @thesuperzapper that those kind of cases like yours will be happening. That's perfect example of why you should not do it.
   
    (@thesuperzapper - I hope you will be able to help diagnose the observed behaviour and help @jensgrnb). 
   
   Instead you should build your own image: https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements  and use that image for all your components. There ar plenty of examples, including how to upgrade provider, how to update packages etc. in our docs. 
   
   Then you can stay with Matthew's chart, or switch to the officially supported package by the Airflow Community - https://airflow.apache.org/docs/helm-chart/stable/index.html (where we would never, ever, advice our users to dyamically install packages in their image) - we don't really force you to use our chart. 
   
   Feel free to open an issue in https://github.com/airflow-helm/charts where it belongs and possibly get Matthew's help on fixing the problem, but this is not really airflow bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20168:
URL: https://github.com/apache/airflow/issues/20168


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989956625


   Well, it's almost the same..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993527948


   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   The big difference is that when you hace your Dockerfile and resulting image.You  build your image in your CI/CD frameworrk and you can run checks for it. You could for example verify airflow version, verify whether all imports are done correctly, verify if basic commands work. 
   
   All that LONG before you start hitting that in production. This is the best  common sense engineering practice. And this is the main reasons why containers, isolation has been introduced for. To make a "frozen" environment that you can securely test and deploy in production. If you are using Helm/K8S/Docker but then you try to use it as if it was "local venv" - you are using the worst of both worlds - volatitlty,  lack of guarantees, lack of stability and no opportunity to test things before deploying - all those are coming with containers but you refuse to make use of it, at the same time terribly complicating your setup. 
   
   It's literallly like using a super-modern automated drilll to hit a nail. Not a good idea. Yes you can do it and it will do the job eventually. But it's quite expensive and dangerous thing to do. 
   
   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   I have no idea how the `extraPipPackages`  work under the hood (and no desire to learn). We deliberately refused to even  implement it when asked by the users in the  [official Airflow Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) - anticipating this kind of problems . I think this is more for @thesuperzapper to debug. PIP has some switches (for example `--eager-upgrade` upgrade strategy, that might cause it, or maybe it is simply a result of using `--upgrade` in the pip install.  No idea. No desire to find out. Don't do it.
   
   If you on the other hand build your own image and follow the [nice documentation](https://airflow.apache.org/docs/docker-stack/build.html) we have with plenty of examples, where you directly control the `pip` commands used  I am quite sure you would not have the same problem. Even if you had - let me repeat it again - you would have a chance to find out and debug before it automatically migrated your database.
   
   And this is what I heartily recommend.
   
   BTW. Docker image testing is precisely what we do in our CI/CD for airflow. With every single PR in Airflow we verify that the image does what it is supposed to do. Heartily recommend. See here for an inspiration (code available https://github.com/apache/airflow/blob/main/docker_tests/test_prod_image.py):
   
   https://github.com/apache/airflow/runs/4519400249?check_suite_focus=true#step:6:310
   
   ```
   docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   [gw0] [  3%] PASSED docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   [gw1] [  6%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   docker_tests/test_prod_image.py::TestCommands::test_python_version 
   [gw1] [ 10%] PASSED docker_tests/test_prod_image.py::TestCommands::test_python_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   [gw0] [ 13%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   [gw1] [ 16%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   [gw0] [ 20%] PASSED docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   [gw1] [ 23%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   [gw0] [ 26%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   [gw1] [ 30%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   [gw0] [ 33%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   [gw1] [ 36%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   [gw0] [ 40%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   [gw1] [ 43%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   [gw0] [ 46%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   [gw0] [ 50%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   [gw1] [ 53%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   [gw0] [ 56%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   [gw1] [ 60%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   [gw0] [ 63%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   [gw1] [ 66%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   [gw0] [ 70%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   [gw1] [ 73%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   [gw0] [ 76%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   [gw1] [ 80%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   [gw0] [ 83%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   [gw1] [ 86%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   [gw0] [ 90%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   [gw1] [ 93%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw1] [ 96%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw0] [100%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20168:
URL: https://github.com/apache/airflow/issues/20168


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990126662


   Maybe when you build your images you use "latest" tag from Airflow and your "latest" was pulled when it 2.1.4 ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-991209264


   NEVER  use `extraPipPackages` or equivalent in production. 
   We warn against trying to do similar thing for similar case with images  https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements  - and we had a number of discussions and warned  @thesuperzapper that those kind of cases like yours will be happening. That's perfect example of why you shoudl not do it.
   
    (@thesuperzapper - I hope you will be able to help diagnose the observed behaviour and help @jensgrnb). 
   
   Instead you should build your own image: https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements  and use that image for all your components. There ar plenty of examples, including how to upgrade provider, how to update packages etc. in our docs. 
   
   Then you can stay with Matthew's chart, or switch to the officially supported package by the Airflow Community - https://airflow.apache.org/docs/helm-chart/stable/index.html (where we would never, ever, advice our users to dyamically install packages in their image) - we don't really force you to use our chart. 
   
   Feel free to open an issue in https://github.com/airflow-helm/charts where it belongs and possibly get Matthew's help on fixing the problem, but this is not really airflow bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994570472


   @potiuk. I don't understand why there's such negativity in trying to figure out why it went wrong in our cluster. 
   
   In this comment, https://github.com/apache/airflow/issues/20168#issuecomment-993687968,  you both state that I use the pip install pattern wrong by specifying `apache-airflow[sendgrid,google]~=2.1.2`
   and link to your own documentation where that syntax is referenced on the top page. 
   
   If you're telling me that pip resolves dependencies of multiple packages by simply using whatever the last package specifies, that's fine. Seems strange to me, but I'll accept it as what happened and keep the docker tag and the pip package versions aligned in the future. 
   
   Let's close it and thank you helping resolve my problem. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989990108


   I see stuff like
   
   ```
   airflow@airflow-worker-0:/opt/airflow$ env | grep AIRFLOW_VERSION
   AIRFLOW_VERSION=2.2.1
   airflow@airflow-worker-0:/opt/airflow$ airflow version
   2.1.4
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989916192


   It seems to be that we have a discrepancy between the docker tag and the actual airflow version we're getting in the image..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989946974


   Currently kubetl specifies `apache/airflow:2.2.1-python3.8` as docker tag on all airflow components in our GKE cluster, however running `airflow version` in any of the containers returns `2.1.4`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989924490


   Yeah. You have to keep your pod template and base image synced after the update. 
   
   I think this will be handled much better soon with #18439 - when DB migrations will be involved and not applied, your K8S job should fail with an appropriate message. But I think this also comes back to our earlier discussion @kaxil  @jedcunningham  @dimberman @ephraimbuddy  (with migration containers) -  what happens if there are two different Airlow versions In KE and airflow when there is no "migration" between them?
   
   Should we flag that as an error? Should we check if the airflow version is right in the image? I think that would make sense to pass Airflow version when KE submits the job and we should at least check if the versions match, because there might be cses like this one that will be very difficult to debug otherwise.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990889538


   The celery pin is due to this: https://github.com/apache/airflow/issues/19136
   
   Not sure which of the pip packages has this sideaffect of installing airflow 2.1.4. I guess it's the `apache-airflow[sendgrid,google]~=2.1.2` pin that forces it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993527948


   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   The big difference is that when you have your Dockerfile and resulting image, you  build your image in your CI/CD pipeline and you can run checks for it. You could for example verify airflow version, verify whether all imports are done correctly, verify if basic commands work. 
   
   All that LONG before you start hiting that in production. This is the best  common sense engineering practice. And this is the main reason why containers, isolation has been introduced for. To make a "frozen" environment that you can securely test and deploy in production. If you are using Helm/K8S/Docker but then you try to use it as if it was "local venv" - you are using the worst of both worlds - volatitlty,  lack of guarantees, lack of stability and no opportunity to test things before deploying - all those are coming with containers but you refuse to make use of it, at the same time terribly complicating your setup. 
   
   It's literallly like using a super-modern automated drilll to hit a nail. Not a good idea. Yes you can do it and it will do the job eventually. But it's quite expensive and dangerous thing to do. 
   
   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   I have no idea how the `extraPipPackages`  work under the hood (and no desire to learn). We deliberately refused to even  implement it when asked by the users in the  [official Airflow Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) - anticipating this kind of problems . I think this is more for @thesuperzapper to debug. PIP has some switches (for example `--eager-upgrade` upgrade strategy, that might cause it, or maybe it is simply a result of using `--upgrade` in the pip install.  No idea. No desire to find out. Don't do it.
   
   If you on the other hand build your own image and follow the [nice documentation](https://airflow.apache.org/docs/docker-stack/build.html) we have with plenty of examples, where you directly control the `pip` commands used  I am quite sure you would not have the same problem. Even if you had - let me repeat it again - you would have a chance to find out and debug before it automatically migrated your database.
   
   And this is what I heartily recommend.
   
   BTW. Docker image testing is precisely what we do in our CI/CD for airflow. With every single PR in Airflow we verify that the image does what it is supposed to do. Heartily recommend. See here for an inspiration (code available https://github.com/apache/airflow/blob/main/docker_tests/test_prod_image.py):
   
   https://github.com/apache/airflow/runs/4519400249?check_suite_focus=true#step:6:310
   
   ```
   docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   [gw0] [  3%] PASSED docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   [gw1] [  6%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   docker_tests/test_prod_image.py::TestCommands::test_python_version 
   [gw1] [ 10%] PASSED docker_tests/test_prod_image.py::TestCommands::test_python_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   [gw0] [ 13%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   [gw1] [ 16%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   [gw0] [ 20%] PASSED docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   [gw1] [ 23%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   [gw0] [ 26%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   [gw1] [ 30%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   [gw0] [ 33%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   [gw1] [ 36%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   [gw0] [ 40%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   [gw1] [ 43%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   [gw0] [ 46%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   [gw0] [ 50%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   [gw1] [ 53%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   [gw0] [ 56%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   [gw1] [ 60%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   [gw0] [ 63%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   [gw1] [ 66%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   [gw0] [ 70%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   [gw1] [ 73%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   [gw0] [ 76%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   [gw1] [ 80%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   [gw0] [ 83%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   [gw1] [ 86%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   [gw0] [ 90%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   [gw1] [ 93%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw1] [ 96%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw0] [100%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993527948


   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   The big difference is that when you have your Dockerfile and resulting image, you  build your image in your CI/CD pipeline and you can run checks for it. You could for example verify airflow version, verify whether all imports are done correctly, verify if basic commands work. 
   
   All that LONG before you start hitting that in production. This is the best  common sense engineering practice. And this is the main reasons why containers, isolation has been introduced for. To make a "frozen" environment that you can securely test and deploy in production. If you are using Helm/K8S/Docker but then you try to use it as if it was "local venv" - you are using the worst of both worlds - volatitlty,  lack of guarantees, lack of stability and no opportunity to test things before deploying - all those are coming with containers but you refuse to make use of it, at the same time terribly complicating your setup. 
   
   It's literallly like using a super-modern automated drilll to hit a nail. Not a good idea. Yes you can do it and it will do the job eventually. But it's quite expensive and dangerous thing to do. 
   
   > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen?
   
   I have no idea how the `extraPipPackages`  work under the hood (and no desire to learn). We deliberately refused to even  implement it when asked by the users in the  [official Airflow Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) - anticipating this kind of problems . I think this is more for @thesuperzapper to debug. PIP has some switches (for example `--eager-upgrade` upgrade strategy, that might cause it, or maybe it is simply a result of using `--upgrade` in the pip install.  No idea. No desire to find out. Don't do it.
   
   If you on the other hand build your own image and follow the [nice documentation](https://airflow.apache.org/docs/docker-stack/build.html) we have with plenty of examples, where you directly control the `pip` commands used  I am quite sure you would not have the same problem. Even if you had - let me repeat it again - you would have a chance to find out and debug before it automatically migrated your database.
   
   And this is what I heartily recommend.
   
   BTW. Docker image testing is precisely what we do in our CI/CD for airflow. With every single PR in Airflow we verify that the image does what it is supposed to do. Heartily recommend. See here for an inspiration (code available https://github.com/apache/airflow/blob/main/docker_tests/test_prod_image.py):
   
   https://github.com/apache/airflow/runs/4519400249?check_suite_focus=true#step:6:310
   
   ```
   docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   [gw0] [  3%] PASSED docker_tests/test_prod_image.py::TestCommands::test_without_command 
   docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   [gw1] [  6%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_command 
   docker_tests/test_prod_image.py::TestCommands::test_python_version 
   [gw1] [ 10%] PASSED docker_tests/test_prod_image.py::TestCommands::test_python_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   [gw0] [ 13%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_version 
   docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   [gw1] [ 16%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   [gw0] [ 20%] PASSED docker_tests/test_prod_image.py::TestCommands::test_bash_version 
   docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   [gw1] [ 23%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   [gw0] [ 26%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   [gw1] [ 30%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   [gw0] [ 33%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   [gw1] [ 36%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   [gw0] [ 40%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   [gw1] [ 43%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   [gw0] [ 46%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   [gw0] [ 50%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   [gw1] [ 53%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   [gw0] [ 56%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   [gw1] [ 60%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   [gw0] [ 63%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   [gw1] [ 66%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   [gw0] [ 70%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   [gw1] [ 73%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   [gw0] [ 76%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   [gw1] [ 80%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   [gw0] [ 83%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   [gw1] [ 86%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] 
   docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   [gw0] [ 90%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   [gw1] [ 93%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] 
   docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw1] [ 96%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root 
   [gw0] [100%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root 
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990887257


   We're adding the follow extra pip packages: 
   
   ```yaml
     extraPipPackages:
       - "apache-airflow[sendgrid,google]~=2.1.2"
       - "gcsfs~=2021.6.0"
       - "apache-airflow-providers-celery==2.0.0"
   ```
   
   From the pip logs that results in 
   
   ```
   ...
   Successfully installed aiohttp-3.8.1 aiosignal-1.2.0 amqp-2.6.1 apache-airflow-2.1.4 apache-airflow-providers-celery-2.0.0 async-timeout-4.0.1 celery-4.4.7 flower-0.9.7 frozenlist-1.2.0 fsspec-2021.6.1 gcsfs-2021.6.1 importlib-resources-1.5.0 kombu-4.6.11 multidict-5.2.0 prometheus-client-0.8.0 tenacity-6.2.0 vine-1.3.0 yarl-1.7.2
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994124325


   And @thesuperzapper @jensgrnb  - if you want to further discuss any of it - I encourage you to open an issue where it belongs - i.e.  https://github.com/airflow-helm/charts/issues


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994942798


   Just to explaining it to by-lookers. This is the first time ever I close an issue for discussion after sending the discussion 3 times to the place it belongs. Never happened to me before.
   
   @jensgrnb  Feel free to discuss this issue further it https://github.com/airflow-helm/charts, where the problem belongs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993681366


   > That's all true, but like what if apt worked the same way, and I specified some version of a package that had a dependency on an older version of linux. Would it just casually overwrite my OS?
   
   1. First of all - this is how PIP works
   2. Secondly. This is also how APT works. If you have dependency that reguires newer version of another dependency it will upgrade it
   3. Thirdl. All package managers work like that.
   
   Airflow is not an OS. Airflow is an application that is installed on top of OS. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994937867


   > @potiuk. I don't understand why there's such negativity in trying to figure out why it went wrong in our cluster.
   
   It's not negativity. This is simply because you continue ignoring what I am telling you repeatedly.
   
   If there are problems related to the wrong feature in the chart done by @thesuperzapper it should be discussed there not here. And yet you conitnue coming back here even if I asked you to continue the discussion where it belongs. It's just wrong place to discuss broken feature on a chart that does not belong here (and I repeat it for the thrird time now -  yet you continue coming back here).
   
   > In this comment, [#20168 (comment)](https://github.com/apache/airflow/issues/20168#issuecomment-993687968), you both state that I use the pip install pattern wrong by specifying `apache-airflow[sendgrid,google]~=2.1.2`
   > and link to your own documentation where that syntax is referenced on the top page.
   
   Read it with understanding please. 
   
   This is an example for a COMPLETELY different thing.
   
   This is for installing airflow in virtualenv from scratch. Not to use it in broken pipExtra feature on image where you already have and want to add a new provider. For that you have [this documentation](https://airflow.apache.org/docs/docker-stack/build.html#examples-of-image-extending)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989954504


   > Currently kubetl specifies `apache/airflow:2.2.1-python3.8` as docker tag on all airflow components in our GKE cluster, however running `airflow version` in any of the containers returns `2.1.4`.
   
   That's NOT GOOD :)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993420966


   @thesuperzapper is right it's said explicitly in the extraPipPackages block on the readme in that repo. However I'm not quite sure how writing the pip install in a dockerfile can save me from this, except it's easier to catch that the base image tag is out of sync with any pip dependencies. 
   
   We use argo, so I haven't actually seen those warning messages from helm. 
   
   To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and pulls an effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile, will the same not happen?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-992261987


   Thanks @potiuk, sorry for bringing the problem to airflow as it really belongs with the helm chart (and me for not knowing how pip works well enough). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993687968


   Then 4thly @jensgrnb - this is a question to @thesuperzapper what command is used to install it. By default PIP will NOT upgrade dependencies that are already installed. It's either the `extraPackage` work or YOUR dependencies require to install higher airflow version. 
   
   And 5thly: Your "apache-airflow[sendgrid,google]~=2.1.2" is likely wrong way of installing providers. Your "apache-airflow-providers-celery==2.0.0" is good one if you want to install provider on top of Airflow.
   
   By installing "apache-airflow[sendgrid,google]~=2.1.2" you are installing (again) airflow and sendgrid and google extras for it. Maybe that caused the problem. I have no idea what happens under the hood. But it's certainly not the way described in our documentation: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrading-of-airflow-providers-separately
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993694726


   So in short: you simply:
   * did not do what our docs describe
   * relied on unknown behaviour of `extraPackages` which you were warned against
   * you run a command that does something you did not want
   
   And then you complan  that it does not do what you want. That's a bit of recipe for disaster IMHO.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-991209264


   NEVER  use `extraPipPackages` or equivalent in production. 
   We warn against trying to do similar thing for similar case with images  https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements  - and we had a number of discussions and warned  @thesuperzapper that those kind of cases like yours will be happening. That's perfect example of why you shoudl not do it.
   
    (@thesuperzapper - I hope you will be able to help diagnose the observed behaviour and help @jensgrnb). 
   
   Instead you should build your own image: https://airflow.apache.org/docs/docker-stack/entrypoint.html#installing-additional-requirements  and use that image for all your components. There ar plenty of examples, including how to upgrade provider, how to update packages etc. in our docs. 
   
   Then you can stay with Matthew's chart, or switch to the officially supported package by the Airflow Community - https://airflow.apache.org/docs/helm-chart/stable/index.html (where we would never, ever, advice our users to dyamically install packages in their image) - we don't really force you to use our chart. 
   
   Feel free to open an issue in https://github.com/airflow-helm/charts where it belongs and possibly get Matthew's help on fixing the problem, but this is not really airflow bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-991209264






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989924490


   Yeah. You have to keep your pod template and base image synced after the update. 
   
   I think this will be handled much better soon with #18439 - when DB migrations will be involved and not applied, your K8S job should fail with an appropriate message. But I think this also comes back to our earlier discussion @kaxil  @jedcunningham  @dimberman @ephraimbuddy  (with migration containers). The case which I am worried about is what happens if there are two different Airlow versions In KE and airflow when there is no "migration" between them?
   
   Should we flag that as an error? Should we check if the airflow version is right in the image? I think that would make sense to pass Airflow version when KE submits the job and we should at least check if the versions match, because there might be cases like this one that will be very difficult to diagnose/debug otherwise.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-989924490


   Yeah. You have to keep your pod template and base image synced after the update. 
   
   I think this will be handled much better soon with #18439 - when DB migrations will be involved and not applied, your K8S job should fail with an appropriate message. But I think this also comes back to our earlier discussion @kaxil  @jedcunningham  @dimberman @ephraimbuddy  (with migration containers) -  what happens if there are two different Airlow versions In KE and airflow when there is no "migration" between them?
   
   Should we flag that as an error? Should we check if the airflow version is right in the image? I think that would make sense to pass Airflow version when KE submits the job and we should at least check if the versions match, because there might be cases like this one that will be very difficult to debug otherwise.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990887680


   Looks like we're overwriting the docker defined apache-airflow app...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jensgrnb commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
jensgrnb commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-990769271


   mm we don't build on top of the dockerhub image here: https://hub.docker.com/layers/apache/airflow/2.2.1-python3.8/images/sha256-e0dd0f34848d0d87e6a372507ebcd844028e095ec29e95c4569dcb5a2042fc84?context=explore
   
   We pull it directly from dockerhub and run airflow cmds to set up the webserver, worker, etc. 
   
   We use this helm chart: https://github.com/airflow-helm/charts/tree/main/charts/airflow
   
   Is it even possible to run a different airflow version, than the one specified in the docker tag?
   
   From my grep and airflow version cmds in the worker pod it also seems like airflow itself is a bit divided on which version it's running..
   
   Funny thing is that we upgraded from 2.1.2, not 2.1.4... So 2.1.4 has never intentionally been deployed by us...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993681366


   > That's all true, but like what if apt worked the same way, and I specified some version of a package that had a dependency on an older version of linux. Would it just casually overwrite my OS?
   
   1. First of all - this is how PIP works
   2. Secondly. This is also how APT works. If you have dependency that reguires newer version of another dependency it will upgrade it
   3. Thirdly. All package managers work like that.
   
   Airflow is not an OS. Airflow is an application that is installed on top of OS. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994111974


   > @potiuk the pip install is very simple, it just uses `pip install --user {package_1} {package_2}`.
   > 
   > [Here is the `pip install` code from the "user-community" chart.](https://github.com/airflow-helm/charts/blob/e1d49498426add959350ab8efacead8f96400759/charts/airflow/templates/_helpers/pods.tpl#L177-L180)
   
   I honestly do not want to spend any of my time on it. 
   
   The whole issue is because very bad usage patterns are used wihich we clearly said are wrong - and is encouraged by the user community chart
   
   My advice for @jensgrnb  is:
   
   * build your own image (following our examples/docs)
   * test if it is what you expected
   * use it in any of the charts
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-993532386


   > @potiuk to be clear, I put a warning after every `helm install` that tells users not to use the `extraPipPackages` in environments that are important.
   
   Yeah. But just having it as an option encourages the users to use it, And this is just wrong. `Convenience <> Stability.`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #20168: KubernetesExecutor reports missing dag_run.data_interval_start on airflow@2.2.1

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #20168:
URL: https://github.com/apache/airflow/issues/20168#issuecomment-994942798


   Just to explaining it to by-lookers. This is the first time ever I close an issue for discussion after sending the discussion 3 times to the place it belongs. Never happened to me before.
   
   @jensgrnb  Feel free to discuss this issue further in https://github.com/airflow-helm/charts, where the problem belongs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org