[GitHub] [airflow] marclamberti commented on issue #12334: kubernetes executor pod not removed

[GitBox <gi...@apache.org>]

marclamberti commented on issue #12334:
URL: https://github.com/apache/airflow/issues/12334#issuecomment-809362985


   I have this issue with Airflow 2.0.1 and the Helm chart.
   It happens since I enabled remote logging with S3 where the credentials are given through a Secret object.
   ```
   mydagdownloading.09c4d4061ada4c058dd91ac4885dd76f   1/1     Running   0          32m
   mydagdownloading.2bc28a8a155f4e29b234e1d153eef52e   1/1     Running   0          37m
   mydagdownloading.f25cdd3178604b3f8b343f6bb13d7840   1/1     Running   0          42m
   mydagdownloading.fcca60688a324a898014cb98bcbffbc2   1/1     Running   0          34m
   mydagprocessing.09d820cdb1f14a919d0e9a7661b295cf    1/1     Running   0          34m
   mydagprocessing.581978d0c2064f56bc6affebc499e489    1/1     Running   0          31m
   mydagprocessing.5d4b18ff4a8f4af4b2acff1595e156a3    1/1     Running   0          42m
   mydagprocessing.9efc24c0813a45789c93139c5ae4fc1d    1/1     Running   0          37m
   mydagstoring.3a9e3635a6294e57b605fd52c465825a       1/1     Running   0          41m
   mydagstoring.86ec9747f7824bcca684fe5e944ef1b4       1/1     Running   0          31m
   mydagstoring.eabf0e43467848e7988fca4e8bf45267       1/1     Running   0          34m
   mydagstoring.eb1533e230ed440e8b40c0c57b361a4b       1/1     Running   0          37m
   ```
   And whenever I try to get the logs I get this message:
   ```
   *** Failed to verify remote log exists s3://marcl-airflow/airflow/logs/my_dag/downloading/2021-03-29T12:33:27.660837+00:00/1.log.
   An error occurred (403) when calling the HeadObject operation: Forbidden
   *** Falling back to local log
   *** Trying to get logs (last 100 lines) from worker pod mydagdownloading.09c4d4061ada4c058dd91ac4885dd76f ***
   
   BACKEND=postgresql
   DB_HOST=airflow-postgresql.airflow.svc.cluster.local
   DB_PORT=5432
   
   [2021-03-29 12:33:34,965] {settings.py:210} DEBUG - Setting up DB connection pool (PID 8)
   [2021-03-29 12:33:34,966] {settings.py:281} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=8
   [2021-03-29 12:33:35,039] {cli_action_loggers.py:40} DEBUG - Adding <function default_action_log at 0x7f91156a5400> to pre execution callback
   [2021-03-29 12:33:37,012] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [<function default_action_log at 0x7f91156a5400>]
   [2021-03-29 12:33:37,025] {settings.py:210} DEBUG - Setting up DB connection pool (PID 8)
   [2021-03-29 12:33:37,026] {settings.py:243} DEBUG - settings.prepare_engine_args(): Using NullPool
   [2021-03-29 12:33:37,027] {dagbag.py:448} INFO - Filling up the DagBag from /opt/airflow/dags/repo/my_dag.py
   [2021-03-29 12:33:37,027] {dagbag.py:287} DEBUG - Importing /opt/airflow/dags/repo/my_dag.py
   [2021-03-29 12:33:37,039] {dagbag.py:413} DEBUG - Loaded DAG <DAG: my_dag>
   [2021-03-29 12:33:37,057] {plugins_manager.py:270} DEBUG - Loading plugins
   [2021-03-29 12:33:37,057] {plugins_manager.py:207} DEBUG - Loading plugins from directory: /opt/airflow/plugins
   [2021-03-29 12:33:37,057] {plugins_manager.py:184} DEBUG - Loading plugins from entrypoints
   [2021-03-29 12:33:37,101] {plugins_manager.py:414} DEBUG - Integrate DAG plugins
   ```
   which I think is misleading as the user has all permissions on the bucket/objects. I tried from the worker pod and I can access the bucket and put files into it. But, the worker pods keep running and don't upload any log files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org