You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/28 02:04:04 UTC

[GitHub] [airflow] grenmester opened a new issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

grenmester opened a new issue #17878:
URL: https://github.com/apache/airflow/issues/17878


   ### Apache Airflow version
   
   2.0.1
   
   ### Operating System
   
   5.13.12-arch1-1
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Managed Services (Astronomer, Composer, MWAA etc.)
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   I tried using the `to_csv` method in one of my tasks created using the new task decorator to save a file to a GCS bucket. The task was marked as successful but the file did not save to the bucket.
   
   ### What you expected to happen
   
   I expected it to either fail or have the file be saved successfully but it was neither. I did some digging and turns out it's a permission issue. I would have expected the logs to show that it was a permission issue but it instead silently fails in the background. This is surprising to me because when I tested reading from a GCS bucket using `pd.read_csv`, it showed in the logs that it was a permission issue.
   
   ### How to reproduce
   
   ```python
   @task
   def my_task():
       df = pd.DataFrame()
       df.to_csv("gs://path-to-file")
   ```
   
   ### Anything else
   
   Airflow logs:
   
   ```
   [2021-08-27 06:10:47,306] {taskinstance.py:1160} INFO - Marking task as SUCCESS. dag_id=phonic_initial, task_id=fb-17.get_survey_data, execution_date=20210827T060936, start_date=20210827T060940, end_date=20210827T061047
   [2021-08-27 06:10:47,608] {local_task_job.py:187} WARNING - State of this instance has been externally set to success. Terminating instance.
   [2021-08-27 06:10:47,609] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 115
   [2021-08-27 06:10:47,622] {taskinstance.py:1241} ERROR - Received SIGTERM. Terminating subprocesses.
   [2021-08-27 06:10:47,989] {process_utils.py:66} INFO - Process psutil.Process(pid=115, status='terminated', exitcode=1, started='06:09:40') (115) terminated with exit code 1
   ```
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal edited a comment on issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #17878:
URL: https://github.com/apache/airflow/issues/17878#issuecomment-907759731


   Can you please explain why/how is it an Airflow bug rather than a bug/issue with your own code locally (problem with your own python function)?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17878:
URL: https://github.com/apache/airflow/issues/17878#issuecomment-907550207


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] grenmester closed issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

Posted by GitBox <gi...@apache.org>.
grenmester closed issue #17878:
URL: https://github.com/apache/airflow/issues/17878


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] grenmester closed issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

Posted by GitBox <gi...@apache.org>.
grenmester closed issue #17878:
URL: https://github.com/apache/airflow/issues/17878


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] grenmester commented on issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

Posted by GitBox <gi...@apache.org>.
grenmester commented on issue #17878:
URL: https://github.com/apache/airflow/issues/17878#issuecomment-908762423


   Thanks for the question. I did a little bit of extra digging and seems like it's Pandas that implicit fails when writing to a bucket with insufficient permissions so it's not an Airflow problem. Thanks for helping.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] grenmester commented on issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

Posted by GitBox <gi...@apache.org>.
grenmester commented on issue #17878:
URL: https://github.com/apache/airflow/issues/17878#issuecomment-908762423


   Thanks for the question. I did a little bit of extra digging and seems like it's Pandas that implicit fails when writing to a bucket with insufficient permissions so it's not an Airflow problem. Thanks for helping.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #17878: Pandas `to_csv` with GCS source implicitly fails in the background with improper credentials

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #17878:
URL: https://github.com/apache/airflow/issues/17878#issuecomment-907759731


   Can you please explain why/how there is an Airflow bug rather than a bug/issue with your own code locally (problem with your own python function)?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org