You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/03 08:17:35 UTC
[GitHub] [airflow] PikaYellow35 opened a new issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
PikaYellow35 opened a new issue #20007:
URL: https://github.com/apache/airflow/issues/20007
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==6.1.0
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Debian GNU/Linux 10 (buster)
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### What happened
When I'm execute `PostgresToGCSOperator` on empty table and set `use_server_side_cursor=True` the operator fails with error:
```
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
self._execute_task_with_callbacks(context)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
result = self._execute_task(context, self.task)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
result = execute_callable(context=context)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/sql_to_gcs.py", line 154, in execute
files_to_upload = self._write_local_data_files(cursor)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/sql_to_gcs.py", line 213, in _write_local_data_files
row = self.convert_types(schema, col_type_dict, row)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/sql_to_gcs.py", line 174, in convert_types
return [self.convert_type(value, col_type_dict.get(name)) for name, value in zip(schema, row)]
TypeError: 'NoneType' object is not iterable
```
Operator command when I'm using:
```python
task_send = PostgresToGCSOperator(
task_id=f'send_{table}',
postgres_conn_id='postgres_raw',
gcp_conn_id=gcp_conn_id,
sql=f'SELECT * FROM public.{table}',
use_server_side_cursor=True,
bucket=bucket,
filename=f'{table}.csv',
export_format='csv',
)
```
### What you expected to happen
I'm expected, that operator on empty table not creating file and no upload it on Google Cloud.
### How to reproduce
- Create empty postgresql table.
- Create dag with task with PostgresToGCSOperator. that upload this table in Google Cloud.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20007:
URL: https://github.com/apache/airflow/issues/20007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rsg17 commented on issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
Posted by GitBox <gi...@apache.org>.
rsg17 commented on issue #20007:
URL: https://github.com/apache/airflow/issues/20007#issuecomment-989107728
Hi - I am new to Airflow and would like to try and take a pass at this.
I have breeze setup; have created one [PR](https://github.com/apache/airflow/pull/19727); but nothing on the operators side. I will go through the corresponding code and contributing page again. Do let me know if you have any pointers..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20007:
URL: https://github.com/apache/airflow/issues/20007#issuecomment-985304672
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rsg17 edited a comment on issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
Posted by GitBox <gi...@apache.org>.
rsg17 edited a comment on issue #20007:
URL: https://github.com/apache/airflow/issues/20007#issuecomment-989107728
Hi @potiuk - I am new to Airflow and would like to try and take a pass at this.
I have breeze setup; have created one [PR](https://github.com/apache/airflow/pull/19727); but nothing on the operators side. I will go through the corresponding code and contributing page again. Do let me know if you have any pointers..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20007:
URL: https://github.com/apache/airflow/issues/20007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #20007:
URL: https://github.com/apache/airflow/issues/20007#issuecomment-1000806500
@rsg17 feel free to join Airflow slack. There are channels for support like: `airflow-how-to-pr`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20007: PostgresToGCSOperator fail on empty table and use_server_side_cursor=True
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20007:
URL: https://github.com/apache/airflow/issues/20007#issuecomment-986318107
Maybe you would like to contribute a PR to fix it @PikaYellow35 - we have > 1800 contributors, you can become one of them :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org