You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Duan Shiqiang (JIRA)" <ji...@apache.org> on 2018/12/10 06:39:00 UTC
[jira] [Updated] (AIRFLOW-3488)
PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL ARRAYs
properly
[ https://issues.apache.org/jira/browse/AIRFLOW-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Duan Shiqiang updated AIRFLOW-3488:
-----------------------------------
Description:
PostgresToGoogleCloudStorageOperator generates bigquery schema wrongly for postgres data type ARRAY (it generates bigquery schema with that filed as string).
The generated bigquery schema set mode for one field to 'REPEATED' if the type_code of the postgres column is one of (1009, 1005, 1007, 1016). see: [https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py#L185] . This doesn't include all possible array types (see [https://github.com/psycopg/psycopg2/blob/master/psycopg/typecast_builtins.c)]
was:
PostgresToGoogleCloudStorageOperator generates bigquery schema wrongly for postgres data type ARRAY (it generates bigquery schema with that filed as string).
> PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL ARRAYs properly
> -------------------------------------------------------------------------------
>
> Key: AIRFLOW-3488
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3488
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, operators
> Affects Versions: 1.10.1
> Reporter: Duan Shiqiang
> Priority: Major
>
> PostgresToGoogleCloudStorageOperator generates bigquery schema wrongly for postgres data type ARRAY (it generates bigquery schema with that filed as string).
> The generated bigquery schema set mode for one field to 'REPEATED' if the type_code of the postgres column is one of (1009, 1005, 1007, 1016). see: [https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/postgres_to_gcs_operator.py#L185] . This doesn't include all possible array types (see [https://github.com/psycopg/psycopg2/blob/master/psycopg/typecast_builtins.c)]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)