You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/15 15:58:46 UTC

[GitHub] [airflow] BhuviTheDataGuy opened a new issue #11551: PostgresToGCSOperator - Exporting the Timestamp column in EPOCH format

BhuviTheDataGuy opened a new issue #11551:
URL: https://github.com/apache/airflow/issues/11551


   Im using PostgresToGCSOperator to export the data. I know there is a mapping in that operator. I have changed that and made as a custom operator. If Postgres has timestamp then in BQ use DATETIME for timestampz use TIMESTAMP. 
   
   ```py
          1114: 'DATETIME',
           1184: 'TIMESTAMP',
           1082: 'DATE',
           1083: 'TIME',
           1005: 'INTEGER',
           1007: 'INTEGER',
           1016: 'INTEGER',
           20: 'INTEGER',
           21: 'INTEGER',
           23: 'INTEGER',
           16: 'BOOLEAN',
           700: 'FLOAT',
           701: 'FLOAT',
           1700: 'FLOAT'
   ```
   
   But while exporting, this data in CSV format, it won't considering this and exporting the value in EPOCH format. How do I fix it? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] BhuviTheDataGuy commented on issue #11551: PostgresToGCSOperator - Exporting the Timestamp column in EPOCH format

Posted by GitBox <gi...@apache.org>.
BhuviTheDataGuy commented on issue #11551:
URL: https://github.com/apache/airflow/issues/11551#issuecomment-710770442


   Im able to modify the original operator and there is a conversion function that is doing this EPOCH conversion. So I customized it and everything looks good now.
   
   ```py
   
       def convert_type(self, value, schema_type):
           """
           Takes a value from Postgres, and converts it to a value that's safe for
           JSON/Google Cloud Storage/BigQuery. Dates are converted to UTC seconds.
           Decimals are converted to floats. Times are converted to seconds.
           """
           if isinstance(value, (datetime.datetime, datetime.date)):
               return str(value)
           if isinstance(value, datetime.time):
               return str(value)
           if isinstance(value, Decimal):
               return float(value)
           return value
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] BhuviTheDataGuy closed issue #11551: PostgresToGCSOperator - Exporting the Timestamp column in EPOCH format

Posted by GitBox <gi...@apache.org>.
BhuviTheDataGuy closed issue #11551:
URL: https://github.com/apache/airflow/issues/11551


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org