You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/09 15:25:56 UTC

[GitHub] [airflow] patricker opened a new issue, #26273: SQLToGCSOperators Add Support for Dumping JSON

patricker opened a new issue, #26273:
URL: https://github.com/apache/airflow/issues/26273

   ### Description
   
   If your output format for a SQLToGCSOperator is `json`, then any "dict" type object returned from a database, for example a Postgres JSON column, is not dumped to a string and is kept as a nested JSON object.
   
   Add option to dump `dict` objects to string in JSON exporter.
   
   ### Use case/motivation
   
   Currently JSON type columns are hard to ingest into BQ since a JSON field in a source database does not enforce a schema, and we can't reliably generate a `RECORD` schema for the column.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #26273: SQLToGCSOperators Add Support for Dumping JSON

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26273: SQLToGCSOperators Add Support for Dumping JSON
URL: https://github.com/apache/airflow/issues/26273


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] patricker commented on issue #26273: SQLToGCSOperators Add Support for Dumping JSON

Posted by GitBox <gi...@apache.org>.
patricker commented on issue #26273:
URL: https://github.com/apache/airflow/issues/26273#issuecomment-1242133979

   Also, somewhat unrelated, the `schema` generated if a column is of type "JSON" is for a column of type "STRING". If you try to load the data using the generated schema it will fail if you don't dump the dictionaries to string.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org