You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/18 21:57:11 UTC

[GitHub] [airflow] pierrejeambrun opened a new pull request, #23063: PostgresToGCSOperator does not allow nested JSON

pierrejeambrun opened a new pull request, #23063:
URL: https://github.com/apache/airflow/pull/23063

   fix issue: https://github.com/apache/airflow/issues/23040
   
   This is due to a double json.dumps when exporting to json format:
    - https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/postgres_to_gcs.py#L153
    - https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/sql_to_gcs.py#L221
   
   I added a parameter to the `convert_type` that allow us to chose what to do with the dict type objects. For `parquet` and `csv` we want to stringify them. But we want to keep them as dict when exporting to the `json` format.
   
   csv and parquet export are hence not modified.
   
   I added data to the tests so we assert json column export.
   
   Regards,
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on pull request #23063: PostgresToGCSOperator does not allow nested JSON

Posted by GitBox <gi...@apache.org>.

pierrejeambrun commented on PR #23063:
URL: https://github.com/apache/airflow/pull/23063#issuecomment-1103010315

   Hello @eladkal,
   
   Here if a first draft of the PR just in case you want to take a look :)
   
   Best,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #23063: Fix `PostgresToGCSOperator` does not allow nested JSON

Posted by GitBox <gi...@apache.org>.

potiuk merged PR #23063:
URL: https://github.com/apache/airflow/pull/23063


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #23063: Fix `PostgresToGCSOperator` does not allow nested JSON

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on PR #23063:
URL: https://github.com/apache/airflow/pull/23063#issuecomment-1115766097

   The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on pull request #23063: Fix `PostgresToGCSOperator` does not allow nested JSON

Posted by GitBox <gi...@apache.org>.

eladkal commented on PR #23063:
URL: https://github.com/apache/airflow/pull/23063#issuecomment-1121222753

   Thanks @pierrejeambrun hopeful someday we will have https://github.com/apache/airflow/issues/21599 resolved so we won't need dedicated operators per db


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on pull request #23063: Fix `PostgresToGCSOperator` does not allow nested JSON

Posted by GitBox <gi...@apache.org>.

pierrejeambrun commented on PR #23063:
URL: https://github.com/apache/airflow/pull/23063#issuecomment-1121395503

   @eladkal, my pleasure. Matter of fact I was wondering about that as there are strong similarities on some part of the code.
   
   Good to know there already is an issue tracking that, it's not going to be an easy one though :smile: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org