You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/23 09:34:56 UTC

[GitHub] [airflow] stavhaco opened a new issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

stavhaco opened a new issue #16008:
URL: https://github.com/apache/airflow/issues/16008


   **Apache Airflow version**:1.10.12
   **Environment**: google cloud composer
   
   **What happened**:
   When using GoogleCloudStorageToBigQueryOperator and providing string as parameter source_objects, the process is iterating on a the string as a valid list.
   For example - 
   `cloud_storage_to_bigquery = GoogleCloudStorageToBigQueryOperator(
           bucket = 'bucket',
           source_objects = 'abc',
       )`
   Will result in looking into the sources: bucket/a, bucket/b, bucket/c.
   
   **What you expected to happen**:
   Throw an error on type (string instead of list). 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #16008:
URL: https://github.com/apache/airflow/issues/16008#issuecomment-846544557


   `GoogleCloudStorageToBigQueryOperator` is deprecated. Please use `GCSToBigQueryOperator` from providers.
   Note that `source_objects` expect `list` not `str`. You should set `source_objects = ['abc']`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #16008:
URL: https://github.com/apache/airflow/issues/16008#issuecomment-846557575


   You can just open a PR
   BTW probably a better solution would be to allow `str or List[str]` rather than raise error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #16008:
URL: https://github.com/apache/airflow/issues/16008


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #16008:
URL: https://github.com/apache/airflow/issues/16008#issuecomment-846550629


   PRs are welcome :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] stavhaco commented on issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
stavhaco commented on issue #16008:
URL: https://github.com/apache/airflow/issues/16008#issuecomment-846556939


   @eladkal Alright. Can this be assigned to me?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] stavhaco commented on issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
stavhaco commented on issue #16008:
URL: https://github.com/apache/airflow/issues/16008#issuecomment-846550323


   @eladkal seems like the behaviour is the same in GCSToBigQueryOperator.
   I get that the parameter should be set as `source_objects = ['abc']` but shouldn't I expect type error when setting it to `source_objects = 'abc'`? instead of iterating on the chars of the string and looking for different paths based on it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16008:
URL: https://github.com/apache/airflow/issues/16008#issuecomment-846533387


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #16008: GoogleCloudStorageToBigQueryOperator reads string as a list in parameter source_objects

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #16008:
URL: https://github.com/apache/airflow/issues/16008


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org