You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/28 21:03:46 UTC

[GitHub] [airflow] mik-laj opened a new issue #13351: Some hacks in google providers

mik-laj opened a new issue #13351:
URL: https://github.com/apache/airflow/issues/13351


   Hello,
   
   While [updating Google libraries](https://github.com/apache/airflow/issues/12116), we found some bugs in `google-cloud-*` libraries.  Fortunately, we managed to get around some of them using various tricks and hacks. However, this may cause various problems in the future after updating the new library, e.g. the hack will not support new features provided by the updates.  We should report problems to the library authors, record the use of hacks, track the progress of tickets, so that we can remove hacks in the future.
   
   **google-cloud-python:**
   https://github.com/apache/airflow/blob/9a1d3820d6f1373df790da8751f25e723f9ce037/airflow/providers/google/cloud/hooks/datacatalog.py#L1024
   https://github.com/apache/airflow/blob/9a1d3820d6f1373df790da8751f25e723f9ce037/airflow/providers/google/cloud/hooks/datacatalog.py#L1101
   https://github.com/apache/airflow/blob/9a1d3820d6f1373df790da8751f25e723f9ce037/airflow/providers/google/cloud/hooks/datacatalog.py#L1024
   https://github.com/apache/airflow/blob/9235af594c54d4a2991c494929deef37eee33ac6/airflow/providers/google/cloud/hooks/datacatalog.py#L232-L234 (unmerged PR: https://github.com/apache/airflow/pull/13224)
   https://github.com/apache/airflow/blob/9235af594c54d4a2991c494929deef37eee33ac6/airflow/providers/google/cloud/hooks/datacatalog.py#L295-L297 (unmerged PR: https://github.com/apache/airflow/pull/13224)
   
   Ticket:  https://github.com/googleapis/python-datacatalog/issues/84
   
   ** google-cloud-bigquery-datatransfer:**
   https://github.com/apache/airflow/blob/10ffc7bfd58f3fa0eea54262f20aeffe1ab72592/airflow/providers/google/cloud/hooks/bigquery_dts.py#L82-L83 (unmerged PR: https://github.com/apache/airflow/pull/13337/files)
   
   Ticket: https://github.com/googleapis/python-bigquery-datatransfer/issues/90
   
   Best regards,
   Kamil BreguĊ‚a


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #13351: Some hacks in Google provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #13351:
URL: https://github.com/apache/airflow/issues/13351#issuecomment-1067200325


   yeah we have a lot of work with updating the google provider
   see https://github.com/apache/airflow/issues/22111#issuecomment-1063433467
   
   You can send me message on slack if you need help with finding another issue to work on


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun edited a comment on issue #13351: Some hacks in Google provider

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on issue #13351:
URL: https://github.com/apache/airflow/issues/13351#issuecomment-1067188899


   Hello @eladkal,
   
   I have taken a look at it and I do not see an easy way to fix this for now:
   - https://github.com/googleapis/proto-plus-python/issues/161 is still open and waiting for a fix.
   - https://github.com/googleapis/python-bigquery-datatransfer/issues/90 has been fixed in [proto-plus-python](https://github.com/googleapis/proto-plus-python) in [v1.19.6](https://github.com/googleapis/proto-plus-python/tree/v1.19.6). This package is used by [python-datacatalog](https://github.com/googleapis/python-datacatalog/blob/v3.7.1/setup.py) that only requires `proto-plus >= 1.4.0` for the oldest version that we support. (`google-cloud-datacatalog>=3.0.0`)
   
   I think we should leave it as is for now, let me know what you think.
   
   Best,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on issue #13351: Some hacks in Google provider

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on issue #13351:
URL: https://github.com/apache/airflow/issues/13351#issuecomment-1046227040


   Hello @eladkal,
   
   Thank you for pointing me to this issue. I will take a look and see if we can remove some of them. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun edited a comment on issue #13351: Some hacks in Google provider

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on issue #13351:
URL: https://github.com/apache/airflow/issues/13351#issuecomment-1067188899


   Hello @eladkal,
   
   I have taken a look at it and I do not see an easy way to fix this for now:
   - https://github.com/googleapis/proto-plus-python/issues/161 is still open and waiting for a fix.
   - https://github.com/googleapis/python-bigquery-datatransfer/issues/90 has been fixed in the [proto-plus-python](https://github.com/googleapis/proto-plus-python) in [v1.19.6](https://github.com/googleapis/proto-plus-python/tree/v1.19.6). This package is used by [python-datacatalog](https://github.com/googleapis/python-datacatalog/blob/v3.7.1/setup.py) that only requires `proto-plus >= 1.4.0` for the oldest version that we supports. (`google-cloud-datacatalog>=3.0.0`)
   
   I think we should leave it as is for now, let me know what you think.
   
   Best,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #13351: Some hacks in Google provider

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #13351:
URL: https://github.com/apache/airflow/issues/13351#issuecomment-1041151101


   @pierrejeambrun maybe you'll be interested in trying to solve some of the hacks? (probably we can't solve all of them)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun commented on issue #13351: Some hacks in Google provider

Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on issue #13351:
URL: https://github.com/apache/airflow/issues/13351#issuecomment-1067188899


   Hello @eladkal,
   
   I have taken a look at it and I do not see an easy way to fix this for now:
   - https://github.com/googleapis/proto-plus-python/issues/161 is still open and waiting for a fix.
   - https://github.com/googleapis/python-bigquery-datatransfer/issues/90 has been fixed in the [proto-plus-python](https://github.com/googleapis/proto-plus-python) in [v1.19.6](https://github.com/googleapis/proto-plus-python/tree/v1.19.6). This package is used by python-datacatalog.
   
   I think we should leave it as is for now, let me know what you think.
   
   Best,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pierrejeambrun edited a comment on issue #13351: Some hacks in Google provider

Posted by GitBox <gi...@apache.org>.
pierrejeambrun edited a comment on issue #13351:
URL: https://github.com/apache/airflow/issues/13351#issuecomment-1067188899


   Hello @eladkal,
   
   I have taken a look at it and I do not see an easy way to fix this for now:
   - https://github.com/googleapis/proto-plus-python/issues/161 is still open and waiting for a fix.
   - https://github.com/googleapis/python-bigquery-datatransfer/issues/90 has been fixed in [proto-plus-python](https://github.com/googleapis/proto-plus-python) in [v1.19.6](https://github.com/googleapis/proto-plus-python/tree/v1.19.6). This package is used by [python-datacatalog](https://github.com/googleapis/python-datacatalog/blob/v3.7.1/setup.py) that only requires `proto-plus >= 1.4.0` for the oldest version that we supports. (`google-cloud-datacatalog>=3.0.0`)
   
   I think we should leave it as is for now, let me know what you think.
   
   Best,


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org