You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/15 19:54:12 UTC

[GitHub] [airflow] potiuk opened a new pull request #20326: Fix missing get_backup method for Dataproc Metastore

potiuk opened a new pull request #20326:
URL: https://github.com/apache/airflow/pull/20326


   Found during #19891 fixing (yay! MyPy actually found some real errors).
   
   The `get_backup` method was missing in Dataproc Metastore
   implementation.
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk merged pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #20326:
URL: https://github.com/apache/airflow/pull/20326


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-1032735465


   BTW: we still have a chance to fix it today as I am preparing to publish RC2 for providers - so if this is wrong - it would be great to get a better fix today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] lwyszomi commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
lwyszomi commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-1032721109


   @potiuk what errors we had without this? because we didn't add this method intentionally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-995263174


   The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] lwyszomi commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
lwyszomi commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-1032746031


   @potiuk  We decided that we should not have a operator for this hook method, I didn't know that we using this in the create backup operator. I don't know how this happen and we didn't discovered this in the tests.
   
   This is really good fix to add this method to the hook. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-1032921631


   Thank MyPy for that, not me :). I merely followed MyPy screaming " pleeeeease add the method " at me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-1032734137


   Why Intentionally? It does not look like this was intentional, because it has been called in the operator (in case of error, but it was). It could be that there was another mistake in the operator that called this method - but this did seem like a missing method that is used elsewhere?
   
   https://github.com/apache/airflow/blob/837ff7ed10d440efe1f5e5149821d9bb2df4f383/airflow/providers/google/cloud/operators/dataproc_metastore.py#L138
   
   ```
           except HttpError as err:
               if err.resp.status not in (409, '409'):
                   raise
               self.log.info("Backup %s already exists", self.backup_id)
               backup = hook.get_backup(
                   project_id=self.project_id,
                   region=self.region,
                   service_id=self.service_id,
                   backup_id=self.backup_id,
                   retry=self.retry,
                   timeout=self.timeout,
                   metadata=self.metadata,
               )
           return Backup.to_dict(backup)
   
   ```
   
   What do you think is the right fix @lwyszomi ? Should id be replaced with something else here? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-1032736491


   BTW ^^ this is what MyPy discovered :). I have no such hawk-eye :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-995166053


   Yay! Found a real error thanks to MyPY :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20326: Fix missing get_backup method for Dataproc Metastore

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20326:
URL: https://github.com/apache/airflow/pull/20326#issuecomment-1032920814


   Cool :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org