You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/08 15:18:00 UTC

[GitHub] [airflow] potiuk opened a new pull request, #24330: Workaround job race bug on biguery to gcs transfer

potiuk opened a new pull request, #24330:
URL: https://github.com/apache/airflow/pull/24330

   Fixes: #24277
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] github-actions[bot] commented on pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24330:
URL: https://github.com/apache/airflow/pull/24330#issuecomment-1150088806

   The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
potiuk merged PR #24330:
URL: https://github.com/apache/airflow/pull/24330


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24330:
URL: https://github.com/apache/airflow/pull/24330#issuecomment-1150061617

   cc: @wojsamjan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24330:
URL: https://github.com/apache/airflow/pull/24330#discussion_r892559999


##########
airflow/providers/google/cloud/hooks/bigquery.py:
##########
@@ -1915,6 +1916,7 @@ def run_extract(
         :param print_header: Whether to print a header for a CSV file extract.
         :param labels: a dictionary containing labels for the job/query,
             passed to BigQuery
+        :param return_full_job: return full job instead of job id only

Review Comment:
   This function is deprecated.
   By adding this fix what does it means for when we will remove the function?
   Would we have to support the new functionality as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24330:
URL: https://github.com/apache/airflow/pull/24330#discussion_r892600837


##########
airflow/providers/google/cloud/hooks/bigquery.py:
##########
@@ -1915,6 +1916,7 @@ def run_extract(
         :param print_header: Whether to print a header for a CSV file extract.
         :param labels: a dictionary containing labels for the job/query,
             passed to BigQuery
+        :param return_full_job: return full job instead of job id only

Review Comment:
   Technically speaking this functionality (returning job rather than job id) is already supported by the function that replaces this deprecated one.
   
   The function that  this one replaces already returns BigQueryJob rather than job id (I actually got the idea on how to fix it by looking at the new "insert_job" function. So if you are moving out from using this deprecated function it will only be easier to migrate, 
   
   But honestly i care more about being able to release new provider than worrying about this :). I actually looked at replacing this deprecated function and even started doing it, and I would just not attempt to fix it properly (i.e. replace the run_extract with insert_job - because you need to thoroughly test it (not only return, but also input parameters changes). The change I implemented can be simply released based on unit testing and review. 
   
   But this bug - technically a regression) block from releasing Google Provider. So I think better question what alternative we have:
   
   * stop releasing the provider? 
   * implement and it properly (i.e. replace it with `insert_job`). 
   
   If you think you - or anyone else - could fix and test it in few hours (I am about to prepare an RC2) - feel free :D. I am happy to merge it :)
   
   But if not - I will just proceed with releasing it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24330:
URL: https://github.com/apache/airflow/pull/24330#discussion_r892600837


##########
airflow/providers/google/cloud/hooks/bigquery.py:
##########
@@ -1915,6 +1916,7 @@ def run_extract(
         :param print_header: Whether to print a header for a CSV file extract.
         :param labels: a dictionary containing labels for the job/query,
             passed to BigQuery
+        :param return_full_job: return full job instead of job id only

Review Comment:
   Technically speaking this functionality (returning job rather than job id) is already supported by the function that replaces this deprecated one.
   
   The function that  this one replaces already returns BigQueryJob rather than job id (I actually got the idea on how to fix it by looking at the new "insert_job" function. So if you are moving out from using this deprecated function it will only be easier to migrate, 
   
   But honestly i care more about being able to release new provider than worrying about this :). I actually looked at replacing this deprecated function and even started doing it, and I would just not attempt to fix it properly (i.e. replace the run_extract with insert_job - because you need to thoroughly test it (not only return, but also input parameters changes). The change I implemented can be simply released based on unit testing and review. 
   
   But this bug - (technically a regression) block from releasing Google Provider. So I think better question what alternative we have:
   
   * stop releasing the provider until someone fixes it "properly"? 
   * implement and it properly (i.e. replace it with `insert_job`). 
   
   If you think you - or anyone else - could fix and test it in few hours (I am about to prepare an RC2) - feel free :D. I am happy to merge it :)
   
   But if not - I will just proceed with releasing it.



##########
airflow/providers/google/cloud/hooks/bigquery.py:
##########
@@ -1915,6 +1916,7 @@ def run_extract(
         :param print_header: Whether to print a header for a CSV file extract.
         :param labels: a dictionary containing labels for the job/query,
             passed to BigQuery
+        :param return_full_job: return full job instead of job id only

Review Comment:
   Technically speaking this functionality (returning job rather than job id) is already supported by the function that replaces this deprecated one.
   
   The function that  this one replaces already returns BigQueryJob rather than job id (I actually got the idea on how to fix it by looking at the new "insert_job" function. So if you are moving out from using this deprecated function it will only be easier to migrate, 
   
   But honestly i care more about being able to release new provider than worrying about this :). I actually looked at replacing this deprecated function and even started doing it, and I would just not attempt to fix it properly (i.e. replace the run_extract with insert_job - because you need to thoroughly test it (not only return, but also input parameters changes). The change I implemented can be simply released based on unit testing and review. 
   
   But this bug - (technically a regression) block from releasing Google Provider. So I think better question what alternative we have:
   
   * stop releasing the provider until someone fixes it "properly"? 
   * implement  it properly and test now (i.e. replace it with `insert_job`). 
   
   If you think you - or anyone else - could fix and test it in few hours (I am about to prepare an RC2) - feel free :D. I am happy to merge it :)
   
   But if not - I will just proceed with releasing it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24330:
URL: https://github.com/apache/airflow/pull/24330#discussion_r892600837


##########
airflow/providers/google/cloud/hooks/bigquery.py:
##########
@@ -1915,6 +1916,7 @@ def run_extract(
         :param print_header: Whether to print a header for a CSV file extract.
         :param labels: a dictionary containing labels for the job/query,
             passed to BigQuery
+        :param return_full_job: return full job instead of job id only

Review Comment:
   Technically speaking this functionality (returning job rather than job id) is already supported by the function that replaces this deprecated one.
   
   The function that  this one replaces already returns BigQueryJob rather than job id (I actually got the idea on how to fix it by looking at the new "insert_job" function. So if you are moving out from using this deprecated function it will only be easier to migrate, 
   
   But nonestly i care more about being able to release new provider than worrying about this :). I actually looked at replacing this deprecated function and even started doing it, and I would just not attempt to fix it properly (i.e. replace the run_extract with insert_job - because you need to thoroughly test it (not only return, but also input parameters changes). The change I implemented can be simply released based on unit testing and review. 
   
   But this bug - technically a regression) block from releasing Google Provider. So I think better question what alternative we have:
   
   * stop releasing the provider? 
   * implement and it properly (i.e. replace it with `insert_job`). 
   
   If you think you - or anyone else - could fix and test it in few hours (I am about to prepare an RC2) - feel free :D. I am happy to merge it :)
   
   But if not - I will just proceed with releasing it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24330: Workaround job race bug on biguery to gcs transfer

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24330:
URL: https://github.com/apache/airflow/pull/24330#discussion_r892600837


##########
airflow/providers/google/cloud/hooks/bigquery.py:
##########
@@ -1915,6 +1916,7 @@ def run_extract(
         :param print_header: Whether to print a header for a CSV file extract.
         :param labels: a dictionary containing labels for the job/query,
             passed to BigQuery
+        :param return_full_job: return full job instead of job id only

Review Comment:
   Technically speaking this functionality (returning job rather than job id) is already supported by the function that replaces this deprecated one.
   
   The function that  this one replaces already returns BigQueryJob rather than job id (I actually got the idea on how to fix it by looking at the new "insert_job" function. So if you are moving out from using this deprecated function it will only be easier to migrate, 
   
   But honestly i care more about being able to release new provider than worrying about this :). I actually looked at replacing this deprecated function and even started doing it, and I would just not attempt to fix it properly (i.e. replace the run_extract with insert_job - because you need to thoroughly test it (not only return, but also input parameters changes). The change I implemented can be simply released based on unit testing and review. 
   
   But this bug - (technically a regression) block from releasing Google Provider. So I think better question what alternative we have:
   
   * stop releasing the provider? 
   * implement and it properly (i.e. replace it with `insert_job`). 
   
   If you think you - or anyone else - could fix and test it in few hours (I am about to prepare an RC2) - feel free :D. I am happy to merge it :)
   
   But if not - I will just proceed with releasing it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org