You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Strikerrx01 (via GitHub)" <gi...@apache.org> on 2025/02/27 19:16:38 UTC

[PR] [BEAM-34076] Cache BigQuery table definitions to reduce API calls [beam]

Strikerrx01 opened a new pull request, #34104:
URL: https://github.com/apache/beam/pull/34104

   This commit implements table definition caching for BigQuery operations to address excessive tables.get() API calls from workers. Key changes: 1) Adds BigQueryTableCache class with configurable TTL 2) Implements thread-safe caching mechanism 3) Adds comprehensive test coverage 4) Maintains backward compatibility. The implementation reduces API quota usage by caching table definitions with proper TTL and thread-safety mechanisms. Fixes #34076                                 ## Description
   
   This PR implements table definition caching for BigQuery operations to address excessive `tables.get()` API calls from workers, which was causing quota issues. The implementation adds a `BigQueryTableCache` class that caches table definitions with a configurable TTL.
   
   ### Key Features
   - Thread-safe caching mechanism for BigQuery table definitions
   - Configurable TTL (default: 5 minutes)
   - Proper error handling and cache invalidation
   - Comprehensive test coverage
   - Zero disruption to existing functionality
   
   ## Motivation
   
   Previously, each worker was making separate `tables.get()` calls to BigQuery, leading to potential quota issues. This change reduces API calls by caching table definitions, improving performance and reliability.
   
   Fixes #34076
   
   ## Modifications
   
   - Added `BigQueryTableCache` class in `bigquery.py`
   - Added validation for table references and TTL
   - Implemented thread-safe cache operations
   - Added comprehensive test suite in `bigquery_test.py`
   
   ## Verifications
   
   ### Testing
   All tests pass successfully:
   - Cache miss behavior
   - Cache hit verification
   - TTL expiration
   - Thread safety
   - Error handling
   - Input validation
   - Invalid TTL handling
   
   ### Documentation
   - Added docstrings for the new class and methods
   - Updated relevant comments
   - Added type hints for better code clarity
   
   ## Please Review
   
   @apache/beam-maintainers 
   
   ## Checklist
   - [x] My code follows the style guidelines of Apache Beam
   - [x] I have performed a self-review of my own code
   - [x] I have commented my code, particularly in hard-to-understand areas
   - [x] I have made corresponding changes to the documentation
   - [x] My changes generate no new warnings
   - [x] I have added tests that prove my fix is effective
   - [x] New and existing unit tests pass locally with my changes
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [BEAM-34076] Cache BigQuery table definitions to reduce API calls [beam]

Posted by "Strikerrx01 (via GitHub)" <gi...@apache.org>.
Strikerrx01 commented on PR #34104:
URL: https://github.com/apache/beam/pull/34104#issuecomment-2692256532

   Apologies for the mistake. I will create a new PR shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [BEAM-34076] Cache BigQuery table definitions to reduce API calls [beam]

Posted by "Strikerrx01 (via GitHub)" <gi...@apache.org>.
Strikerrx01 commented on PR #34104:
URL: https://github.com/apache/beam/pull/34104#issuecomment-2690943474

   @liferoad Thank you for the review. I've addressed the formatting and lint issues in commit c908cec. Changes include:
   
   1. Code Style:
      - Updated indentation to use 2 spaces (Apache Beam style)
      - Fixed docstring formats following Beam guidelines
      - Improved type hints and documentation
      - Consistent naming conventions
   
   2. Code Organization:
      - Removed duplicate test class
      - Consolidated test cases into a single comprehensive test suite
      - Improved test organization and documentation
   
   3. Error Handling:
      - Enhanced error handling with proper exception propagation
      - Added comprehensive error handling tests
      - Improved error documentation in docstrings
   
   4. Documentation:
      - Updated class and method documentation
      - Added clear docstrings for all methods
      - Improved error handling documentation
      - Updated CHANGES.md with clear feature description
   
   All tests are now passing locally. Please let me know if you'd like me to make any additional improvements.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [BEAM-34076] Cache BigQuery table definitions to reduce API calls [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #34104:
URL: https://github.com/apache/beam/pull/34104#issuecomment-2690808500

   @Strikerrx01 Thanks for contributing to Beam. Please fix formats and lint.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [BEAM-34076] Cache BigQuery table definitions to reduce API calls [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #34104:
URL: https://github.com/apache/beam/pull/34104#issuecomment-2692251850

   Any reason to close this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [BEAM-34076] Cache BigQuery table definitions to reduce API calls [beam]

Posted by "Strikerrx01 (via GitHub)" <gi...@apache.org>.
Strikerrx01 closed pull request #34104: [BEAM-34076] Cache BigQuery table definitions to reduce API calls
URL: https://github.com/apache/beam/pull/34104


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [BEAM-34076] Cache BigQuery table definitions to reduce API calls [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34104:
URL: https://github.com/apache/beam/pull/34104#issuecomment-2689057058

   Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org