You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "claudevdm (via GitHub)" <gi...@apache.org> on 2026/10/15 17:19:52 UTC
[PR] Trigger BQ CDC IT. Fix impl [beam]
claudevdm opened a new pull request, #40618:
URL: https://github.com/apache/beam/pull/40618
<!--
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
- [ ] Replace these instructions with a description of your change and why you have made it.
- [ ] Mention the appropriate issue in your description (for example: `addresses #123` or `fixes #456`).
- [ ] Update `CHANGES.md` with noteworthy changes.
See [CONTRIBUTING.md]() for more details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40618:
URL: https://github.com/apache/beam/pull/40618#issuecomment-6293105172
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #40618:
URL: https://github.com/apache/beam/pull/40618#issuecomment-6199830683
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/40618?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 79.99%. Comparing base ([`76f6e6d`](https://app.codecov.io/gh/apache/beam/commit/76f6e6d7c0c1b2ce582c73d068ad9260cf15c5f3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6096ef9`](https://app.codecov.io/gh/apache/beam/commit/6096ef92fd7e52258f1b8d18688418168c694eda?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 198 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #40618 +/- ##
=============================================
+ Coverage 56.16% 79.99% +23.82%
=============================================
Files 1124 488 -636
Lines 178336 82548 -95788
Branches 1489 0 -1489
=============================================
- Hits 100170 66032 -34138
+ Misses 75645 16516 -59129
+ Partials 2521 0 -2521
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/40618/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/40618/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java](https://app.codecov.io/gh/apache/beam/pull/40618/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/beam/pull/40618/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.99% <100.00%> (+0.43%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/beam/pull/40618?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #40618:
URL: https://github.com/apache/beam/pull/40618#discussion_r4326688015
##########
sdks/python/apache_beam/io/gcp/bigquery_change_history.py:
##########
@@ -499,8 +511,7 @@ def _emit_query_ranges(
self,
start_ts: Timestamp,
end_ts: Timestamp,
- watermark_estimator: _PollWatermarkEstimator
- ) -> Iterable[TimestampedValue[_QueryRange]]:
Review Comment:
I think there's an issue with typehinting where if process is annotated to yield TimestampedValue[T] related to https://github.com/apache/beam/issues/20344
E.g.
```
from typing import Iterable
import apache_beam as beam
from apache_beam.transforms.window import TimestampedValue
class AddTimestamp(beam.DoFn):
def process(self, x: int) -> Iterable[TimestampedValue[int]]:
yield TimestampedValue(x, x)
with beam.Pipeline() as p:
ints = p | beam.Create([1, 2, 3]) | beam.ParDo(AddTimestamp())
print('element_type:', ints.element_type)
_ = ints | beam.Map(lambda x: x + 1).with_input_types(int)
```
results in
```
element_type: apache_beam.transforms.window.TimestampedValue[int]
TypeCheckError: The transform 'Map(<lambda ...>)' requires PCollections of type '<class 'int'>'
but was applied to a PCollection of type 'apache_beam.transforms.window.TimestampedValue[int]'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm merged PR #40618:
URL: https://github.com/apache/beam/pull/40618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on PR #40618:
URL: https://github.com/apache/beam/pull/40618#issuecomment-6294145435
> A few questions on the review; just out of curiosity, how did the zoneinfo change surface the lack of test execution/breakage here?
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/08daa781654625f26eeacb10c5bb7adadddffc21/python/src/main/python/bigquery-anomaly-detection/src/bqmonitor/cdc.py is a fork of the bigquery_change_history. So the pyarrow change that broke cdc.py should have also turned these tests red if they were actually running in post commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #40618:
URL: https://github.com/apache/beam/pull/40618#discussion_r4326754397
##########
sdks/python/apache_beam/io/gcp/bigquery_change_history.py:
##########
@@ -1294,20 +1300,22 @@ def expand(self, pbegin: beam.pvalue.PBegin) -> beam.PCollection:
# [start_time, now - buffer) in a single poll.
config = _PollConfig(start_time=start_time)
+ poll = beam.ParDo(
Review Comment:
No reason, moved back inline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #40618:
URL: https://github.com/apache/beam/pull/40618#discussion_r4326437216
##########
sdks/python/apache_beam/io/gcp/bigquery_change_history.py:
##########
@@ -499,8 +511,7 @@ def _emit_query_ranges(
self,
start_ts: Timestamp,
end_ts: Timestamp,
- watermark_estimator: _PollWatermarkEstimator
- ) -> Iterable[TimestampedValue[_QueryRange]]:
Review Comment:
Were the typehints applied here incorrect?
##########
sdks/python/apache_beam/io/gcp/bigquery_change_history.py:
##########
@@ -1294,20 +1300,22 @@ def expand(self, pbegin: beam.pvalue.PBegin) -> beam.PCollection:
# [start_time, now - buffer) in a single poll.
config = _PollConfig(start_time=start_time)
+ poll = beam.ParDo(
Review Comment:
Is there a reason for the refactor of defining the ParDos prior to the pipeline definition both here and elsewhere?
##########
CHANGES.md:
##########
@@ -120,6 +120,9 @@
* (Python) DataFrame API: `agg` with a list that includes `'count'` or `'size'` now labels those rows `count`/`size` instead of `sum` ([#40504](https://github.com/apache/beam/issues/40504)).
* (Python) `ConcatSource` no longer raises `ZeroDivisionError` when every source bundle reports a weight of 0, e.g. bundles smaller than 1 MB ([#21316](https://github.com/apache/beam/issues/21316)).
* (Python) DataFrame API: `Series.value_counts(bins=...)` no longer always fails with a `TypeError` ([#40503](https://github.com/apache/beam/issues/40503)).
+* (Python) `ReadBigQueryChangeHistory` no longer fails at pipeline construction with `TypeCheckError: The transform 'ExecuteQueries' requires PCollections of type ... _QueryRange` when pipeline type checking is enabled (the default); a regression in 2.76.0 ([#X](https://github.com/apache/beam/issues/X)).
+* (Python) `ReadBigQueryChangeHistory` with `change_function='CHANGES'` no longer fails reading its temp table with a Storage Read API `500 InternalServerError`: the `_CHANGE_IS_FOR_UPDATE` pseudo-column is now excluded from the query instead of being written to the temp table ([#X](https://github.com/apache/beam/issues/X)).
+* (Python) `ReadBigQueryChangeHistory` now backtick-quotes the column names it puts in the generated CHANGES/APPENDS query (`columns`, `change_type_column`, `change_timestamp_column`), so reserved words and names with spaces work, and `columns=[]` selects only the pseudo-columns instead of generating invalid SQL ([#X](https://github.com/apache/beam/issues/X)).
Review Comment:
Missing issue links
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Trigger BQ CDC IT. Fix impl [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on PR #40618:
URL: https://github.com/apache/beam/pull/40618#issuecomment-6293033079
R: @jrmccluskey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org