You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tgaddy (via GitHub)" <gi...@apache.org> on 2023/02/06 19:00:58 UTC

[GitHub] [beam] tgaddy opened a new pull request, #25352: add write options with create disposition for bigqueryio

tgaddy opened a new pull request, #25352:
URL: https://github.com/apache/beam/pull/25352

   Addresses [25318](https://github.com/apache/beam/issues/25318), allows one to set a create disposition when writing to BigQuery so that the user can control whether a table should be created or not. I tried to make the changes backwards-compatible. 
   
   I'm a newbie to Go and beam, so I was hoping for some guidance around testing. I wasn't quite sure if it was worth creating a unit test for this small logic, or perhaps whether I should create some sort of integration test to see whether a pipeline integrates correctly with bigquery. Any suggestions would be most welcome! 
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1453444777

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @damccorm for label go.
   R: @pabloem for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jrmccluskey commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1470765498

   Run Go PostCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jrmccluskey commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1422786805

   CC: @lostluck 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck commented on a diff in pull request #25352: add write options with create disposition for bigqueryio

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #25352:
URL: https://github.com/apache/beam/pull/25352#discussion_r1100420083


##########
sdks/go/pkg/beam/io/bigqueryio/bigquery.go:
##########
@@ -208,29 +208,58 @@ func mustParseTable(table string) QualifiedTableName {
 // TODO(herohde) 7/14/2017: allow CreateDispositions and WriteDispositions. The default

Review Comment:
   Please update the TODO, since it's at least partially accomplished now.



##########
sdks/go/pkg/beam/io/bigqueryio/bigquery.go:
##########
@@ -208,29 +208,58 @@ func mustParseTable(table string) QualifiedTableName {
 // TODO(herohde) 7/14/2017: allow CreateDispositions and WriteDispositions. The default
 // is not quite what the Dataflow examples do.
 
+// WriteOptions represents additional options for executing a write
+type WriteOptions struct {
+	// CreateDisposition specifies the circumstances under which destination table will be created
+	CreateDisposition bigquery.TableCreateDisposition
+}
+
+// newWriteOptions creates a new instance of WriteOptions
+// "CreateIfNeeded" is set as the default write disposition
+func newWriteOptions() WriteOptions {
+	return WriteOptions{CreateDisposition: bigquery.CreateIfNeeded}
+}
+
+// WithCreateDisposition specifies the circumstances under which destination table will be created
+func WithCreateDisposition(cd bigquery.TableCreateDisposition) func(wo *WriteOptions) error {
+	return func(wo *WriteOptions) error {
+		wo.CreateDisposition = cd
+		return nil
+	}
+}
+
 // Write writes the elements of the given PCollection<T> to bigquery. T is required
 // to be the schema type.
-func Write(s beam.Scope, project, table string, col beam.PCollection) {
+func Write(s beam.Scope, project, table string, col beam.PCollection, options ...func(*WriteOptions) error) {

Review Comment:
   Fun fact: This is actually a backwards incompatible change since if some user somewhere were metaprograming and passing around the `Write` function, it's type would have changed!
   
   But since the Go SDK isn't expecting that level of metaprogramming, we accept this sort of breaking change, since it's a vanishingly small fraction of users who would be affected. They already know they're being tricky.
   
   And on the plus side, you've made it so we don't need to make this style of breaking change for this function in the future. Well done!



##########
sdks/go/pkg/beam/io/bigqueryio/bigquery.go:
##########
@@ -208,29 +208,58 @@ func mustParseTable(table string) QualifiedTableName {
 // TODO(herohde) 7/14/2017: allow CreateDispositions and WriteDispositions. The default
 // is not quite what the Dataflow examples do.
 
+// WriteOptions represents additional options for executing a write
+type WriteOptions struct {
+	// CreateDisposition specifies the circumstances under which destination table will be created
+	CreateDisposition bigquery.TableCreateDisposition
+}
+
+// newWriteOptions creates a new instance of WriteOptions
+// "CreateIfNeeded" is set as the default write disposition
+func newWriteOptions() WriteOptions {
+	return WriteOptions{CreateDisposition: bigquery.CreateIfNeeded}
+}
+
+// WithCreateDisposition specifies the circumstances under which destination table will be created
+func WithCreateDisposition(cd bigquery.TableCreateDisposition) func(wo *WriteOptions) error {

Review Comment:
   Instead of using a vanilla function, declare the function type *as* the exported options type.
   
   `type WriteOption func(*writeOptions)`
   
   This avoids the "you're using unexported types in an Exported API" style messages, and provides something we can document on.
   
   This is preferable to simply having an aribitrary visible struct, when the intended mode isn't for users to write their own options functions. Having both "user can write this function" and "helper methods" it makes the API harder to use than "call functions X Y Z to set X Y Z"



##########
sdks/go/pkg/beam/io/bigqueryio/bigquery.go:
##########
@@ -208,29 +208,58 @@ func mustParseTable(table string) QualifiedTableName {
 // TODO(herohde) 7/14/2017: allow CreateDispositions and WriteDispositions. The default
 // is not quite what the Dataflow examples do.
 
+// WriteOptions represents additional options for executing a write
+type WriteOptions struct {
+	// CreateDisposition specifies the circumstances under which destination table will be created
+	CreateDisposition bigquery.TableCreateDisposition

Review Comment:
   I'm a tiny bit concerned whether this will serialize/deserialize correctly since it's a string derived type.  I think it should work, since DoFns are still serialized with JSON and that supports this behavior. 
   
   https://pkg.go.dev/cloud.google.com/go/bigquery#TableCreateDisposition
   
   If we finish moving DoFn serialization to beam schemas, we'll simply have to ensure that this behavior is supported.



##########
sdks/go/pkg/beam/io/bigqueryio/bigquery.go:
##########
@@ -208,29 +208,58 @@ func mustParseTable(table string) QualifiedTableName {
 // TODO(herohde) 7/14/2017: allow CreateDispositions and WriteDispositions. The default
 // is not quite what the Dataflow examples do.
 
+// WriteOptions represents additional options for executing a write
+type WriteOptions struct {

Review Comment:
   It's not necessary to export this type. Types can be unexported and still be serialized /deserialized properly. It's their field names that Must be exported.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] lostluck merged pull request #25352: add write options with create disposition for bigqueryio

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck merged PR #25352:
URL: https://github.com/apache/beam/pull/25352


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tgaddy commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "tgaddy (via GitHub)" <gi...@apache.org>.
tgaddy commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1448767963

   @lostluck I tried adding some integration tests now if you want to take a look. 
   
   Ran the tests in my own gcp project with the following command (had to increase the timeout to 15 minutes in order to run all the tests) 
   ```
   go test -v ./test/integration/io/bigqueryio/... --runner=dataflow --project=<PROJECT> --region=europe-west1 --staging_location=<STAGING_LOCATION> --worker_harness_container_image=apache/beam_go_sdk:latest --bq_dataset=<BQ_DATASET> --timeout=15m
   ```
   
   I borrowed heavily from the integration tests in `xlang/bigquery` so that one can potentially add more integration tests in the future for `Read` and `Query`. 
   
   There were a few things I wasn't so sure about (in addition to just normal idiomatic golang stuff I've probably missed):
   
   - Whether to add anything in the `integration.go` filters. There's already a `TestBigQueryIO` (from xlang), so perhaps the existing filters take care of it? Or perhaps I should change the name of this test, although `TestBigQueryIO` seems like the most natural name..
   - I assumed that dataflow is running in the same project as the one in which the bigquery dataset is located. As far as I could tell, the tests using the dataflow runner run in `apache-beam-testing`, which is also where the testing dataset is.
   - I wanted to check that the write actually worked and wrote the expected number of rows. Since it's a streaming write I had to look at `streamingBuffer.EstimatedRows`, but given that its the "estimated" rows perhaps it doesn't make sense to expect that value to be exactly the `inputSize`? 
   - Whether I should add `bq_dataset` option to `dataflowValidatesRunner` task in `beam/sdks/go/test/build.gradle`
   
   Thanks for taking a look! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tgaddy commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "tgaddy (via GitHub)" <gi...@apache.org>.
tgaddy commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1474871372

   > Ah, sorry about that! 
   
   No problem! :) 
   
   >  I'll run the dataflow tests to see how those execute.
   
   Awesome, thank you
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tgaddy commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "tgaddy (via GitHub)" <gi...@apache.org>.
tgaddy commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1425770056

   @lostluck @jrmccluskey Thank you for all the helpful comments! I will work on addressing them and adding the integration tests :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1419763921

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @jrmccluskey for label go.
   R: @Abacn for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] codecov[bot] commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1421039882

   # [Codecov](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#25352](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b8a4c2c) into [master](https://codecov.io/gh/apache/beam/commit/cd20288318d4478c1a62a4d18b4989121def3015?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cd20288) will **decrease** coverage by `0.11%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #25352      +/-   ##
   ==========================================
   - Coverage   73.14%   73.03%   -0.11%     
   ==========================================
     Files         735      738       +3     
     Lines       98161    98437     +276     
   ==========================================
   + Hits        71796    71894      +98     
   - Misses      25002    25178     +176     
   - Partials     1363     1365       +2     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | go | `51.69% <0.00%> (-0.15%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [sdks/go/pkg/beam/io/bigqueryio/bigquery.go](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9iaWdxdWVyeWlvL2JpZ3F1ZXJ5Lmdv) | `9.27% <0.00%> (-0.62%)` | :arrow_down: |
   | [sdks/go/pkg/beam/io/mongodbio/read.go](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9tb25nb2RiaW8vcmVhZC5nbw==) | `9.03% <0.00%> (-21.94%)` | :arrow_down: |
   | [sdks/go/pkg/beam/io/mongodbio/coder.go](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9tb25nb2RiaW8vY29kZXIuZ28=) | `83.33% <0.00%> (-16.67%)` | :arrow_down: |
   | [sdks/go/pkg/beam/io/mongodbio/common.go](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9tb25nb2RiaW8vY29tbW9uLmdv) | `0.00% <0.00%> (ø)` | |
   | [...s/go/pkg/beam/io/mongodbio/id\_range\_restriction.go](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9tb25nb2RiaW8vaWRfcmFuZ2VfcmVzdHJpY3Rpb24uZ28=) | `26.47% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/io/mongodbio/id\_range\_tracker.go](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9tb25nb2RiaW8vaWRfcmFuZ2VfdHJhY2tlci5nbw==) | `73.00% <0.00%> (ø)` | |
   | [sdks/go/pkg/beam/io/mongodbio/id\_range\_split.go](https://codecov.io/gh/apache/beam/pull/25352?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9tb25nb2RiaW8vaWRfcmFuZ2Vfc3BsaXQuZ28=) | `45.57% <0.00%> (ø)` | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jrmccluskey commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1421035628

   Run GoPortable PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jrmccluskey commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1422785674

   I'd definitely recommend an integration test to make sure that the options are being applied correctly, and unit testing the panic() when processing the options is a reasonable bit of code coverage


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1446223394

   Reminder, please take a look at this pr: @jrmccluskey @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jrmccluskey commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1470765349

   Ah, sorry about that! At first glance it looks good to me. For the integration.go filters you should be good to go, we would have seen a failure for the direct runner pre-commit if it wasn't caught. I'll run the dataflow tests to see how those execute. As far as the tasks/gradle updates, that may be something @damccorm can answer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tgaddy commented on pull request #25352: add write options with create disposition for bigqueryio

Posted by "tgaddy (via GitHub)" <gi...@apache.org>.
tgaddy commented on PR #25352:
URL: https://github.com/apache/beam/pull/25352#issuecomment-1470564546

   Hey @jrmccluskey I added the integration tests a while back and tagged @lostluck, but looks like he's been away for a while. Would you mind taking a look?? Thank you :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org