You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "xianhualiu (via GitHub)" <gi...@apache.org> on 2023/04/13 23:07:48 UTC
[GitHub] [beam] xianhualiu opened a new pull request, #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
xianhualiu opened a new pull request, #26273:
URL: https://github.com/apache/beam/pull/26273
When STORAGE_WRITE_API is used as the method for BigQueryIO.Write, autoSharding needs to be enabled. To avoid issues, by default autoSharding will be enabled when STORAGE_WRITE_API is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] xianhualiu commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "xianhualiu (via GitHub)" <gi...@apache.org>.
xianhualiu commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1559410954
> If I understood correctly autoSharding uses GroupIntoBatches.withShardedKey. It is only supported in Dataflow runner v2. Under runner v1 this flag would cause error. Context: #26416 switching the default would break runner v1 user
The autoSharing default change in this PR only impacts how data is written into BigQuery table in the BigQueryIO.Write client when STORAGE_WRITE_API is used as method. When STORAGE_WRITE_API is used, autoSharding needs to be enabled by default. Also, this change only impact SchemaIO transforms used in Syndeo templates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1533146357
If I understood correctly autoSharding uses GroupIntoBatches.withShardedKey. It is only supported in Dataflow runner v2. Under runner v1 this flag would cause error. Context: #26416 switching the default would break runner v1 user
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1517731848
Reminder, please take a look at this pr: @Abacn @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn merged pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #26273:
URL: https://github.com/apache/beam/pull/26273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1559361365
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1543887525
Reminder, please take a look at this pr: @robertwb @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1507730256
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @Abacn for label java.
R: @Abacn for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1523316878
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @robertwb for label java.
R: @johnjcasey for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1532920910
Reminder, please take a look at this pr: @robertwb @johnjcasey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1559169366
Reminder, please take a look at this pr: @kennknowles @pabloem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26273: Use default value true for autoSharding if not configured for BigQuery with STORAGE_WRITE_API
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26273:
URL: https://github.com/apache/beam/pull/26273#issuecomment-1547743074
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @kennknowles for label java.
R: @pabloem for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org