You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/04/13 19:33:17 UTC
[GitHub] [beam] MarcoRob opened a new pull request, #17365: [BEAM-12482]-Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
MarcoRob opened a new pull request, #17365:
URL: https://github.com/apache/beam/pull/17365
As [PR-14113](https://github.com/apache/beam/pull/14113), adding a zero row load job to bigquery to update the schema when using temporary tables before temp tables copy data into the final destination table, so temp tables schema matches the destination table.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
aaltay commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1112495051
@pabloem - Could you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] steveniemitz commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1190690917
This seems like it introduced a bug, this used to work in beam 2.39:
```
BigQueryIO
.write()
.optimizedWrites()
.to(OutputTable)
.withAvroFormatFunction(abc)
.withAvroSchemaFactory(xyz)
.withCreateDisposition(CreateDisposition.CREATE_NEVER)
.withWriteDisposition(WriteDisposition.WRITE_TRUNCATE))
```
but now throws an exception if you end up in MultiPartitionsWriteTables:
```
Caused by: java.lang.IllegalArgumentException: Unless create disposition is CREATE_NEVER, a schema must be specified, i.e. DynamicDestinations.getSchema() may not return null. However, create disposition is CREATE_IF_NEEDED, and org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinationsHelpers$ConstantTableDestinations@52e95e1 returned null for destination tableSpec: <table>
at org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull(Preconditions.java:436)
at org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:207)
```
The reason seems like now, the create disposition is set to `CreateDisposition.CREATE_IF_NEEDED` no matter what the user sets it to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1121694580
my bad. I'll review today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
aaltay commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1125510282
Could this be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1121416302
Post commit looks healthy. Next steps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1126297595
thanks LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
pabloem merged PR #17365:
URL: https://github.com/apache/beam/pull/17365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MarcoRob commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
MarcoRob commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1098560482
R: @pabloem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #17365: [BEAM-12482]-Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1098408894
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #17365: [BEAM-12482]-Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1098408893
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #17365: [BEAM-12482] Update Schema Destination during Bigquery load job when using temporary tables using zeroloadjob
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #17365:
URL: https://github.com/apache/beam/pull/17365#issuecomment-1116458009
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org