You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "xianhualiu (via GitHub)" <gi...@apache.org> on 2023/03/21 16:55:54 UTC
[GitHub] [beam] xianhualiu opened a new pull request, #25915: validated the input schema matches with the bigquery output schema
xianhualiu opened a new pull request, #25915:
URL: https://github.com/apache/beam/pull/25915
Validated the input schema is assignable to the output schema for BigQuery SchemaIO write transform.
------------------------
Currently when input schema does not match with the BigQuery output table schema, the pipeline runs normal without reporting any error. The schema validation implemented in this PR will cause pipeline with mis-matched schema fail to start with appropriate error reported in log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] xianhualiu commented on pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "xianhualiu (via GitHub)" <gi...@apache.org>.
xianhualiu commented on PR #25915:
URL: https://github.com/apache/beam/pull/25915#issuecomment-1480045210
> could you write a test for this here? https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProviderTest.java#L61 -
>
> perhaps one where the check fails and one where it passes to have confidence in the change?
added successful and failed test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25915:
URL: https://github.com/apache/beam/pull/25915#issuecomment-1478562463
could you write a test for this here? https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProviderTest.java#L61 -
perhaps one where the check fails and one where it passes to have confidence in the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] slilichenko commented on pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "slilichenko (via GitHub)" <gi...@apache.org>.
slilichenko commented on PR #25915:
URL: https://github.com/apache/beam/pull/25915#issuecomment-1496102214
There is a valid use case where this approach will not work - writing to a table with a required field and a default value and relying on BigQuery to use the default value. The only way to use default values at the moment is to explicitly specify the schema without the field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25915:
URL: https://github.com/apache/beam/pull/25915#issuecomment-1490543067
this seems to be causing issues because we expect the projectId to also be passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem merged PR #25915:
URL: https://github.com/apache/beam/pull/25915
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25915:
URL: https://github.com/apache/beam/pull/25915#issuecomment-1478319891
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @robertwb for label java.
R: @chamikaramj for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25915:
URL: https://github.com/apache/beam/pull/25915#issuecomment-1485514176
lgtm1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #25915: validated the input schema matches with the bigquery output schema
Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25915:
URL: https://github.com/apache/beam/pull/25915#issuecomment-1490543935
I'm seeing this error in several places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org